How it started
Everything starts with something. This one starts with a video of ThePrimeTime, you know I love his videos. And as usual with me, I arrive one month later to talk about this video after I watched it. The video is an interview with a QA that worked in Blizzard, quite long, so maybe you want to watch it later.
In essence, it is about the clusterfuck that big game companies represent nowadays. And as it is tradition, whatever we see, infer or imagine from outside, it is even worst. Unfortunately, no much surprise here. But let’s stick with the initial story that started this all. In the middle of a shift, they gathered all the QAs and told them to go to a room to do a Critical Task. In the middle of the 30 points it had, there was one instruction that said: “turn off the computer, stand up and lead the room”. The last two guys that did that were fired at the spot.
There are some anecdotes that let you feel the terrible place to work that studio has become. In essence, they despised QA and they considered them to be less than people, not just engineers. And that got me upset and decided to write this…
QA will save your life. Every time.
Early in my career, I had little experience with QAs. I developed my task, send for review, and once that passed, it was merged into a master branch. Whatever happened later I only knew if something didn’t work in my changes, because some QA would come and tell me how to reproduce the bug. But for me, and based on how the companies were organizing QA departments and QA work, there was no difference between that “fix from QA” and a regular task. Not that it was, but the perception due to the company culture and the small knowledge I had.
But after some more corporate experience, I got myself in a startup. And the cluster where we would run tests and code was in the process of being configured. My team worked in building a ETL (data pipeline) so it was data intensive. And there was no QA, so I finally understood what they were doing because I had to deal with it… and it is tough engineering. It was plagued of scripts that we were doing frantically to try to simplify the configuration of nodes of the cluster, connect different stages together, downloading the test data in an easy way… all of that just for having some reasonable testing environment that could allow us to focus on the testing of the data and features themselves. It was a mess.
A few months into this, with a more or less functional set of scripts to use, the company hired a guy for QA. I have to say that there was a QA team, but the team leader was an incompetent and he didn’t hire right. So no one really knew much about testing, but the frontend. So we didn’t have much hope in the new guy, but how wrong we were!
Petr P. was the name. What a legend!
Memories are flaky, but this is what I remember about him and his work there. A young quiet guy that really knew his craft. During the first weeks he was asking questions, reading the scripts we wrote, understanding the way we tested and the goals we had for the testing cluster. And during this time he kept working and improving the set of scripts. Finally he came and showed us what he did. He unified and improve the scripts into a smaller and cohesive set of functionalities. He automatized everything and more and created some scripting tools to extract result data, compare it with the input data and show some other analysis to check if we were in the right path.
Soon after that, and for the time he was in the company, he took over testing completely. You just send him a message with the feature branch you wrote, the location of the testing dataset and what keys you were interested in for that task. After a while, he would come back to you with the location of the results and some info he gathered. If you needed to check something else, it was a matter of seconds before he could pull it off from the cluster. And he did this for the 4 or 5 developers we were in the team, working in different features and parts of the pipeline, managing everyone’s testing needs. It was a x10 multiplier for the team, no doubt.
Companies do not understand QA importance
At that point, he was working within our team on the ETL. Technically, he belonged to the QA team, lead by that incompetent guy I mentioned before (really, I am trying to be nice here, he gave us too many problems with his “ideas” and plans). And that team was severely underperforming. Most of them didn’t really know programming, and I mean even scripting. That’s why they were focused on front end, because they had a visual tool to set paths in the web, etc.
So the company fired the whole department. Including Petr. The team manager fought tooth and nails to keep him in our team and presented evidence of his work and how it was helping us to be efficient and fast. Some of us that had been there since the beginning of the team, chipped in and made the case for him to stay.
Nothing mattered. He was fired with the rest of the team of QA’s. And then I felt the importance of a good QA engineer. Our speed decreased, releasing a new feature became 1 week of implementation and local testing and another week of testing in the cluster, gathering data, finding errors going back to coding, then back to the cluster. There was a lot of context switching and that didn’t help.
Because companies do not really understand the importance of QA. A lot of managers think developers ask for QA because they don’t like that job and they try to get someone to do it for them. Stupid opinion, but it is out there. But it is about efficiency, about having people focused on the details of testing environment and tools. But a lot of companies I knew think of it as a waste of money as developers should do that.
Developers and QAs: the symbiosis
I am not saying that developers should not do testing. You are developing something and you need unit tests to be sure nothing broke, and also test the thing you are developing to see if it works as expected. But those are local tests. In the sense that you will test mainly what you are touching and implementing. But once you have a complex system that interacts with several other parts, produce different outcomes, files and data… you have to have a more solid testing process that convince you it works. Think about tons of data, million records, that you need to parse and produce outputs based on different conditions. You can have unit and integration tests in the codebase that the developers will run several times during the development process. But later you need solid extensive tests that do regressions, stress tests, etc. On top of that, in this case statistic were needed and gathering the right data in an automatic way that is reliable is a development on itself. There is where you want your QAs.
Are QA developers?
Of course! Focused on a different part of the process, but to be good you may need to write code. And in my experience, they need to be very flexible and able to catch up fast with other programming languages to do their job.
However, the problem is that as companies do not really understand this in general, and as there is not many places where really knowledgeable QA engineers are recognized, no one seems to think they do much more than just repeat the process of testing or running some commands in a server.
On top of that, QA looks like opponents for Dev team. Why? Because their job is to break your code. To find the bug, the screw up, the piece that you did wrong and breaks some other part that you didn’t even know it was influenced by it, because you were focused on your development. So when you finish some hard or painful work and the QA comes back at you with a list of things that you have to fix… sometimes you hate them.
But I have been in the situation that you are your own QA. Actually, after the company closed the department. And it happens sometimes that something you did a few months ago, beaks after another change and it comes to bite you. I can say that the stress when releasing new changes increases. Maybe it is a personal thing only, but without a good QA team, or a QA engineer in the team, the confidence in the releases was lower and carried more tension. QA gave me confidence because they are in a quest to destroy my code. They were the gatekeepers and if they let my changes to go through I could be confident that it would work and the only problem it would have would be uncovered bugs, but not my changes.
So they are in our team. They are us, actually. We work in the same team (even if not in the organigram) and we should advocate for having a QA engineer next to us every time possible.
Now, this is what I wanted to say about the video I linked at the top, because I met and worked next to great QAs like Petr and I miss them every time I work in a complex system and they are not there. Go watch the video if you didn’t yet. And defend you QAs.