← Back to blogs

Common sense: the hardest thing to test

qasoftware-developmentengineering-culture

I still remember something my former Head of QA said many years ago after reviewing some engineering decisions made by other teams in the company.

He sighed (he was able to put a lot of emotion in his sighs) and said:

"Common sense is the hardest thing to find these days..."

At the time, I thought it sounded a bit harsh. I understood what he meant, but I also thought it was a little unfair. We had some great people in the company and everyone was really trying their best. Also, what is obvious to one person is not always obvious to someone else and as QA we just often saw a bigger picture than most others.

After many years working in QA, I think I now understand the point he was making. The hardest thing to test is often not the product or the code, but everything around it: the assumptions people make, the decisions nobody questions, and the things that seem too obvious to write down. Common sense is in a way the requirement nobody creates a ticket for.

Let's see if this sounds familar! Most teams have user stories, acceptance criteria, designs, refinement meetings and documentation (well, sometimes!). From the outside, everything looks great.

And then a team member, often one of the annoying QA, asks:

"What happens if we actually do this?"

Complete silence. It turns out nobody thought about it yet.

As I mentioned, those questions are often asked by QA, and especially when QA gets involved later in the process. Not because QA has all the answers, but because part of the job is looking at what happens when things are used differently from what was expected.

The ticket is ready, but are we?

The word "ready" is interesting. You will hear it a lot, but everyone means something completetely different with it.

For example: a ticket has been refined, acceptance criteria exist and everyone understands the goal, so the team starts building. A developer picks up a ticket and starts working on it. The implementation looks straightforward until they discover an issue: the API isn't ready yet, the backend team still needs to decide what the data should look like, or another team owns part of the functionality and nobody checked whether it fits their planning. The ticket that was considered ready is now blocked.

This doesn't mean someone made a bad decision, these situations happen. It is just that "ready" can mean different things. A ticket can be ready from a product perspective, but not ready from a delivery perspective.

Often, it only takes a few minutes of discussion to uncover these things:

"Who else needs to be involved?"

"Are there dependencies we haven't discussed?"

"Do we know how the teams will work together?"

"How are we going to test this?"

"What will we pick up first?"

"Do we need to talk to someone?"

"Do we actually need this right now?"

"Did we consider all options and is this really the best and easiest way to do this?"

These are not questions asked to slow things down. In fact, they usually prevent people from waiting later (part of Shift Left).

The infamous "we can just quickly test this in production"

There are not many sentences that make QA nervous faster than:

"Oh, we can probably just quickly verify it in production."

And to be fair, sometimes that is completely logical and fine. Some teams intentionally test in production, either voluntary or forced due to a very complex architecture or third part dependency. They have the right safeguards, monitoring and rollback options in place. However, it is more concerning and also much more common that "quickly check it in production" becomes the easiest option.

What I find interesting is that people who would never run a risky database query on production can sometimes feel completely comfortable doing a quick verification there because it sounds harmless.

But it isn't harmless, it is risky. Product is different from a staging or testing environment. There are real users, real data, and customers use and depend on that environment. A quick 'innocent' check can create data, send emails, trigger payments or leave behind something that somebody has to clean up afterwards. It can even affect other real users.

Before doing something there, I think the most useful question is:

"What happens if this does not behave as expected?"

Again, testing in production is not automatically wrong. But doing it without thinking about the consequences is where problems start.

"It is only a small change"

I think almost everyone who has worked in IT has heard this sentence as well:

"It is only a small change."

Usually, that is the moment where someone should ask a few more questions. Why? Well, because small changes have a habit of touching things nobody expected...

It might affect:

  • another system
  • another team
  • an integration
  • reporting
  • a mobile application
  • an (incorrect) assumption that has existed for years

This becomes harder as companies grow as less people have the complete picture. A developer can understand their part perfectly while having no visibility into what depends on it somewhere else. That is normal. Nobody can know everything. So the goal should be to create enough space for people to ask questions before something reaches users.

Assuming another team knows

Another classic situation:

"We already informed them."

That's great!

But, did you really inform the right people? And how? Was it a Slack message? A ticket? A meeting? And did the right people actually understand what they needed to do?

Cross-team problems are often not caused by people refusing to communicate. More often, everyone believes the communication happened somewhere else. Sharing information is not always the same as creating understanding. A team knowing that something is changing does not automatically mean they know what action they need to take or when they need to take it.

Automation is not always the answer

QA can fall into the same trap. We obviously make assumptions as well.

Sometimes we build something because it is interesting, new or technically satisfying, and not because it solves the (biggest) problem.

Automation is a great example. It is extremely valuable, but I have seen teams spend months building checks that nobody runs or trusts afterwards or even that they only catch low impact issues and completely miss the biggest and crucial ones. I have seen End-to-End frameworks with over 10000 tests!! Yet, every release medium to high impact issues would slip through the net.

The questions I usually ask are:

  • How often will this actually run?
  • How much effort does it take to maintain?
  • What problem does it solve?
  • What happens if this feature breaks? What is the risk level?
  • How do we report the outcomes?
  • Do we have data and evidence that users actually use this a lot?
  • Is this the best place to do this check, or is there a 'more left' or lower level where we can do this?

I have definitely been guilty of this myself. I love coding much more than manual testing and building automation can be very satisfying. Creating a clean framework or improving a pipeline feels good. But sometimes you can end up with something that is technically impressive, but nobody really needs it. It does not offer value to the company.

There is nothing wrong with the automation itself. The mistake was deciding that automation was the answer before understanding the problem.

QA does not own common sense

The problem with the questions I mentioned before is that QA should not be the only role asking these questions. Quality is, or should be, created throughout development. Not just QA, but Developers, designers and product managers all influence how something behaves for users. On top of this, QA can/should challenge assumptions and help teams think about risks.

But everyone involved owns the result, not just QA. I have seen more problems caused by people being afraid to ask simple questions than by people asking too many of them. It is better to be a bit 'annoying' in the beginnign by asking a lot of questions than to run into problems later on.

A high bug count is unfortunately still seen as QA doing well, as bugs are being found. However, in reality, the best teams I have worked with were not the ones where QA found a lot of bugs but where people felt comfortable raising concerns before something became a problem. Prevention over fixing.

Process versus thinking

There are a lot of people that strongly dislike processes. I am not one of those people, as I actually like processes. They have helped me prevent so many issues and streamlined a lot of painful and frustrating ways of working. Good processes remove confusion and help teams work together better.

The main reason why some people start to dislike processes is when we have processes for the sake of having processes. The process becomes more important than understanding why it exists. You should always ask yourself what the problem is you are trying to solve first.

I have seen situations where every box was checked, every meeting happened and every document existed, but people still had different expectations about what was being built and the outcome was ultimately not satisfying.

So how is that possible? Everyone followed the process and all safety gates were in place. What went wrong is that there was no thinking behind it. A process should help people make better decisions. It should not become something people complete just because it exists and nobody can tell you why.

There is nothing wrong with having processes, they can really make your life easier and more pleasant. Just make sure that you have a small set of rules that are consistently reviewed and updated, actively solve a problem and are properly understood by all team members. It is rarely the team with the highest amount of processer or the one without any processes that performs the best. Those teams are often also the ones where people feel responsible for the outcome and are comfortable asking questions.

Final thoughts

So, after all these years, I understand much better what my former Head of QA meant.

He didn't question people's intelligence and he didn't question their best intentions. He did question how much actually think about what they are doing. Think before acting.

Software systems rarely stay simple for long and team members come and go. This makes it easier for incorrect assumptions to happen, for people following processes without thinking, to act without thinking about possible consequences, etc.

It is crucial to keep things simple (KISS), to keep asking (obvious) questions and to keep challenging assumptions.

The first question I still try to ask first:

"Wait... should we really do it this way?"


Alex Hovenkamp