July 26, 2021 by Faisal Alghurayri
The same, only different
In their book “A Practical Guide to Testing in DevOps.” Katrina Clokie discusses what is needed to adopt a healthy testing culture in multi-disciplinary, highly-skilled, interconnected product teams. The first logical step is establishing a shared context for the whole team to understand what testing really means. I find their approach to achieve this feat to be brilliant.
They suggest conducting a one-hour workshop for the whole team - product managers, designers, developers, operations, support, and testers to gauge the current comprehension around the implemented but not necessarily documented testing strategy by visualizing it.
To start, the session leader draws a timeline from idea to production. Then, they ask the team to name every testing activity and who should perform it by filling up the three differently-colored sticky notes according to the following rules:
Workshop to uncover the different ideas behind testing
Then, the whole team can reflect on how people will have their other ideas about testing, the types of testing, and when each type should or should not be occurring by answering the following questions:
I conducted this mini-workshop by doing multiple small talks with my colleagues. I was amused by how every person has their ideas around testing.
I share this fascinating story to spur your thinking about what does coverage means.
Search for the word coverage in the testing dictionary. For example, you can find code coverage, test coverage, path coverage, branch coverage, statement coverage, decision coverage, condition coverage, and name your criteria coverage.
In my opinion, all these types of coverage are the same but different. As engineers, we try to increase our confidence about what we know by slicing, dicing, and mashing different kinds of metrics. In general, I think there are two types of coverage:
Take this classic example - a division only calculator:
Division only calculator
A programmer can build the logic as follows:
const div = (a, b) => a / b;
If you have a tool that measures coverage and you have only the following test, such a tool will yield a 100% code coverage
const expected = 5; const actual = div(10, 2); assert(expected === actual);
However, you can easily spot the defect in the logic. The app will instantly crash if you run
div(10/0). Achieving 100% code coverage doesn’t mean that you have 100% test coverage.
So should you care about coverage? I don’t know. Every team should decide the what, how, and why of their testing strategy. In the above example, I uncovered how tool-reported coverage could be deceiving in making assumptions about the state of an application.
Code coverage is beneficial to detect unreachable code and to add confidence about handling what we know. On the other hand, test coverage should widen the surface of things we should know, but you can never truly measure it.
In conclusion, I think the question of coverage should always be a rhetorical one. However, to make sound decisions, I encourage you to learn more about the principles of software testing.
Take it easy,