Assumptions-in-testingQuestioning assumptions
The first Testing meet-up I went to was a session on Test Planning. Nearing the end of the experience report that started the conversation, the presenter quoted James Whittaker as saying “Good testers can never assume anything.” I hadn’t heard of James Whittaker at the time, but this was something that stuck out to me. The topic was eventually raised and discussed briefly, but this would inspire a long time of thinking about the humble (or not so humble) assumption, and its place in relation to Software Testing.
In my childhood, my introduction to assumptions was from a common phrase that I still hear a lot out in the world that goes along the lines of “assume, and you make an ass out of you and me”. The obvious negative connotation and pun-ish word-play makes it stick in my head, but really, are assumptions a bad thing? The phrase seems to expect that any assumption would be a bad assumption, would have a bad consequence. This was my first and only exposure to the term and the concept for many years, and this built an inherent negative view of assumptions. The reason explained was always that assumptions are an exercise in laziness that can bite you in the proverbial – that it was better to just ask than to assume.
Are assumptions always, or predominantly negative? My learnings have taught me ‘no’ – in fact, assumptions are very important. I thought about the possibility of good, neutral or required assumptions, and the idea that even if it was a bad assumption to make, does that mean it wasn’t the best thing to do at the time? Something I learned as a teenager was that assumptions are a necessary part of dealing with the world – we can’t know everything, and anything that we don’t know everything about that we have to interact with will involve making assumptions. Are we assuming all assumptions are negative?

If assumptions are necessary, we need to know when it is justified to make them, and what makes them good or bad. I pose two direct questions to explore:
● What makes an assumption a ‘reasonable assumption’?
● How does this relate to me, and to Software Testing?

How does this relate to me, and to Software Testing?
We’ll start with the second question – well, assumptions relate to everything. They’re a part of your everyday life just as much as our jobs or hobbies. We generally assume when we go into a store that they’ll accept reasonable forms of payment (such as cash, card or others depending on your country). We generally expect, unless we have reason to believe otherwise, that people who live in our country speak the country’s primary language, and we might be put out, annoyed or frustrated if these things aren’t the case.
As for testing, you can’t reasonably expect (there’s that ‘reasonable’ word again) to have all the information on-hand before starting testing – I’m not sure how much Testing would be required in a situation where we had all the information we needed about the software under Test. Regardless, if you’re going to be dealing with people and you can’t read minds, then you definitely don’t have all the information. When we lack information, we tend to make assumptions.

How it directly relates to testing is in how you use the information you gain from knowing about assumptions, both your assumptions and those of others; generally, knowing about something is better than not knowing, and if you know what your assumptions are you can question them, you can learn from them. Questioning your assumptions before you make an ass of you[rself] might save you (and the implied ‘me’) precious time and effort in your testing. Additionally, as assumptions are a form of expectations, they influence many test practices – how many actions do we take with declared expected outcomes? One of my old colleagues quipped that some testing that is done for the purposes of checking Developer or User expectations (e.g. extended Unit Testing, UAT) could even be referred to as “assumption-based testing”.

If you don’t know what your assumptions are (because sometimes things aren’t that simple) it’s not always easy to question them yourself. Similar to how testing is traditionally done by people or teams independent from the developers, having a separate person to question your assumptions can be helpful – this is one of the benefits of Peer Testing. The other person doesn’t even have to be a tester – asking questions or even better, use of the Socratic Method, can draw assumptions out into the light.

So we can see already that assumptions are already a necessary, if not integral part of testing. That still leaves the topic of what sort of assumptions we need to be focusing on, either to know and accept them, or improve them. Which of these we do comes down to whether the assumption is reasonable or not.

What makes an assumption a ‘reasonable assumption’?
Back to the first of our two questions. I’ll start with a simple definition, and we’ll expand on it:
● An assumption is reasonable for as long as the benefit in holding it outweighs the risk in holding it.
This definition seems to fill the needs for an assumption to be ‘reasonable’, but that’s because it’s overly simple. All it really states it that it is easier to hold the assumption than it is to not hold it. From a pragmatic point of view this would be all that is needed, and ironically takes us to the reason that I was told that assumptions were bad – laziness, which can be the parent of efficiency.
Let’s test the definition: This statement seems to assume we know the benefits and risks involved, and it doesn’t clearly define the types of these values (absolute values, known values etc). It also reads to me that there’s no reason to question it unless it changes. But it’s somewhere to start.

As implied earlier, the benefit of an assumption is that it means that we don’t have to spend our resources (such as time and effort) on thinking about things that we have theoretically decided we don’t need to be thinking about – ‘decided’ being key here; not knowing your assumptions is dangerous. If we assume a certain baseline that we’re working from, then we cut down how much questioning and information gathering we need to do in order to understand what we’re looking at. As much as testing is questioning and information gathering, there’s always more testing that can be done than time to do it in. Assumptions allow us to focus our activities where they might be best placed, given what we know.

The main risks involved in assumptions stem from wrong assumptions. What if you assume that information given to you is correct, but it isn’t? What if you assume that the main purpose of a system is one thing, when it’s secondary to another one? As much as finding and challenging your own assumptions can send you off down a long path, running off a wrong assumption can send you down a different one. A secondary risk on a recursive level is that questioning them takes time and effort – there is an opportunity cost involved in getting the information to challenge assumptions (just as there is an opportunity cost to deciding whether to question them – try not to go too far down the rabbit hole with recursion).

So when do we change our (no longer reasonable) assumptions? The direct but non-specific answer is “when we have a reasonable reason to do so” – we want to make sure that the value we might get out of changing it is high enough to warrant the opportunity cost of reassessing it.
So, knowing value (possibly measured in risk vs. reward) and [opportunity] cost as the things driving when we reassess our assumptions, we should now have a fuller picture of what a reasonable assumption is:

An assumption is reasonable for as long as:
● the known benefits in holding the assumption are positive, and
● you have reason to believe the opportunity cost of reassessing the assumption exceeds the potential gain in benefits

Beware the unknown assumption
All the above advice is great for assumptions you’re aware of, but a large danger with them lies in those you aren’t aware of. In order to examine an assumption to see if it’s reasonable or not, we need to know about it. Two good methods I have found of bringing them to light are:
Firstly, changing scale. If you change the scale of your focus (thinking of a system in smaller or larger parts) of an idea and turn it into several smaller interconnected ideas, it forces you to evaluate how it/they really (or if they really) function. Similarly, if you look at the larger scale and view your idea in context to the system it’s a part of, it will force you to evaluate it for internal consistency.

Secondly, changing format. If you transpose your thinking into a different format or structure – such as in pictures or diagrams, rather than words and writing – then you might find that where your perspective on the idea changes enough to allow you to see where you don’t understand it well enough. This also extends to other format changes, for instance the Feynman Technique, which is changing the way you explain an idea, which will change the way you have to put it together.

It’s only once you learn you’ve made an assumption, or once you learn something that could challenge a known assumption, that you can decide whether it’s worth challenging it. Here another hurdle starts on a different topic – how to assess new information, given how bad humans are at measuring risk, uncertainty, and knowledge that differs from our current beliefs; but that it is a topic for another day.

In summary
● A known, reasonable assumption is your friend. It helps you focus on the things that matter right now, it helps you focus on things that are in the scope you want to focus on.
● An unknown assumption isn’t necessarily unreasonable, but it isn’t subject to challenge in the event that it becomes unreasonable.
● Bring your assumptions to light by re-examining what you know from a different perspective
● Spend your time and effort on gathering information to challenge your assumptions when the effort isn’t best spent elsewhere. We don’t have the time to do everything, so you need to pick your battles.

This article was published in our October 2014 edition.

https://i0.wp.com/www.testingcircus.com/wp-content/uploads/Assumptions-in-testing.png?fit=730%2C372&ssl=1https://i0.wp.com/www.testingcircus.com/wp-content/uploads/Assumptions-in-testing.png?resize=150%2C150&ssl=1Joshua RaineArticlesTesting ArticleQuestioning assumptions The first Testing meet-up I went to was a session on Test Planning. Nearing the end of the experience report that started the conversation, the presenter quoted James Whittaker as saying 'Good testers can never assume anything.” I hadn’t heard of James Whittaker at the time, but this...