We have no evidence to suggest we shouldn’t release

“We have no evidence to suggest we shouldn’t release.”

I sometimes use this statement or a variation of it, when asked if something we’re testing is ready to ship. For me it succinctly describes the current state of our knowledge of the software under test.

A longer form could read

“We have completed all of the testing we could think. From that testing, no new information has appeared which suggests that we should not ship, based on the goals and risks associated with it.”

If the person who asked the initial question about our release readiness is interested in our testing efforts and their value, I’ve found that the statement becomes a good conversation starter to go deeper.

The statement is designed to suggest that we’ve done all we can within the allotted time. It also leaves room for unknowns which may impact the software should we release.

These unknowns could be categorised as:

“We haven’t been able to describe the unknowns sufficiently, therefore allowing them to be considered tangible risks and therefore ‘known unknowns’.”

or

“There are known unknowns i.e. gaps in our application knowledge that have been identified, but for some reason or another we haven’t been able to fill through testing.”

The language that we use is often a product of our state of mind. Acknowledging that it is impossible to know everything and therefore we’re unable to offer an absolute, I chose to offer the next best thing that my language skills allowed.

As long as we can back up the response with the testing efforts so far completed, and/or have not completed if the time for testing has expired, I believe that this answer is an ok response in this context. It leaves everyone on the same page, sharing the uncertainty that comes with complex systems and allowing the decision to release to become a group decision.

But why do I still feel a little dirty using it? James Bach terms it ‘safety language’ and that’s what it is to me. It’s a nod to the uncertainty of software and that we can’t know everything. Having said that, part of me feels like there may be a better alternative.

Firstly, the English is appalling! A double negative would be an instant fail in many an English exam and because of this, it leans towards being cryptic, as if it was meant to catch people out.

It was not designed to be cryptic, but when spoken in a particular way, it can definitely be made to make most people raise an eyebrow and ask for it to be repeated slowly.

If we were to take out the double negative, alternatives could be

“The release can go ahead because we do not have sufficient evidence to stop it.”

Or

“The evidence we have gathered suggests we can release.”

Both seem ok on face value. On closer inspection, the first option has broken my rule on deferring the decision to the appropriate owner. If I were to use option 1, I would feel like I’d just made the decision to release and that’s not something I think Testers and test teams should do.

Option 2 fairs better on that score. We’ve qualified ‘can’ with a level of uncertainty in the use of the word ‘suggests’. But to me it still feels a little too much like the tester is making the decision.

Depending on your relationship with your decision makers and the culture of the team, option 2 might come across as a better alternative, one that is phrased without a possibility of being misunderstood as deceptive.

“What other alternatives are there?”, you ask. How about this:

“We have/have not completed all of the critical testing we can. With more time we may/are unlikely to, find a reason not to release.”

This isn’t as concise as our opening gambit, but is it better in other aspects?

We’ve gone from talking about a lack of evidence to making a statement on what has or has not been tested. We’ve also covered the level of value of the testing, by focussing on what is critically important.

We’ve also described a possible future situation whereby if we were to continue testing, we believe new risks would or would not be uncovered.

Finally we’ve also finished off with a refined version of our passive yet positive outcome, that the release could go ahead, but still without being explicit and appearing to make the decision ourselves.

I’ll give this a go for a while and share my experience.

It would be great to find out what works for you!