Why do you need exploratory testing?
Imagine the following scenario: You are developing the "CarConfigurator" software for configuring new cars for an automotive manufacturer. You have already automated all regression tests, both module tests, integration tests, acceptance tests and even the end-2-end system integration tests. Everything runs in an exemplary Continuous Integration environment and is automatically tested through with each new release. Actually, it should now be ensured that no new errors can creep in at your end. Nevertheless, "obvious" - not to say "stupid" - errors show up in productive operation. For example, a changed text is only partially displayed, and a button has such inappropriate colors that the label is difficult to recognize. This carries the risk that the user will also associate the poor quality of the software with poor quality of the new cars. You can avoid this with exploratory testing: By testing with experience-based glasses, many obvious, but also hidden and unusual errors are found before release.
What is exploratory testing?
ISO 29119 calls exploratory testing a testing approach in which tests are designed and executed dynamically, based on knowledge, exploration of the test object, and the results of previous tests. For an easy-to-understand explanation of exploratory testing, see our glossary video on exploratory testing.
An overview of the main elements of experience-based testing and their context can be found in the adjacent graphic.
Based on past experience and recognized knowledge, we develop conjectures with a lot of creativity where errors could be hidden in the test object at hand. We explore in a subsequent test session, with a lot of curiosity whether our guesses are confirmed or not. During the exploration we gain new knowledge and expand the experience - thus closing the circle.
Example: Imagine you have the task to test the new version of the CarConfigurator. An essential function is the price calculation of the configured vehicle. With the new release the feature "fleet discount for major customers" is introduced. You know how the price calculation should behave in connection with this new feature. In the past, you have made the experience that many errors were hidden in the price calculation. Therefore, you suspect that new errors have crept into the price calculation area again. You therefore plan a test session on price calculation with several experts and users. Together, you explore the new fleet discount feature for major customers.
At the end of the test session, you learn in a debriefing session what additional knowledge the participants have built up during the exploratory testing. You realize that your assumption was fortunately wrong, because no significant errors were found.
How can the reactive approach be integrated into the software development process?
In systematic testing, one follows the previously created test plan. With precise plans, there may be too little room to react to new findings. If everyone is just reacting, there is a risk of no longer following a common strategy.
Session-Based Test Management (SBTM) offers the possibility to react to new findings on short notice and to develop a common strategy at the same time. It is usually called Session-Based Testing. SBTM relies on the experience of agile team members and assumes that they are good at making assumptions. In our example with the vehicle configurator, such a conjecture could arise from the fact that there have often been problems with invisible texts in the info boxes of an equipment package. These assumptions are condensed into clear goals for the execution of a session, the test charters. Test charters provide time-constrained, intensive testing activities with clear test objectives and, therefore, a clear sense of purpose. Test charters are guideposts for the testers as to which test element they should investigate and what specifics should be considered. It is easiest to compare the test charter with a clear mission for the session.
In our example, the test charter formulates the goal of describing new text box elements for equipment packages in the vehicle configurator with overlong texts. This test charter is taken over by a developer in the test execution, who knows which GUI elements have been newly created in the framework, together with a tester within a period of 90 minutes.
In this way, the supposedly random experimentation on the test object becomes a targeted experiment (= explorative test) in a given time frame (= test session). Finally, the reactive approach of explorative testing can be promoted in which test managers, development team members or other interested persons jointly discuss the essential findings in a debriefing at the end of a session and develop further ideas on how to proceed. And just like that, experience becomes valuable knowledge again!
How do you generate good test charters?
With the following questions, you can reach good objectives and thus a test charter quite quickly:
- What changes were made to the product in the current sprint?
- What side effects on existing functionalities could the changes have?
- How do users use the product?
- Where do errors particularly "hurt"?
- Where have errors occurred frequently in the past?
- What qualities are not talked about in the specification, but relevant stakeholders would miss?
In general, common creativity techniques can help formulate valuable test charters for exploratory testing. To be able to find gaps in test coverage, you always need ideas for areas where you have experience gaps yourself or for which your own creativity is currently lacking. Checklists, testtoure of error attacks are examples of creativity techniques that are often used in SBTM.
What distinguishes a tour from a test session?
Strictly speaking, tours are one of the possible creativity techniques that use metaphors to find test charters to design test sessions. For example, the analogy of exploring a city is drawn. The testers are asked to approach the exploration of the test object as if they were visiting an unfamiliar city. They should make sure that they get to know the main sights. The testers usually start by looking at a city map together. The "tour group" decides which parts of the city are interesting. Once the parts of the city are decided, send the individual interest groups to different areas.
In the evening, the excursionists meet over a drink and share what they have experienced. Inspired by this, the group plans the next steps and considers which parts of the city should be explored next, or looked at again more intensively.
With the help of this "visualization" of the test object as a city and description of the test tours, it becomes easy to derive exploration goals, i.e. test charters, and to plan meaningful sessions for explorative testing.
Who is involved in a test session?
In exploratory testing, ideally the whole team or people outside the team can participate. The focus here is not explicitly on participants with testing expertise. In this way, many test charters can be followed in the same period of time and all team members also get to know the entire application from new perspectives. Since experience grows particularly well together, it is very proven to repeatedly explore the test charters in pairs or even larger groups.
How do I introduce SBTM if I have no experience with it?
The question is, what exactly is missing? Below are a few suggestions:
- If basic knowledge about the components of SBTM is missing, an experienced coach can support in the early days.
- The use of coaching and training, such as Agile Tester - Exploratory Testing are helpful for profitable use (link on AGEXP).
- If you are wondering how to get to more checklists: Checklists are written down experience. For example, you can ask yourself what has gone "wrong" in the past.
- Knowledge about classical test procedure is also helpful for SBTM, so ISTQB trainings help with problems regarding test design.
- With domain knowledge it often helps to involve domain experts temporarily.
- Do you have inexperienced testers in your team? Have two testers perform the experiment together: Pairing transfers experience and builds new ones.
How does imbus help you with exploratory testing?
imbus supports you with trainings, in which we teach your team members the mentioned techniques in a comprehensible and practical way.
We can also support you in the daily doing by providing your team with an experienced quality engineer who can prepare these explorative tests and coordinate their execution.
Would you prefer to tackle the topics yourself, but be guided by an experienced coach?
Please contact us!
Your contact person at imbus
Mrs. Klaudia Dussa-Zieger
mail: quality-improvement@imbus.de
phone: +49 9131 / 7518-740
fax: +49 9131 / 7518-50