Exploratory testing (in agile projects)

Imagine the following scenario: You are developing the "CarConfigurator" software for configuring new cars for an automotive manufacturer. You have already automated all regression tests, both module tests, integration tests, acceptance tests and even the end-2-end system integration tests. Everything runs in an exemplary Continuous Integration environment and is automatically tested through with each new release. Actually, it should now be ensured that no new errors can creep in at your end. Nevertheless, "obvious" - not to say "stupid" - errors show up in productive operation. For example, a changed text is only partially displayed, and a button has such inappropriate colors that the label is difficult to recognize. This carries the risk that the user will also associate the poor quality of the software with poor quality of the new cars. You can avoid this with exploratory testing: By testing with experience-based glasses, many obvious, but also hidden and unusual errors are found before release.

In systematic testing, one follows the previously created test plan. With precise plans, there may be too little room to react to new findings. If everyone is just reacting, there is a risk of no longer following a common strategy.

Session-Based Test Management (SBTM) offers the possibility to react to new findings on short notice and to develop a common strategy at the same time. It is usually called Session-Based Testing. SBTM relies on the experience of agile team members and assumes that they are good at making assumptions. In our example with the vehicle configurator, such a conjecture could arise from the fact that there have often been problems with invisible texts in the info boxes of an equipment package. These assumptions are condensed into clear goals for the execution of a session, the test charters. Test charters provide time-constrained, intensive testing activities with clear test objectives and, therefore, a clear sense of purpose. Test charters are guideposts for the testers as to which test element they should investigate and what specifics should be considered. It is easiest to compare the test charter with a clear mission for the session.

In our example, the test charter formulates the goal of describing new text box elements for equipment packages in the vehicle configurator with overlong texts. This test charter is taken over by a developer in the test execution, who knows which GUI elements have been newly created in the framework, together with a tester within a period of 90 minutes.

In this way, the supposedly random experimentation on the test object becomes a targeted experiment (= explorative test) in a given time frame (= test session). Finally, the reactive approach of explorative testing can be promoted in which test managers, development team members or other interested persons jointly discuss the essential findings in a debriefing at the end of a session and develop further ideas on how to proceed. And just like that, experience becomes valuable knowledge again!

Strictly speaking, tours are one of the possible creativity techniques in which metaphors can be used to find test charters in order to design test sessions. For example, the analogy of exploring a city is drawn. The testers are told that they should approach the exploration of the test object as if they were visiting an unknown city. The aim is to ensure that they get to know the main points of interest. The testers usually start by looking at a city map together. The "tour group" decides which parts of the city are interesting: once the districts have been determined, the individual interest groups are sent to different areas.

In the evening, the excursionists meet over a drink and talk about what they have experienced. Inspired by this, the group plans the next steps and considers which parts of the city should be explored next or looked at more closely.

With the help of this "visualization" of the test object as a city and description of the test tours, it becomes easy to derive exploration goals, i.e. test charters, and plan meaningful sessions for exploratory testing.

Ideally, the entire team and external persons can participate in exploratory testing - the focus here is not explicitly on participants with testing expertise. This allows many test charters to be tracked in the same period of time and all members also get to know the entire application from new perspectives. As experience grows particularly well together, it is advisable to always explore the test charters in pairs or in even larger groups.

If you compare the classic test procedure with the SBT, the following difference becomes apparent: In the structured test procedure, the test cases are usually derived from requirements documents, the so-called test basis. Test coverage can be monitored - are all requirements covered by a test? These test cases are often already fully specified before the test object is even available for testing. If the test cases are then executed, e.g. in the system test, there is a risk that the tests will be executed without reflection - the testers are often not explicitly instructed to do so or simply have little time to search for errors to the left and right of the test case. In other words, it is often considered sufficient to check that the required functionality is present. This approach is typical for software projects that are carried out according to the waterfall or V-model.

More flexibility and creativity in exploratory testing

In exploratory testing, the test cases are not necessarily available at the start of the test. More precisely, there are no test cases in the classic sense - the testers stick to the idea or creative approach of their test charter. Their discoveries may lead to the areas to be examined being refined and planned for the next exploratory session. It can therefore be said that the "test cases" "mature" in the sessions. In addition, new areas are discovered in the test object that should be examined more closely. This flexibility in focusing on a specific goal, as well as the associated creativity in devising test cases, is a significant difference to the classic test procedure and can therefore generate significant benefits.

SBTM is also suitable for test objects for which no test basis (missing source documents) is available.
Exploratory testing can generate added value, particularly in an agile environment - when testers need to make a quick statement about the quality of the test object within a sprint. SBTM is an optimal addition to the acceptance criteria.

Exploratory testing should always find bugs! If the relevant sessions are neglected over a longer period of time, the lists of obvious bugs (uncovered by exploratory testing) are often alarmingly long. Due to the broad and creative focus of the test charters on subject areas in which risks are suspected, exploratory test sessions often lead to unclear results. Whether the findings reveal a bug, a (previously unspecified) feature request or just the exploratory tester's own bias should be clarified in a joint debriefing after the session. It is possible to hold overview sessions that focus more on generating experience than on finding bugs from the outset.

Exploratory testing is the decisive "sanity check" that can prevent other - equally important - test approaches from going off track. The question should therefore not be whether session-based testing is taken into account, but rather whether the mix of "classic" and session-based testing is well chosen.

If you want to introduce SBTM, it may not initially be possible to shoulder this "on top". Are there areas where you need less classic testing effort if you introduce SBTM? If so, this would be one way to make time for SBTM. For example, demo appointments with customers are structured with the help of SBTM - in the end, most of them want to use the system themselves anyway, which is why it makes sense to simply send the participants on a fact-finding mission directly with a test charter.

imbus provides training to help you understand exploratory testing. Here we teach your team members the techniques mentioned in an understandable and practical way.

We also support you in your day-to-day work by providing your team with an experienced quality engineer who can, for example, prepare exploratory tests and coordinate their implementation.

Would you prefer to tackle the topics yourself, but be guided by an experienced coach?
Then contact us - we will be happy to help you!

Further information by e-mail.