Dendogram-Based Testing

Friday afternoon I was looking through the latest tweets when my eye was caught by the phrase Dendogram-Based Testing. I like all words that have a Greek origin and sound like science, so I had a closer look, and of course it was James Bach introducing a new concept. One that - as far as I understand - is still pretty much missing a definition. No reason to let a small detail like that stop you, right?

After reading up on dendograms I realized that I have actually used them before, but didn't know they were called dendograms. A dendogram is basically a way to take data points and cluster them based on their properties. They are commonly used in computational biology, and that's were I encountered them. About a year ago I spent a week of my vacation making dendograms from genome data.

In testing, one way to use dendograms would be to cluster defects. In order to do this you would need to define a set of properties for each defect, and based on these properties it would be possible to calculate distances between the defects and cluster them in a dendogram. I will save the discussion on whether this is useful or not for later.

Example: The android game SuperTester

To the best of my knowledge this game does not exist, but if it did it would be a game in which the player has to find critical bugs in imaginary applications under time pressure. Just for the record, I haven't thought too much about this so I'm just making it up as I go along.

Let's say five bugs have been found when testing the actual SuperTester game:

1. Can't save game (D1)
2. Can't change sound volume (D2)
3. It is possible to register the same bug twice (D3)
4. Application crahses if you find exactly 13 bugs (D4)
5. Application crashes if you play for more than 59 minutes and 59 seconds (D5)

Now we need to assign these defects properties in order to cluster them. This is the tricky part and requires some careful thinking, Which properties you pick decide what information you will get out of the dendogram. For now I'm just going to pick two properties for the sake of creating an example,

1. Frequency of occurrence on a scale 1-10, where 1 is rarely and 10 often (P1)
2. Severity of defect on a scale 1-10, where 1 is not severe and 10 is very severe (P2)

Time to make a table:

Defects with assigned properties P1 and P2.

Ok, the values might not make so much sense but let's ignore that for now. We have everything we need to calculate the distances. Note that all this assumes that the properties are numeric, if you have other properties such as "red" or "green" you need to decide how to calculate the distance between "red" and "green". For numbers we use the Euclidean distance. I'm not going to go through all the boring details, but the distance table will look like:

Distance table.

D2 and D4 are closer to each other - that is, more similar - than any other defects. Hence we cluster them in cluster A. And so it goes on. What we end up with is the following dendogram:

Dendogram. Defects are clustered based properties.

That's it! Two important points here are i) you need a tool because you definitely do NOT want to do this by hand and ii) how about non-numeric properties, how do you measure distances? I definitely think dendograms can be useful, but a tool needs to be found and then there must be some thought on which properties to use - what kind of information do we want from the dendogram?

This post is just a collection of initial thoughts and is focused on what a dendogram is. I will now crawl back under the rock I came from and think more about how to actually use it for testing.

Craving conferences

On beautiful summer days like these, I have to admit that the main thing on my mind is being out in the sunshine, preferably eating ice-cream! However, I am also starting to feel keen ongoing to another test conference. Luckily, the late summer and fall look very promising.

For starters there is CAST 2011 which is held in Seattle by the Association for Software Testing in early August. The theme is "Context-Driven Testing", and I am of course going together with some colleagues - how could I resist the opportunity to hear about, and discuss, context-driven testing with like-minded people?  It will be a blast!

Then I am proud to announce that I am speaking together with Michael Albrecht at the STARWEST conference, October 2-7, in Anaheim, California. I am especially happy to be able to provide North America with some Swedish test thinking. Why don't you join us? Register using special promo code SKWS and save up to $600 if you register by Super Early Bird August 5th! Click here to register online.

Like most people around the office, I cannot wait for my vacation to start, but I am really looking forward to coming back to an exciting fall!

Sometimes what tastes like mold actually is mold

Last night I enjoyed a test-oriented after work with Simon Morley, Oscar Cosmo and Daniel Berggren. Quite a bit of time was actually spent on group dynamics and the value of working in a team where the members have mixed backgrounds and experiences. We also talked about the importance of having opportunities to discuss testing outside of your team in order to get fresh ideas as well as feedback on your own ideas.

Then came the mold discussion. We got on to the subject of gut feeling – sometimes you just know something is not quite right but you do not have any hard evidence. Or you have a weird incident that only happened once, and cannot be reproduced, but you know that it is important and should be investigated, and still no one can be bothered.

Recently I was in Portugal on vacation, eating good food and drinking great wines. One evening I ordered a piece of blueberry pie for dessert. The pie arrived, beautifully covered in blueberries and nicely presented on the plate, and I dug in. It tasted a bit funny though, sort of like…mold. But I was in a nice restaurant, recommended by a local whom I was having dinner with, the main course had been fantastic and the slice of pie looked delicious so of course I kept eating it even though I could not get rid of that nagging feeling that something was not quite right.

Finally, when I only had a piece of crust left, my eye was caught by something bluish and fuzzy. Of course the crust was moldy! Probably the entire pie bottom had been moldy, and I had eaten it all up. Not once had I stopped to question if that funny flavour really should be present, nor had I stopped to examine the pie more carefully. I was fooled by the fact that I had been told that it was a good restaurant, and that the pie looked good.

This happens in testing too. You might be testing a third party product that ‘is known to be stable’, or asked to ‘just have a quick look because we know nothing has changed’. And it looks so good! But still, deep down, you know something is wrong. Trust your gut feeling, be courageous and be persistent. Sometimes what tastes like mold actually is mold.

…oh, I survived the pie just fine, no unpleasant after effects. Still, I learned my lesson.

The return of the context-driven physicist

I signed up for CAST 2011 as soon as registration opened since Henrik Andersson had told me it was The Conference To Attend. I didn't think too much about the topic - Context-Driven Testing - until I realised "everyone" was discussing whether they were context-driven testers or not. It's even the current poll on the AST homepage! It was time to do some fact finding followed be serious thinking - am I a context-driven tester?

The physicist in me (still going strong three years down the road) is nonplussed. It has never really occurred to me that it is possible to not be context-driven. Physicists are trained to be context-driven. In physics there is no such thing as one theory or formula that applies under all circumstances - on the contrary everything is highly context dependent. Like speed - when objects move fast enough relativistic effects have to be taken into account.

The (somewhat more quiet) statistician that also lurks at my inner core agrees. In order to interpret your data you have to know the context. Without context you can't know whether the data is best described by the standard distribution, or maybe the chi-square distribution.

As a scientist my approach is to first evaluate the context and then try to find the most suitable technique for solving the problem.  You won't get far if you have a favourite formula that you insist on always using. Nature has no intention to adapt to you.

I did not give TBTM a try because I thought it was cool (it is) and it would make a good blog post (it did), but because I thought it would suit my context. My focus is always on solving the task at hand, not on the methodology or technique. I do try to learn as many techniques and methodologies as possible, but not to have a nice CV but rather to really be able to be context-driven. If my tool-box only contains one tool it’s darn difficult to adapt to circumstances. 

I am a context-driven tester and proud of it.

Going to the extreme - xBTM

Now that the project has finished it is time to sum up my experiences of adapting Thread-Based Test Management (TBTM). Since I generally do not believe in rigorously adhering to a protocol, I ended up not using TBTM strictly, but instead embraced a hybrid of Session-Based Test Management (SBTM) and TBTM. Naturally this hybrid will be denoted xBTM. 

The project

In order for this text to make sense, a few words on the project are needed. The customer runs an application that generates output which is stored daily in a single XML file, and later used by the customer’s own applications to derive vital business statistics. There were defects that needed correction, and the customer also asked for new features regarding how log on/log off was recorded. The corrections as well as the new requirements should be implemented in a new file that was supposed to be generated in parallel with the old file for a transition period. 

The team

It was a small project. I acted as project manager, test manager and tester. Later in the project I was joined by a second tester. There was one single developer. 

TBTM

Most of the time my working environment in general is simply too hectic and borderline chaotic for SBTM to be suitable. There tends to be a lot on interruptions and distractions, and it is rarely possible to sit down and focus on a single test task for a given time period. Therefore I decided to give TBTM a try when a new project started.

My first step was to make a mind map containing all my test ideas as threads. Since I am very fond of open source products I used the software FreeMind. In the mind map below, e.g. File Name is a thread. XML file is the actual product. ID denotes a defect, and CR denotes a new change request. The test threads are grouped in two ways. Most groups represent key areas, e.g. Generation which means file generation. Other groups are formed based on the type of testing, e.g. Stress testing. In some cases I felt short notes were needed to explain the thread, and in those cases I attached text files – knots – to the thread. These files are shown as red arrows in the mind map. You click on the arrow to open the text file.

Test plan.

I tried using colours and icons to make the mind map easier to read. The warning signs mark key areas that I judged to be high risk and especially important. The stop light marks a thread that was tied off. That particular change request was retracted by the customer just before the project started. The initial mind map as it was before I started testing made up my test plan and was sent to the customer.

During the test period I would constantly be updating the mind map and it would always give me an accurate picture of the current status of the testing. As soon as I started working on a thread I would mark it with a smiley, see image below. Threads where I found defects were marked with a red cross, and threads where I felt sufficient testing for delivery had been done (not the same thing as claiming to be done testing!) were checked off in green.

Snapshot of mind map in the middle of the test period.

 

Initially my goal was to write daily status reports containing a few short notes on what I had done. In reality I only kept this up for four days. With my constantly updated mind map and the occasional session report (see below) I really did not feel a need for it.

When there was no more time for testing, I took the current status of my mind map and used it as my test report, see below. This test report was sent to the customer.

Test report.

 

SBTM

I do still like SBTM and I find the session reports as well as the time-boxing very useful, so when circumstances would allow, I created test charters and ran time-boxed test sessions. Typically a couple of threads would make one session, but in come cases one thread deserved a session of its own. For example, looking at the test plan the two threads File name and File header belonging to the key area File generation would be tested in the same session, whereas Install from scratch and Install upgrade belonging to the key area Installation were tested in separate sessions.

In most cases I would draw a simple activity diagram – pattern – for each test charter, see example below. I prefer using yEd for my diagrams.

Test charter.

 I wrote short session reports according to a new AddQ session report template, see image below.

SBTM session report template.

Using the AddQ tool SBTExecute I could then derive metrics such as Total session time (for all sessions), Number of test charters, Number of test charters per key area, and so on from my session reports. However, since I was still experimenting in this project, as well as working on the template and the tool as I went along, I could not obtain any useful statistics. Also note than any metrics derived would only be valid for my SBTM sessions – and a considerable part of the testing was done according to TBTM.

Summary report generated by SBTExecute.

Summary and Conclusions

What is the main difference to previous projects I have worked on? Well...the first thing that comes to mind is that this time I actually used the test plan! And I have read and come back to the test report. Keeping the mind map up to date has been much easier than updating any other kind of status document, which means that it actually has been updated. As it turned out, even the developers liked it – they preferred looking at the mind map over using our bug tracking tool!

I immediately liked the combination of TBTM and SBTM – it is a simple matter of applying the methodology that is best suited for the task at hand. Some threads were fairly isolated and straight forward to test, making them suitable for SBTM. Other threads were very convoluted and had to be explored together with the developer, making them better suited for TBTM.