28 February, 2011

Going to the extreme - xBTM


Now that the project has finished it is time to sum up my experiences of adapting Thread-Based Test Management (TBTM). Since I generally do not believe in rigorously adhering to a protocol, I ended up not using TBTM strictly, but instead embraced a hybrid of Session-Based Test Management (SBTM) and TBTM. Naturally this hybrid will be denoted xBTM. 

The project
In order for this text to make sense, a few words on the project are needed. The customer runs an application that generates output which is stored daily in a single XML file, and later used by the customer’s own applications to derive vital business statistics. There were defects that needed correction, and the customer also asked for new features regarding how log on/log off was recorded. The corrections as well as the new requirements should be implemented in a new file that was supposed to be generated in parallel with the old file for a transition period. 

The team
It was a small project. I acted as project manager, test manager and tester. Later in the project I was joined by a second tester. There was one single developer. 

TBTM
Most of the time my working environment in general is simply too hectic and borderline chaotic for SBTM to be suitable. There tends to be a lot on interruptions and distractions, and it is rarely possible to sit down and focus on a single test task for a given time period. Therefore I decided to give TBTM a try when a new project started.

My first step was to make a mind map containing all my test ideas as threads. Since I am very fond of open source products I used the software FreeMind. In the mind map below, e.g. File Name is a thread. XML file is the actual product. ID denotes a defect, and CR denotes a new change request. The test threads are grouped in two ways. Most groups represent key areas, e.g. Generation which means file generation. Other groups are formed based on the type of testing, e.g. Stress testing. In some cases I felt short notes were needed to explain the thread, and in those cases I attached text files – knots – to the thread. These files are shown as red arrows in the mind map. You click on the arrow to open the text file.

Test plan.


I tried using colours and icons to make the mind map easier to read. The warning signs mark key areas that I judged to be high risk and especially important. The stop light marks a thread that was tied off. That particular change request was retracted by the customer just before the project started. The initial mind map as it was before I started testing made up my test plan and was sent to the customer.

During the test period I would constantly be updating the mind map and it would always give me an accurate picture of the current status of the testing. As soon as I started working on a thread I would mark it with a smiley, see image below. Threads where I found defects were marked with a red cross, and threads where I felt sufficient testing for delivery had been done (not the same thing as claiming to be done testing!) were checked off in green.

Snapshot of mind map in the middle of the test period.
 
Initially my goal was to write daily status reports containing a few short notes on what I had done. In reality I only kept this up for four days. With my constantly updated mind map and the occasional session report (see below) I really did not feel a need for it.

When there was no more time for testing, I took the current status of my mind map and used it as my test report, see below. This test report was sent to the customer.


Test report.
 
SBTM
I do still like SBTM and I find the session reports as well as the time-boxing very useful, so when circumstances would allow, I created test charters and ran time-boxed test sessions. Typically a couple of threads would make one session, but in come cases one thread deserved a session of its own. For example, looking at the test plan the two threads File name and File header belonging to the key area File generation would be tested in the same session, whereas Install from scratch and Install upgrade belonging to the key area Installation were tested in separate sessions.

In most cases I would draw a simple activity diagram – pattern – for each test charter, see example below. I prefer using yEd for my diagrams.

Test charter.
 I wrote short session reports according to a new AddQ session report template, see image below.

SBTM session report template.
Using the AddQ tool SBTExecute I could then derive metrics such as Total session time (for all sessions), Number of test charters, Number of test charters per key area, and so on from my session reports. However, since I was still experimenting in this project, as well as working on the template and the tool as I went along, I could not obtain any useful statistics. Also note than any metrics derived would only be valid for my SBTM sessions – and a considerable part of the testing was done according to TBTM.

Summary report generated by SBTExecute.

Summary and Conclusions
What is the main difference to previous projects I have worked on? Well...the first thing that comes to mind is that this time I actually used the test plan! And I have read and come back to the test report. Keeping the mind map up to date has been much easier than updating any other kind of status document, which means that it actually has been updated. As it turned out, even the developers liked it – they preferred looking at the mind map over using our bug tracking tool!

I immediately liked the combination of TBTM and SBTM – it is a simple matter of applying the methodology that is best suited for the task at hand. Some threads were fairly isolated and straight forward to test, making them suitable for SBTM. Other threads were very convoluted and had to be explored together with the developer, making them better suited for TBTM.






11 comments:

  1. I like how you used color coding and icons on the mind map - great visual feedback. We already use mind maps for test planning, will try this new twist on it, thanks!

    ReplyDelete
  2. Reply to Lisa: The colours and icons made all the difference, but I soon realised that FreeMind was a bit limited in that area so these days I use XMind instead. Can't belive I haven't always used mind maps for test plans!

    ReplyDelete
  3. Great work, Christin! I really like the visualisation of the test result in the mindmap. It makes it easy for people outside the test group to get an overview of what is tested and what is untested, which areas that have defects and which have not. Excel sheets and activity lists are good in many ways but I always felt that there is something missing in them to get the overview and the complete picture. I will try to incorporate more mindmaps in my reports and planning in the future.

    Have you thought about ways to give different bransches in the mindmap different "weights"? E.g. This bransch is more important to the customer than that one, or this branch takes much longer time to test than that one, etc.

    ReplyDelete
  4. Christin,

    Bravo! What a fantastic experience report and very educational as well!

    As you probably know I'm very into mind maps, however I'd never actually considered ditching a written test plan before and incorporating it into a mind map. Great idea! I'd probably do mine slightly different to reflect more the key aspects we try to communicate in our test plans here in my workplace.

    So you've written, but never actually used a test plan before? Me too! I've been banging on about the waste these incur for a long time now. We looked at making our plans more lean, and we did indeed cut out the fluff, however I think I'll follow your idea and switch over to a mind map and see how the team and recipients like it.

    Fantastic post, there is a lot to be learned here. You are one of the few bloggers I get excited about when I see new content has been posted.

    Keep up the good work!

    Cheers,

    Darren.

    ReplyDelete
  5. You might want to add that you also incorporated another idea on test reporting in your mind. The way you organized the test plan and the test report, you created a testing dashboard based on the mindmap. This became obvious to me when you pointed out the fact, that your programmers liked it. Nice experience report.

    ReplyDelete
  6. Hi Christin,

    This post is a great help as I am looking at moving to mind maps for test case writing and execution, and moving away from complicated tools such as Quality Center. We have hundreds of manual test cases and keeping them up to date is proving to be time consuming, and not very agile.

    My team in general is excited to try this, but there are some concerns that we have and some potential solutions:

    A) How do we report results. I'm looking at using Test Dashboards such as this http://www.satisfice.com/presentations/dashboard.pdf

    B) If my manager asks how far the progress is during a test run, how would I do that? I have a feeling showing the mind map should be sufficient. I simply have to go ask them to see.

    C) How can multiple testers work on testing at the same time? I'm looking at having several smaller maps that users can "check out" of our source control, run and then "check in" when done.

    ReplyDelete
  7. Reply to OutsideTheBlackBox: You don't have to be radical and immediately replace your usual test plan with a mind map, start by making it a complement.

    In other projects (where I'm not almighty :)) we still have Excel test plans, but now we also have mind maps, and it turns out that the mind maps are what we actually use and update - the Excel files are created, sent to the customer and never opened again. Sort of at least.

    Regarding weights, I do actually give different groups/branches different priorities/weights now that I'm using XMind instead of FreeMind. XMind has (I think) a better selection of icons. In my (Swedish) blog post http://www.addq.se/testmetoder/xbtm-det-basta-av-tva-varldar/ I have used number icons for priority.

    ReplyDelete
  8. Reply to Darren: What can I say - just happy you're reading!

    Another interesting thing with a mind map test plan is that it tends to generate more ideas. Not only while making it, but every time I look at it. The visual overview makes me more creative.

    In the old days I would pretty much stick to the test plan and new items were rarely added once testing had begun. The mind map I look at and update at least once a day, and quite often just looking at it gives me new ideas that I add as threads. Maybe I'll test them, maybe I won't!

    ReplyDelete
  9. Reply to Markus: You're absolutely right, it is a testing dashboard.

    What I was after originally was the idea of multi-threading, but once I had the mind map I think the benefits of the testing dashboard is what really caught my attention.

    ReplyDelete
  10. Reply to nmac: Happy to be of assistance :)

    A) Well, as Markus pointed out in his comment - you already have the testing dashboards. I update my mind map every day with icons and colours and I think that is a good way to report progress.

    B) No one can resists smiley faces :) Seriously though, I know very well that management can be hard to deal with but I see it as my responsibility as a tester to teach management what good measures of progress are. If they are not testers they don't know. I probably (hopefully) have a better idea, and I'm more than happy to tell them.

    A mind map that changes daily is a good measure of progress. If more and more threads get an icon that is a sign of progress, regardless of whether they get a red cross (nok) or a green check (ok).

    If you manage to pull off some SBTM sessions you can also derive metrics/statistics from those if that makes your management happy. Just make sure any metrics or statistics you come up with are valid and useful - from y
    our perspective too.

    We usually have one master mind map that is stored in a repository with version control, SharePoint normally. We mostly work in the same lab so sometimes I project the mind map on a whiteboard. That way everyone can doodle on it, and at the end of the day the doodles are transferred to the mind map by one person who then also checks it in.

    I think some of the benefits of mind maps might be lost if you make sub maps.

    Let me know how you do!

    ReplyDelete
  11. Great post Christin. Thanks for sharing.

    ReplyDelete