Maaret Pyhäjärvi

Experience report on taking notes and reporting
During my years as tester, I’ve been fortunate to work with organizations that value the contribution of smart individuals and allow for the thinking space intellectual work requires. I would call that Exploratory Testing, although we tend to talk of just Testing. I’ve been allowed to experiment and try out approaches and tools that would help me and my colleagues to keep the complexities of testing under control, without a lot of external requirements on what and how to document. Most of the documentation pressures, for me, rise from within, from my need to recall things that happened, and to make sense of the overall picture of what I’ve learned in many small steps, allowing others to get some of the relevant information faster.

Two years ago, I joined a new project as the only tester in a team of developers. The team had worked hard on getting their testing to take form with the lead of the project manager. As there was no specialized tester, they had organized to create test cases for the developers’ ideas, and asked another developer to help out in executing those tests. The experience was that without the written test cases, testing would not happen. The fear and distrust was clearly a driving force, so instead of me asking to trust me to do things differently, I asked if it was ok to document the testing I did in detail but a different style than test cases – using a notetaking tool Rapid Reporter, created by Shmuel Gershon (http://testing.gershon.info/reporter/).

Going for the Rapid Reporter
I introduced the tools with a demo and a picture explaining the central concepts (picture 1 – next page) of what I was about to do to get a go ahead to not do things as before.

Rapid Reporter is a small standalone tool to create tagged notes with timestamps. The tool itself sits on top of my open applications, being readily available as I proceed with testing a web application in a Windows environment. It allows me to make notes of my latest information, classify it with predefined tags, and later go through whatever notes I created, in case I would like to take the most recent information into some other form of documentation. My notes get saved as per session csv-files and I place them in a folder structure outlining the areas of the application I’m testing.

I have a set of tags I use to classify my notes with Rapid Reporter with a predefined meaning. For my set of tags to use, I’ve extended Rapid Reporter giving it my set of tags on a command line. Some tags I use for calculating use of my time (Test /Bug /Setup /OffCharter). Some tags I use to count amount of results from Testing (Bug /Question /NextTime). Some tags are there to make sense for assessing coverage (Environment /Data). And some just so that I would remember the bits I really consider relevant to use later on (Note).

With the tags and their meanings to how I use the tool, it makes sense to use a Rapid Reporter csv-scanning tool SBTM created by Ru Cindrea, to automatically create reports in Excel that show the division of time realized allowing me to explain if my testing is progressing and sums up the amount of results tags.

I also outlined, that I would collect the NextTime -ideas on a backlog (which we kept in Jira), to show how the amount of work remaining is developing as testing proceeds. And that if there was other useful documentation (e.g. specifications I would write learning by testing as the project did not have such), I would collect that information in our wiki, not as test cases but more as things to enhance someone else trying to learn the areas I was testing.

Learning by Doing
I progressed with testing using Rapid Reporter, and doing also a lot of OffCharter work to learn the application I was new to. I would read a bunch of Jira issues with newer information overwriting the older to learn what the application was supposed to do as there was no up-to-date specification. I would read the existing test cases to learn that it’s possible to have 3 tiny pieces of relevant information in 39 pages of test cases. And I would use the software to see how it works, and log bugs.

As days passed and testing continued, taking notes of time allocation was relevant in particular. Whenever I use tags Test, Bug, Setup and OffCharter, I intend not only to write a note that is related to the tag, but also mark down the time of when I did that. This allows me to see how long I was working between the last Test-tag and another tag that has impact on time calculations. With the information available, I could mine out for a report the time I used for Test (I’m actually covering the software and making progress), Bug (I found something that I’m investigating), Setup (I’m getting ready to do testing) and OffCharter (I got interrupted with something other than I was planning to do in the session). Picture 2 provides an example of the automatically generated information I could have available from Rapid Reporter csv-files.

The tool helped me make it visible that with the amount of bugs I was logging from the testing, the time used went in a significant portion to isolating and logging the issues, and I wasn’t making much progress on covering features of the application, as I needed to come back to everything after it had been fixed. Then again, just the amounts of bugs in Jira I would log were a surprise and created similar visibility as the problems had not been seen or isolated with the testing done before my time with the organization.

As time passed, I got better at writing down things I would be interested in for the future. I would not write down everything I do while testing, but always consider the future value of what I would write. Does it help me understand what I’ve covered? Does it help me recall what I’ve learned so that I can build on that? Categorizing the notes for coverage areas (folders for the csv-files to reside on) also helped me create a map of how to go find the information.

I also started paying attention to where the effort and my focus went while using the tool. While this style of documenting was not particularly painful, it created a structure that did not necessarily allow me the full freedom that trust in my focus and skills gives. A month with Rapid Reporter ended up being a proof of concept of my trustworthiness, and as I suggested to not use that anymore based on opportunity cost (time I could use on producing more value than on the notes that make me trustworthy), I no longer needed to address the questions of not doing testing without marking test cases done.

Lessons learned while using the tool
I found the tool very valuable as an option to making notes. While it’s a great and useful tool, there were many things I learned to tweak on how I would use it so that the tool would serve the purposes I had in mind – including reporting – and not get in the way of how I think while I test.

Some advice I have marked down for my future self on this that could be useful for others taking the tool into use:
— I found it useful to fine-tune my message on the time calculation tag types with Note-tagged things following the original Test/Bug/Setup/OffCharter – tag. For example, when I was working to understand a problem, I often marked down a quick Bug-tagged note to get the time to be calculated under Bugs, and I might work to isolate the problem for a long time after that, either using Note-tagged notes or more often, writing steps to repro in Jira.

— You can take screen shots with the tool. When you use Shift-button together with clicking the camera icon, you can edit and mark your notes directly on the picture. I found it useful to take a screen short right after a Bug-tagged note, because the screenshot shows your most recent note and it made it easier to make sense of my notes and pictures later on this way.

— If a brief Bug-tagged note helps you remember to come back to isolate the bug later, I found it useful to not isolate the bug right in that session but use different sessions on isolating the bug to be able to progress with my original idea of what I would like to know after the session. Often I learned, however, that my brief note was not enough to recall what had happened, and finding the steps to reproduce would have been easier if I did not postpone it. The notes helped me learn the types of things where I would log the bug immediately and the types I could postpone.

— While the session is often suggested to be a focused timeframe, I found it useful to consider a day of work focused enough for our purposes. My sessions were significantly different in sizes, ranging from the full day (skipped lunch and breaks for the fun of testing) to times between the breaks or meetings. I learned that my typical day would have 2-3 1,5-hour focused sessions and a lot of asking around to give and get information that was relevant for the testing.

— I used the OffCharter – tag a lot to mark time on testing something that I originally did not think would be the theme of the day. Towards the end of experimenting with the notes, I became less disciplined with my time keeping, as I felt it took quite much energy.

— One set of notes is saved on one csv-file, and one csv-file can be placed in one folder representing an area of coverage. Whenever moving from a coverage area to another, I had the habit of starting a new session just for coverage and time tracking to make sense.

— I found it easier to assess what I had covered than to plan what I would cover. The time allocation isn’t precise but it did not need to be, it gave direction on what areas had had the time allocation they would deserve as per their complexity. What I looked at and pointed out mostly is that if no time on an area is used, it does not mean it works since there are no known issues, but that if there are issues, we just don’t know of them

— The folder structure allowed with the scanning tool I used had one level of scanning depth, so no subfolders allowed. When I tested the full product, my folders were areas of the product. When I tested one area, I had a separate folder structure with division to folders that made sense in that area.

— I also had sessions that were not about any of the coverage areas I had identified but touched all around the areas. Those I saved on the root folder I would create reports from.

What then?
With nearly two years in the company, I work closer to the developers than the idea was at time of me joining. My team knows I love my testing work, and they see a part of my results in the bugs they get to fix. The concerns of sharing information and helping others know what happens in testing will not be resolved in writing more documentation of any form that people don’t read, so instead we discuss more of what and why I do, allowing smart individuals to deduct the details.

We’ve learned not to create 39 pages of test documentation with 3 mention-worthy details, and instead discuss around mindmaps and checklists. I still occasionally remind my team members that while the 39 pages had only 3 things worth the paper, there was still 98 features we identified on that area alone, together, that are now listed in a checklist. Running the test cases was a cause for missing all the bugs I have helped to find since then, that we can find when we don’t box testing with appearance of test cases that just lowers the quality of the work resulting from running them.

Instead of making notes of the details of what we do while testing, we tend to focus more on discussing what more is there to test – what ideas we haven’t covered. And making a map of the areas available that help in recalling what there is to remember on a higher level.
Rapid Reporter – or any other note taking tools for testing that focus on categorizing things as you learn them – are great and useful tools. But the core of it is the thinking tester. Use the tools in smart fashion.

Learn more than the tool
The tool was just a cherry on the ice cream – just what I needed to make things flow. But the basis of using the tool comes from the lessons over the years from James and Jon Bach. I would suggest reading up on session-based test management (SBTM) for ideas of one approach to manage exploratory testing where Rapid Reporter fits in nicely.
More recently, I’ve enjoyed using a mobile application adaptation of the Rapid Reporter ideas I describe here and you might want to check iTester for iOS – especially if Siri (the voice recognition technology) actually hears you correctly and allows you to skip much of the writing by talking.

Have questions on the tool? Leave a comment below.