Category Archives: Tester Merit Badges

The status is not quo

09 Friday Mar 2012

Posted by in Context, Experiences, Experiments, Hackathon, Retrospective, Tester Merit Badges

≈ 3 Comments

Dr. Horrible http://drhorrible.com/

We tend to run “FedEx” with a fairly open format where you can do whatever you want as long as you can somehow relate it to our products.
– Atlassian

Last week, my company gave us an exciting opportunity: 5 days of work on a project related to our business.

Apparently, they’ve done something like this before, long before my time, so you’d have to ask some of the more tenured folks at Daxko about it.

I worked with the same folks who volunteered with me at the WebVisions Hackathon earlier this year and we kept in mind what my colleague Will said about that experience: “The short time box and no feature constraints necessitated a laser-sharp focus on one thing.”

So we noodled over several viable candidates and finally settled on building a better mousetrap – or, in this case, UsabLog.

A clarification on terminology from my UX colleague:
“Logging” in this context doesn’t mean “system logging of events.” It means human capture of what the user said, what the user did in the app (e.g., where user clicked), and any additional comments to provide context. The point of logging is to provide us with a record of what went down so we have an accurate recollection for later analysis.

I had the good fortune to be a user of the original UsabLog application over the course of many usability sessions as a session logger, so I was rather familiar with its strengths and weaknesses. I was able to contribute some bug reports and feature suggestions for consideration during our lunchtime planning discussions, but my Scrum team’s UX designer was our team’s sponsor. She compiled an experiment plan that identified our purpose and detailed the problems we considered in the pre-existing Usablog and the opportunities we had to satisfy those needs.

Our usability sessions up to this point involved an interview led by the facilitator (i.e. UX designer) and logged by another team member (e.g. me) via the free, open source, web application Usablog, which then exported logs to CSV for use in a program such as Excel and which we in turn manually fed into a mindmap program such as FreeMind. While this process did work for us, the export and manual copy-paste was rather tedious and laborious, or as she put it “it would directly contribute to user research process efficiencies.” We knew there could be a better way.

Goals of the experiment:

  • Rapidly capture rich user feedback during research interviews and usability tests through logging of user events and comments
  • Organize logs from multiple sessions into one study for ease of access and visibility
  • Use log entries to synthesize findings
  • Quickly jump to a spot in the session’s video by clicking on the associated log entry

In particular, we wanted these features:

  • Multi-session logging.
  • Log entries are timestamped when the logger starts typing for video synchronization.
  • Custom tags.
  • Multi-logger logging.
  • One tool for logging and post-session analysis.

We established a definition of done and recognized our dependencies since any impediments would have serious impact on our progress during the limited time of the competition.

I would love to tell you that we were entirely successful in meeting our goals and implementing all of our features, and then going on to take first prize in the competition. Alas, this was not to be. We only accomplished some of our goals and features and awesome projects from other teams placed above us.

However, the experiment was a roaring success in many ways:

  • I had first-hand experience with paired UX design under the tutelage of my UX designer colleague. She suggested that I man the helm and she steered me back on course when I went astray. I won’t claim that my first UI mockups were beauties, but the process and conversation certainly were.
  • I made my first commit to a Github open-source repository and thereby qualify for the Open Source Nerd Merit Badge (which happens to feature the Github mascot Octocat) which I had been hankering to do ever since I discovered its existence. Also, this was the first time I fixed a bug in the source code, so even though my changes were minor it was thrilling.
  • Exploratory testing based on Github commit notifications in the HipChat chat room we used for the team. Rather than pursuing session-based test management, I tried a looser structure based around the latest and greatest changes instead of setting charters and time-boxing exploration around the stated goal.
  • Real-time bug reporting of issues found during exploratory testing via HipChat messages and screenshot attachments was new and interesting. This is the lowest overhead asynchronous bug management approach I’ve tried and it was effective. Granted, we didn’t come out with a backlog of known issues written down somewhere, but we rectified the most critical problems before they had a chance to fester.
  • We didn’t let a little thing like heading home for the day stop us from collaborating remotely when we got back to business after hours. Being able to work at odd hours put some of my insomnia to good use. I also learned a bit about .NET and model/view/controller architecture, which turned out to be good preparation for the following – and last – day.
  • When one of our programmer teammates fell ill, I paired with our remaining developer to push on toward the goal. Although I think I spent more time asking questions to help think through the implementation than actually contributing code, it was a fruitful day, wrapping up an important feature a mere 30 minutes before the Big Reveal.
  • I used the resulting product to real-time log the presentations during the Big Reveal. Oh so meta, but also hopefully illustrative of the capabilities of the application for future use. If nothing else, it gave our sick friend a way to catch up on the excitement as he recovered over the weekend.
  • We accomplished only some of our goals and features but they were the most essential. Our product is usable as-is, though with some known bugs that do not inhibit happy-path use.
  • Why do they call it FedEx days? Because you have to ship! Our resulting application is ready for use – or enhancement if you’re feeling ambitious!
  • And last, but certainly not least, victory lunch! Nothing so sweet as celebrating effective teamwork.

Image source

February results for Tester Merit Badge

09 Friday Mar 2012

Posted by in Approaches, Experiments, Techwell, Tester Merit Badges

≈ Leave a Comment

Tester Merit Badge - Explorer

My February was pretty crazy, so I didn’t pursue all of the parts of the Explorer Tester Merit Badge this month. I skipped these parts due to time constraints:

3. How Long and How Far
4. Walk the Distance
8. Bus and Train Maps

No sweat. I’ll certainly be coming back to this one since exploratory testing (ET) is meat-and-potatoes testing for me.

Read the rest on the Techwell blog and tell me about your results!

Tester Merit Badges live with 1 week to go for February!

22 Wednesday Feb 2012

Posted by in Approaches, Experiments, Exploratory Testing, Techwell, Tester Merit Badges, Training

≈ 1 Comment

With only a week left in February, I’ll be wrapping up my own trial of the first Tester Merit Badge and posting my results, so I encourage you to try it for yourself. Do let me know if you’re playing along at home!

For reference, here is the introduction and here is the motivation.

Originally posted on the Techwell blog, the badge description is reposted here for convenience:

Tester Merit Badge - Explorer

Tester Merit Badge: Explorer

Skill challenge: Try exploratory testing
Requirements
1. Know Your Maps.

Be able to explain different approaches to exploratory testing (e.g. testing tours, chartered session-based testing, freestyle)

Some web resources:
Session-based testing
Scenario testing
Exploratory testing in agile

Some books:
Crispin, Lisa; Gregory, Janet (2009). Agile Testing
Whittaker, James (2009). Exploratory Software Testing

2. North, South, East, West.

Who says scripted test cases can’t be exploratory? Just because you have a protocol written down doesn’t mean your brain turned off when you began to execute it. As you go along working through a set of instructions, perhaps drawing from a user manual if you lack test scripts or specific test cases, keep your eyes open for what is going on around you, not just what fits the happy path of the case. Make notes of testing ideas and chase down something that’s off-script and keep track of what you do as you go along. If no test scripts or highly structured test cases exist for a particular aspect you want to test, skip ahead to requirement #3.

3. How Long and How Far.

Using an existing reference (if any) estimate the time to exploratory test an aspect of a feature of a software product. If you have been using test cases or test scripts for this testing, then use those as jumping off points for your exploration but don’t follow them. I tend to make a list of some test ideas and then pick and choose which ones to attempt during a particular exploratory session. If no ready-made resources exist for a particular aspect you want to test, skip ahead to requirement #4.

4. Walk the Distance.

Estimate time exploratory test an aspect of a feature of a software product without the aid of any existing materials. Use your knowledge and experience to take an educated guess at what needs to be done and build up a guess-timate. We’re going for ballpark and not precision here. Then, try it and see how close you were. That’s the beauty of iterative learning.

5. Map Maker. Map of the Place. Make a Model.

I see these physical representations of travel as essentially the same when it comes to software testing. This one is about precisely describing what you observe, which makes it a perfect artifact of your exploratory session. This could be a site map, a pairing of requirements description snippets with implemented user interface components, or even a sketch of different paths through a block of code (if you’re doing some white box preparation for your exploratory testing). We want to show what we observed rather than what we expect, so you can explicitly record your expectations if that helps you to clear your mind for the road ahead.

6. Finding Your Way Without Map or Compass.

Freestyle! Do some testing without any more explicit structure than a time box. Give yourself 15 minutes to wander around in an application without stating a particular agenda. You might even try accessing a piece of software through a less favored access point (e.g. a traditional website viewed from a smart phone).

7. Trail Signs Traffic.

This is an opportunity to write a different kind of test guide for another tester to follow. If we stick to the trail metaphor, you want to provide indications of the way out of the woods but you don’t dictate how the hiker travels between the boles of the trees bearing the blaze (or if you’re a big nature nerd the piles of rocks and sticks with their encoded messages about the trail ahead). I think Whittaker’s landmark tour is particularly apt for this example, so I recommend picking a part of your app to extract some landmarks. Avoid step-by-step instructions about how to wander between these milestones! You want to recognize the variation in execution that naturally occurs, even in the presence of a test script. In this case, doing it differently is a strength since you collectively will cover more of the application over time, although you may not encounter exactly the same scenery along the way.

8. Bus and Train Maps.

Use a publicly available source to map out a test. If you have a user manual for the application under test, that would be a good source for producing an expected route through an application. Just like a driver stuck in traffic, you don’t need to adhere to the planned route, so feel free to follow any detours that seem like better alternatives if you are feeling blocked or just want to take a more scenic route. Lacking a user manual for this particular product, try a description of some similar or competing product. Again, we’re exploring here, so having an inexact guide is no barrier to the experiment.

 

When you complete any or all parts of these badge “requirements” take a moment to reflect on whether the technique could be helpful to you in your regular testing work. You don’t have to migrate away from your current approach, but having some options always helps me to switch it up a bit when testing starts to feel monotonous – and I really think I’m doing it wrong when testing bores me! There is always too much testing to complete, so I certainly need to go exploring more often.

Drop me a line or post a comment here to let me know how the experiment went for you and I’ll post my own results here within the month.

Happy testing!

Image source (embroidered version of this image)

Tester Merit Badges: Finding Your Way

31 Tuesday Jan 2012

Posted by in Experiments, Exploratory Testing, Tester Merit Badges

≈ 4 Comments

Recently, I began blogging over at Techwell with my Active Lifestyle resolution. As a follow up, I am writing a guest series there.

As I have mentioned before, I was a Girl Scout for a while when I was growing up and loved the exposure to new and different things that I wouldn’t have occasion to try in my everyday life as well as the structure around life skills that would later be essential to self-sufficiency. For me, there’s nothing new around the recent enthusiasm to game aspects of our day-to-day lives and, as I’ve blogged before, having structure around learning helps me to progress.

As a grown woman, I again found an interest in earning badges when a former-Girl Scout friend of mine mentioned Lauren Grandcolas’ book You Can Do It!: The Merit Badge Handbook for Grown-Up Girls to me. It’s full of experiments modeled in the style of Girl Scout badges and reduces the potential intimidation of attempting new skill acquisition.

One day, after I had seen a 3D-printed octocat at the Atlanta Mini-Maker Faire at Georgia Tech, I was searching for octocat images online and I discovered the Nerd Merit Badges that reference Github, the movie Office Space, and the book The Pragmatic Programmer among other wonderful and obscure aspects of geekery. These inspired me to apply the badge concept to software testing. After all, developers shouldn’t have all the nerdy fun, though I’m pretty sure I’ve already earned my Family Tech Support and Homonyms badges… (For my word nerds, contrast with homophone here.)

Allow me to preface this experiment by recognizing the interval of time a Girl Scout takes to earn a badge is not a month. Girls fulfill these requirements over time, probably interspersing activities from several badges over the course of many months. I recognize that neither you nor I will necessarily complete all the requirements for each month’s Tester Merit Badge and that’s fine. Checking everything off the list is not the point. We’re here to learn and step outside our comfort zones, so start where you’re comfortable and stretch yourself a bit!

For the inaugural Tester Merit Badge, I have designed the Explorer badge as an introduction to exploratory testing. I am modeling it after the requirements of the Girl Scouts of America’s Finding Your Way badge.

Girl Scout badge Finding Your Way

Girl Scout of America badge: Finding Your Way

Requirements
  1. Know Your Maps. Be able to explain 3 diff. types of maps.
  2. North, South, East, West. Show you know how to use a compass.
  3. How Long and How Far. Use map to determine time to specific place.
  4. Walk the Distance. Estimate time to walk distance and try it.
  5. Map Maker. Draw map of a route with a legend or key.
  6. Map of the Place. Draw map to scale of a specific place.
  7. Make a Model. Make 3 dimensional model.
  8. Finding Your Way Without Map or Compass. Use sun, stars and nature.
  9. Trail Signs Traffic. Use trail signs to set up a mini-trail for others to follow.
  10. Bus and Train Maps. Learn to use bus or train maps. Try your route.

See the first Testing Merit Badge on the Techwell blog!

Image source