Can DevOps afford to ignore cognitive testing?

by | Syndicated

I attended an event on the 23rd of July where DickSmith’s CIO, Paul Keen gave a presentation about his organisation’s experience implementing DevOps. After speaking with him briefly before the event, I would say that he is one of those technically competent CIOs who actually understand and know what their team is working on. Paul appeared to be a reasonably honest person which was also evident in his talk. What I liked most was his trust in his team and his openness about improving the culture of his organisation. To be honest, before speaking with Paul I was skeptical about his knowledge of DevOps. My skepticism is natural, after all I am a tester!
Paul took the audience on the journey of DevOps implementation at DickSmith. If you are not aware, DickSmith is one of the biggest consumer electronics retailers in Australia. During my chat with Paul, he admitted that testing was very important. What I could not find out was whether or not he has any testers in his team. From his presentation I had the impression that his team relies heavily on automation tools. To Paul’s credit, he did admit that the website may not be 100% perfect (which no website can be anyway). Concerned that his team seemed to be performing automated checking without cognitive testing, I decided to demonstrate the benefits of good testing. My intention is not to insult Paul or his team or undermine their ability to deliver, it is just to show what negative effects over-reliance on automation can cause. I had a chat with some of my colleagues, specifically Scott Miles and Paul Crimmins about this and we decided to run few exploratory testing (ET) sessions to test the website. If you believe that ET is unstructured, unplanned, focusless, ad-hoc testing in which a tester just punches keys on a keyboard, you are gravely mistaken. ET is everything but that. BTW, if you think ad-hoc means unstructured and unplanned, then you must look in a dictionary at its meaning. We completed two 45 minute sessions which included setup, planning, learning, note taking, exploring, reproducing bugs and debriefing. We found a lot of bugs by performing exploratory testing on the website. Since it was just for demonstrating the importance of cognitive testing, we could have had more ET sessions. Just imagine what would have happened if we had spent more time in testing the product. Remember, we did not focus on security, payments, accessibility, browser compatibility, mobile app/mobile website and many other important areas of DickSmith’s product. While automated checking has been useful in their DevOps environment to respond to change; it is evident that the lack of cognitive testing has allowed many undetected bugs in the product. Some of those bugs are listed in our bug report below, which I co-authored with Paul Crimmins and Scott Miles. The details of bugs including pictures & videos are available, but we decided not to post them here for all bugs.

Bug Report

Testers: Rajesh, Paul, Scott

Method: Exploratory Testing using heuristics and oracles
Duration: Two sessions, 45 minutes each
Environment: OS: Windows 7, Browser: IE9
Functions focussed: Registration, Login, Search, shopping Cart, checkout, discounts and promotions, store locator
————————————————————————————————————————


Search bugs:
  1. Tool-tips inconsistent
  2. Price filter displays a backslash (\) before dollar sign ($)
  3. After adding items to the card and performing a second search, the cart would sometimes display 0 items in the cart. When selecting another item, the previously added items would again be displayed in the cart.
  4. When the cart has 11 vouchers and the user attempts to add 1 more, the maximum number of vouchers is reached and all vouchers are removed instead of remaining at 11
  5. Videos displayed had no visible close button
  6. No character restriction on price filter when using “right-click paste”


Checkout bugs:
  1. The most annoying bug: Apple iTunes coupon does not go away after adding to cart.
  2. After adding the suggested voucher to cart, if
    your other items were store pickup only you could no longer check out.
  3. Applying wrong discount coupon hangs the site
  4. Going back to checkout page causes “apply discount” toggle to behave in the wrong direction


Registration bugs:
  1. Terms and Conditions open in same window and clears your password when you press back
  2. The website removes whitespaces at the beginning or end of a password, but this is not indicated. There was much confusion over why the website was giving the error that not enough characters were entered when the password field displayed more than the minimum required
  3. The confirm password field briefly displays a green tick for 1 second when the correct password is entered. This then disappears confusing the user.
  4. Since when are spaces and a dot (.) are correct names? The name I used for registration was (      .).
  5. You can use anyone’s email and register them for DickSmith. There is no email verification required.
  6. The registration confirmation email shows no brand images. I tried it on different browsers, on cable network, wi-fi and 3g.


Add Address bugs:
  1. Address fields display a green tick when completed, but fail validation when “save” is pressed
  2. Address fields continue showing “required field” error after text has been entered
  3. Can change state and postcode after selecting from locality service. Doing this, we were able to set our address to a suburb in a state that does not exist with a null postcode. How would you ship items to this address?


Store Locator bugs:

  1. Store Locator gets confused when you enter % sign. Actually % confuses many other areas of functionality.