Can DevOps afford to ignore cognitive testing?

by | Syndicated

I attended an event on the 23rd of July where DickSmith’s CIO, Paul Keen gave a presentation about his organisation’s experience implementing DevOps. After speaking with him briefly before the event, I would say that he is one of those technically competent CIOs who actually understand and know what their team is working on. Paul appeared to be a reasonably honest person which was also evident in his talk. What I liked most was his trust in his team and his openness about improving the culture of his organisation. To be honest, before speaking with Paul I was skeptical about his knowledge of DevOps. My skepticism is natural, after all I am a tester!
Paul took the audience on the journey of DevOps implementation at DickSmith. If you are not aware, DickSmith is one of the biggest consumer electronics retailers in Australia. During my chat with Paul, he admitted that testing was very important. What I could not find out was whether or not he has any testers in his team. From his presentation I had the impression that his team relies heavily on automation tools. To Paul’s credit, he did admit that the website may not be 100% perfect (which no website can be anyway). Concerned that his team seemed to be performing automated checking without cognitive testing, I decided to demonstrate the benefits of good testing. My intention is not to insult Paul or his team or undermine their ability to deliver, it is just to show what negative effects over-reliance on automation can cause. I had a chat with some of my colleagues, specifically Scott Miles and Paul Crimmins about this and we decided to run few exploratory testing (ET) sessions to test the website. If you believe that ET is unstructured, unplanned, focusless, ad-hoc testing in which a tester just punches keys on a keyboard, you are gravely mistaken. ET is everything but that. BTW, if you think ad-hoc means unstructured and unplanned, then you must look in a dictionary at its meaning. We completed two 45 minute sessions which included setup, planning, learning, note taking, exploring, reproducing bugs and debriefing. We found a lot of bugs by performing exploratory testing on the website. Since it was just for demonstrating the importance of cognitive testing, we could have had more ET sessions. Just imagine what would have happened if we had spent more time in testing the product. Remember, we did not focus on security, payments, accessibility, browser compatibility, mobile app/mobile website and many other important areas of DickSmith’s product. While automated checking has been useful in their DevOps environment to respond to change; it is evident that the lack of cognitive testing has allowed many undetected bugs in the product. Some of those bugs are listed in our bug report below, which I co-authored with Paul Crimmins and Scott Miles. The details of bugs including pictures & videos are available, but we decided not to post them here for all bugs.

Bug Report

Testers: Rajesh, Paul, Scott

Method: Exploratory Testing using heuristics and oracles
Duration: Two sessions, 45 minutes each
Environment: OS: Windows 7, Browser: IE9
Functions focussed: Registration, Login, Search, shopping Cart, checkout, discounts and promotions, store locator
————————————————————————————————————————


Search bugs:
  1. Tool-tips inconsistent
  2. Price filter displays a backslash (\) before dollar sign ($)
  3. After adding items to the card and performing a second search, the cart would sometimes display 0 items in the cart. When selecting another item, the previously added items would again be displayed in the cart.
  4. When the cart has 11 vouchers and the user attempts to add 1 more, the maximum number of vouchers is reached and all vouchers are removed instead of remaining at 11
  5. Videos displayed had no visible close button
  6. No character restriction on price filter when using “right-click paste”


Checkout bugs:
  1. The most annoying bug: Apple iTunes coupon does not go away after adding to cart.
  2. After adding the suggested voucher to cart, if your other items were store pickup only you could no longer check out.
  3. Applying wrong discount coupon hangs the site
  4. Going back to checkout page causes “apply discount” toggle to behave in the wrong direction


Registration bugs:
  1. Terms and Conditions open in same window and clears your password when you press back
  2. The website removes whitespaces at the beginning or end of a password, but this is not indicated. There was much confusion over why the website was giving the error that not enough characters were entered when the password field displayed more than the minimum required
  3. The confirm password field briefly displays a green tick for 1 second when the correct password is entered. This then disappears confusing the user.
  4. Since when are spaces and a dot (.) are correct names? The name I used for registration was (      .).
  5. You can use anyone’s email and register them for DickSmith. There is no email verification required.
  6. The registration confirmation email shows no brand images. I tried it on different browsers, on cable network, wi-fi and 3g.


Add Address bugs:
  1. Address fields display a green tick when completed, but fail validation when “save” is pressed
  2. Address fields continue showing “required field” error after text has been entered
  3. Can change state and postcode after selecting from locality service. Doing this, we were able to set our address to a suburb in a state that does not exist with a null postcode. How would you ship items to this address?


Store Locator bugs:

  1. Store Locator gets confused when you enter % sign. Actually % confuses many other areas of functionality.

A short spiel on Agile

by | Syndicated

As always I keep hearing people talking about Agile ‘methodology’. Many of them do it because of ignorance. They have heard about the word, have read someone’s article or LinkedIn post and now they seem to know Agile. Whether it is Agile with a capital ‘a’ or not, they don’t know; or don’t care. After all they know about ‘agile methodology’.

Methodology or not?

Let’s look at the definitions. We are not studying etymology, but understanding the meaning of the word will guide this discussion. I looked at dictionary meanings as well as Wikipedia ones. Wikipedia is a good source of information, but we should be aware that it is not an utmost authentic source of knowledge. The reason I looked at Wikipedia definition is because it provides what we call common knowledge. Unless you are a (good) tester, chances are that you wouldn’t complain or question the definition. If you do, then you would probably investigate and hopefully change the definition in Wikipedia.

The Wikipedia definition of methodology:

Methodology is the systematic, theoretical analysis of the methods applied to a field of study. It comprises the theoretical analysis of the body of methods and principles associated with a branch of knowledge. Typically, it encompasses concepts such as paradigm, theoretical model, phases and quantitative or qualitative techniques. (https://en.wikipedia.org/wiki/Methodology)


In other words, a methodology is a way of doing certain things that is widely accepted.

Agile does not define a specific way or custom for doing things, especially developing software. It does not prescribe anything. It is a widely accepted word, mostly mistaken for a methodology, but it is not a particular way of doing things.
Framework

When I typed ‘framework’ in Google, the definition that I saw was: “a basic structure underlying a system, concept, or text.” 

Wikipedia definition of software framework is a complicated one and hence I decided not to discuss that here. I did not see value in that. Instead, I offer an easy definition:


A framework is a structure around an idea or concept that provides enough details so that it can either be reused as-is or be further developed by different people in different shapes.

The above definition makes SCRUM is a framework because it prescribes a way of doing things. If you do not do Sprints, you don’t do SCRUM.  Even XP ‘practice’ is somewhat prescriptive with its values. 

Agile has no structure of its own. It is a like an ice cream parlor where you go and ask for a specific flavor but do not ask that you would like to eat the parlor. (I have read this analogy before, but not sure where).

Agile as a Value Center:

‘Agile is a set of values.’ Partially correct. But Agile is not just a set of values.
Agile guides other methods, frameworks, approaches and practices through its values and principles. Different frameworks follow different values but they still follow the Agile principles. I mentioned XP values (the mnemonic I use for this is SCCRF). SCRUM or DSDM may follow different values than XP, but their values still fit well into those core values and principles. Same applies to all other flavors, methods and approaches of Agile. They all sit under the same umbrella, the Value Center.

Test Engineering Alliance Melbourne (TEAM)- Connecting Testers!

by | Syndicated


Any profession or skill grows when it gets support from community. I have been part of various testing and IT communities across the world. So when I moved to Melbourne, I started looking for testing communities which were doing good work. I found few groups which were more ad-hoc in nature. Some of them were set up as part of the marketing strategy of their parent organisations. After searching for few months, I approached Erik Petersen and Lee Hawkins in the beginning of 2015. 

We had many discussions among us and with others in the CDT community of Australia about launching a meetup group. Everyone in the community supported the idea of having a meetup in Melbourne as it lacked a solid group that was focussed on the advancement of testing. AST supported us through the Grant program.

We had our first meeting on July 03. The event was sold out and there were about 17 people in the waiting list until last minute. We had limited the number to 35 due to space constraints and we expected 20-25 people to arrive. For our surprise, almost everyone who RSVPd for the event, arrived. I must thank Colin Cherry for his support while we were busy organizing the event even after his personal commitments and travel to Europe.

The event started with me introducing the meetup, the motivation behind commencing it, how we derived the name, about our CDT inclination and benefits of becoming part of a group like this. I acknowledged AST and other sponsors like Chandler Macleod and Test Insane for their support.

After my introduction, Lee presented about his experience of attending and presenting at the Nordic Testing Days conference. He gave some background on the conference, showed pictures of beautiful Estonia, told audience about the NTD attendees and renowned speakers at the event. The important part of Lee’s presentation was about telling our audience the benefits of attending conferences and how to convince bosses for conference attendance approval.  At this point, Lee rightly introduced SpeakEasy programme to all.

The next up were the Lightning talks. Before the event we had asked meetup attendees to submit proposals for their topics and there were two volunteers interested in giving talks. Erik had agreed to MC the lightning talks and he also volunteered for giving a talk himself. So we had three lightning talks lined up. Erik gave his talk first and explained the Heuristics Testing Table that Test Insane had sent us. We also gave away the tables as a reward to the speakers. Erik also talked about a little puzzle game to illustrate the use of testing heuristics which got everyone interested. Next came Pavan with his very interesting slideshow which used some cool graphics. He spoke on “The Age of the Divergent Tester”, an interesting topic around psychology and how we think as testers. Last, my colleague Paul Crimmins gave his talk and shared his experience of undertaking the BBST Foundations course, run by the AST. Both Paul and Pavan were first time speakers and it was great to see them volunteering to give a lightning talk.

After the lightning talks, we decided to have a 10 minute networking break. It was good to see people chatting and networking over drinks and food. The event was after work hours and it was important to arrange for some food to keep energy going for dice games which were planned after the networking session. When I spoke to Lee about dice games when we were planning for the event, we assumed that there will be a lot of interested people participating in the Dice game. Considering that I bought a lot of dice of various designs which gave Lee an opportunity to laugh because there were so many of them. Anyway, the game proved to be blockbuster and if I hadn’t stopped the two groups playing, the games would have continued overnight.  A simple game, but it’s amazing to see how it challenges the critical thinking.  I got feedback from meetup attendees afterwards who want the Dice game to be part of all next events.

When we chose the date of this event I was a bit concerned that being a Friday not many people may show up. I was indeed excited to see so many passionate testers who gathered to talk about testing, learn more about it and network with others. Our main motto behind starting this meetup was for Connecting Testers, which this meetup proved right.

The pictures from the event can be seen at:

The blog post from Lee about the meetup is here:

For all those who would be interested in learning more about this meetup and even in joining it, below are the details:

TEAM Twitter handle: @AussieTesters
TEAM LinkedIn group: https://www.linkedin.com/grp/home?gid=6968269

Our Experiments with a Webinar

by | Syndicated

Lee Hawkins inspired me to share my own experience of the webinar that we hosted together last week (14 May 2015). Unicom contacted me few months ago to be the keynote speaker at their conference in Melbourne. I agreed and later they asked me if I would like to present a webinar too for which I agreed too because the topic suggested by them sounded interesting to me. To be honest with you (even though I don’t have to), I mostly find webinars boring because often they have a single presenter who is either talking to herself or trying to sell something to the listeners. Well, I do not have both of these qualities and I am afraid that if someone sees me talking to myself, they will consider me well, ummm, not in the right mind; which I anyways am.

I still haven’t told you the topic of this webinar. It was “Business Value of Testing”. I haven’t also told you how Lee came to become a co-presenter. Well, since I don’t much like listening to webinars (people talking to themselves or selling to listeners or both), I wanted to do a webinar which was more interactive and interesting for listeners (all three of them…or may be more). I have no idea how many were there listening, but I got few comments afterwards which were mostly positive. May be people just wanted me to feel good even though they didn’t mean to. So, I asked Lee if he would like to present with me so that we can make this webinar more useful to listeners by making it more of a conversation with them (even though we put them on mute). Lee being a gentleman and my good mate agreed to help out. It turned out that because of our love and passion for testing, we enjoyed the webinar a lot. We were talking, discussing, somewhat arguing, referring to books, people and quotes and in all we were having fun. I am not sure whether listeners were having fun or not, though. I worked for a radio station as a show host during university days and that was fun (if they had paid me well, I’d still be one). I guess I was pretending that I was a radio host in this webinar too. Otherwise it is boring (and a bit strange) to talk to a laptop screen from up close in a conference room.

The webinar can be found at https://www.youtube.com/watch?v=UM79oNi4rng

Lee’s experience of this webinar can be read at:

The other Unicom presentation (a real one) is on 27 May at Melbourne Convention and Exhibition Centre. I will be talking about the Future of Testing. I know you have probably listened to others talking on this same subject many times before but you know what, no one can predict future and therefore it is going to slightly different than all other talks (unless someone recognizes their work). :o)

The full program can be found here:

Organizers have offered me a 50% discount for anyone who is interested in attending. So if you would like to attend, use this opportunity.




Are you noticing?

by | Syndicated

Many years ago, I read Sherlock Holmes novels. Once I started reading them, I read almost all of them. I loved the short stories (The Adventures of Sherlock Holmes), probably because they were short and were easy to remember.



Sherlock Holmes is a detective; has many skills and he seems to be passionate about Chemistry and anatomy. But one thing that seems to be very common in all of his detective activities is that he sticks to logic, has absolutely fantastic observation and deductive reasoning.



When I read those novels, I noticed (observed) that my observation (noticing) skills improved. I was able to notice minute things and deduced outcomes from those observations. I also noticed that sometimes people commented (and complimented) on those observation skills. Then I became a software tester and met Michael Bolton. Michael is not a detective, neither he is a singer. He is a great teacher and an awesome software tester. He introduced me to better testing and about using my noticing skills in testing. 

I will share the secret with you soon, or eventually you will find out if you are noticing.




I wanted this legacy to be continued. And therefore, few days ago I threw a quick challenge to my team. I usually do it because I believe we sometime get so engrossed in our day-to-day work and somewhat mundane activities that we forget to do what may add value to us.

A little distraction of this kind refreshes the brain cells and I guess it helps recharge the depleting neuron batteries.


I told the team that this challenge should not take more than 3 minutes to answer (prioritisation). I asked them to call out (decision making & communication) as soon as they find out (exploration!) what they understand from what was there in the challenge. 

It was also one of my experiments with Gamification. I recently finished studying Gamification which I found very interesting. The course was delivered by Prof. Kevin Werbach (@kwerb) of Wharton School. Details are here.

Below are the steps that you need to follow to complete the challenge:


1. Visit this website : http://searchsoftwarequality.techtarget.com/answer/Does-a-tester-actually-need-test-cases
2. Start reading the article

3. Do NOT sign in or enter your email ID in the text box. 


4. Continue reading without pressing the green ‘Continue reading’ button.


5. After few more lines, stop reading


BTW, I am not endorsing this website but because I used it as an example, I would give it some credit. I did not bother reading the complete article as well. 

6. Retrospect on your experience

If you were able to visit the website, you made good progress. Well done! If you were also able to read the article without pressing the green button; brilliant! You get extra point.  If you couldn’t find none of these, I have explained it at the bottom of the page.

The point that I am trying to make here is about ‘noticing’. We come across many situations where we say, “sorry I didn’t notice it”; or “we missed the bug because we didn’t notice it” etc.
There are many situations we ask the question, “how couldn’t you notice it, it was so obvious”. “We did notice it, but we didn’t think it was important”. etc.

I would strongly recommend that you look at James Bach and Michael Bolton’s presentation on “Testing and Noticing” to understand it better.
In the presentation they explain that ‘priming’ reduces noticing among other factors. What happened to some of my team members when they could not even find the website was predominantly due to priming effect.
Understanding it from another perspective may help too. I wrote a post about Situational awareness some time ago. You can read it here. 
If you could not find the website but after seeing the explanation you said to yourself that you always knew it was somewhere there, then let me tell you that many things appear obvious in hindsight. (“Tools of Critical Thinking: Metathoughts for Psychology ” by David Levy). 

What can help?

James & Michael’s presentation talks about factors that can help in improving these skills. One of those is very important, Practice.


There are books on critical and lateral thinking that may help too, especially in practicing the skills. I read De Bono many years ago. Lately I read ‘Thinking, Fast & Slow‘, which is an awesome book. There was another book called ‘The Art of Thinking Clearly‘ by Rolf Dobelli is a decent book. While these books are not specifically about observation, my personal view is that clarity of thought improves observation. 

I am not sure whether reading Sherlock Holmes will help you, though.  


Explanation of the challenge:

1. Website is in white font. Mouse over would show it.

2. If you are able to see the right page that I want you to see, you will notice that there is a scroll bar of a page length. The question that I had when I looked at it was, if this is a blank page and if I cannot read the article without entering the email address, why do they have this scroll bar. Either it was a catch or it was bad usability.


3. By scrolling down you can read the whole article.

Acknowledgements: 
James Bach (http://www.satisfice.com/)

Image courtesy: www.Wikipedia.org