So what has testing got to do with DevOps and the mindset evolved? (Part 2 of "The Agile Testing Mindset in the DevOps World")

In part one I covered the basic terms of DevOps and the related mindset.  The need to react to a complex, fast-paced and ever changing environment, make ongoing changes to high-quality working software, created the DevOps culture and framework that enables all of this to happen. I also discussed the need to establish the right mindset or culture, regardless of which tools or processes we use in order to achieve the goal of delivering quality working software in a complex, rapidly changing environment.

In part two I will cover the changes that software testing went through during the last few years, which have led to the rise of an Agile testing mindset and up to the point of a DevOps culture.

DevOps aims to break the barriers between Dev and Ops. This also necessitates breaking the barriers between Dev, Test and Ops. Nothing stays the same for a traditional testing operation.

Testing operations undergo a major change in a DevOps culture. The traditional testing methods, tools, approaches and mindset will not do.  Automation testing is the glue where all this rapid delivery of value can exist. Automation testing becomes a critical and major component of being able to deliver rapid value and to react to changes. Therefore everything changes for testing. The tester's skillset changes, methods change, the way we test, what we test, when we test and the way we plan and execute, all change completely.

http://blog.lifespantechnology.com/it-asset-disposition-blog/bid/257173/Are-All-Computer-Recyclers-The-Same

**Right before I start covering some of the testing practices applied throughout history, please keep in mind that thousands of pages were written on the subject and I probably gave more attention to some things over others. Please forgive me for that and feel free to add comments, points of view or landmarks.

Agile testing procedures and mindset were starting to take shape even before the term DevOps was coined. The traditional approach to testing had to remove a few layers and change some other layers in the procedures, while adopting a different type of mindset. This new mindset was particularly crucial, both for testers, developers and product owners, as well as executives and all others involved. It could not work without it.

Being (and using) Agile was overwhelming to testers and developers and in order to achieve the correct outcome, i.e. frequent working software, new practices and skills were developed by emerging professionals in this field, in addition to a shifting mindset that was crucial in order to produce working software in a rapidly changing environment.

Let's look at this shift that testing operations went through in the last few years, considering methods, tools, and mindset, which has brought us here – to the exact point of what we call a DevOps culture.

In short, looking back at the traditional testing approach we can immediately identify its characteristics.

• It was mostly a step in the process. This step was implemented towards the end of the process
• The notion of separating between development, testing and product activities was prevalent.
• The notion that testing is meant to control the quality of the product, detect defects and prevent flaws from appearing in the end product

The problem with this was that the quality-related feedback came too late – at the end of the process.

• Documentation was an essential part of every testing operation, something testers would refuse to work without it. Why? One reason is because in the traditional waterfall approach (in the many shapes and forms it can take) we mainly experienced the gap between requirement-design-coding and testing. This was a relatively big gap (considering that today a one day gap is too long, imagine what it was like when having gaps that lasted for weeks or month was the default state) , and when changes were made, naturally before going to testing they could not relay on anything but good documentation. Everything testing did at the beginning was to review what had been changed from the requirements to the time of testing. This was a hugely time consuming and wasteful, but inevitable operation. We had to do it, we were located at the end of the process, accountable for every bug found, and how could we even think not to compare documentation at the end?

Direct communication and planning instead of big documentation and plans were not a common or regular process or mindset in the development production line.

• We usually didn’t have time to test. It was always too late, and we always had to compromise on quality in order to be able to keep the timelines.
• We spent time and resources on large-scale meetings, trying to analyze huge amount of defects and move them to the next big release, thus contributing to continuously low quality.
• Developers were passing pieces of code marked as done in big chunks. 

Everyone working in testing knew that "90% of development effort done" meant nothing about the project’s actual state – it could all revert to 0% progress once the product began being tested.

• Automation depended on heavy tools, usually using GUI (the top-down method of testing). The prevalent approach was using GUI automation but it was the hardest component to test, the most expensive, and the most unstable component with the latest feedback possible. Analyzing and engineering automated tests in the top-down method, from the GUI to particular scenarios, trying to cover as much as we can in the process, was tedious, expensive and difficult.  And everything was done "after" we really needed the feedback – at the end – making it too late and too expensive.
• Many siloes, handoffs, documentation to follow those handoffs.
• Moving information back and forth between siloes created a major setback in project schedule. Discoveries made late in the development cycles often required changes to code, requirements or design. Problems that were identified in late testing stages changed everything. Unexpected requirements or changes also needed to be integrated in the process and caused changes that were discovered late during testing, imposing a heavy practical and emotional burden in dealing with changing plans, managing defects and more…

But this was how everyone did it. To the majority of software practitioners it didn’t seem wrong.

But the fact of the matter was that 90% of the software projects failed (low quality, delivery dates missed, different content, over budget and more...) Things started to change once customers began refusing to wait, once quality was something customers refuses to miss, once the entire environment started to demand things faster (otherwise we would not survive as a product), once changes needed to be integrated into software faster than the organization could normally react while maintaining high quality and addressing rapid customer demands.

What was wrong with everything we had done that far?

Too slow, too late feedback, too expensive, less effective than it could be. 

So what did we try to do?

Even before Agile, Scrum or any other tools started to emerge; some practices for solving those problems began gaining traction. And how couldn’t they? It was a necessity.

The Agile methods that were being introduced were helpful, but were not that easy to implement and were definitely not a ‘silver bullet’ solution for coping with the complex, rapidly changing environment. Amongst them the most popular tool was Scrum (although there were additional methods that adopted a somewhat similar mindset in order to make it work). We adopted Scrum in an attempt to shorten delivery time, while in the process introducing the testers to the Scrum team daily activity schedule.

If we wanted to move fast and produce small chunks of working software, we had to shorten the cycles and integrate the testers the Scrum team methodology. It was not easy for anyone. The mindset needed to change as well as the practices.

Testers where overwhelmed by the fact the developers will be accountable for quality, that developers will ‘manage’ testing. How will they work? What about regression? What about defects? Who will make the decision on quality and many more… We needed to jumpstart the mindset of working together and the shared accountability for quality, as well as some additional attitudes which I will detail in part 3.

At that stage we needed to create a mindset of collaboration, of mutual understanding, of shared ownership and shared accountability, in regards quality and more… 

But this was not enough: it’s not just the testers who were introduced to the Scrum team, it’s the testing activities that were introduced to the Scrum team.

What does this mean?

The goal was that development managers would be accountable for E2E delivery, having all the tools needed at their disposal = developers and testers as one whole, complete group.

It means that a Scrum team is accountable E2E to deliver quality working software, including all the tasks needed for this to happen. Including the testing. It’s a mindset of accountability for quality. Testing became a task rather than a person’s job description.

We also started examining the process and in order to move faster we tried to reduce critical destructive siloes which created big risks for the entire software release and necessitated time-consuming, wasteful efforts to be made. 

(http://tracks.roojoom.com/u/shirly-ronen-harel,60/lean-principles-effective-way-of-thinking,1788)

What were they?

There were two points of friction

1.       Integration

2.       Deployment

Henrik Kniberg ScruML

Both points where referred to as "integration \ deployment hell" why? Because a huge amount of defects, inconsistencies, and changes were introduced in these stages, forcing the entire production line to reopen a lot of code, and backtrack to make fixes and changes. This was a hellish period of uncertainty and not knowing when and how all of this will actually be done.

Overcoming this obstacle began with including integration and deployment tasks earlier in the process of software development: customer data tests, smoke tests, continuous integration along with testing and more...

At those times you would hear the developers complaining that it is impossible to create working software fast, especially at the end of a sprint, without automation. It was also hard to include integration and deployment early in the process without proper tools or proper automation (the latter, when implemented at the end of the entire development cycle, was ineffective for this purpose).

And at that time, they were right. But automation techniques and practices had already started to change to accommodate this new and rapidly evolving environment.

Agile automation testing (which we might refer to as the ‘glue’ of modern DevOps) introduced a completely opposite approach to testing compared to the traditional one. It means start testing from the first line of code, working your way up in the testing pyramid while continuously integrating, delivering and deploying working software, constantly dealing with defects, getting early feedback and keeping quality high at all times.

The big change came from early feedback on the software’s quality, the ability to fix early, change fast; to produce a safety net where developers can fail early and fix fast. Testing is everywhere. Instead of detecting defects, providing early feedback on the software’s quality.

It moved testing to the early stages of development.

If you wish to read more about the testing pyramid you can do so here: http://blog.goneopen.com/2010/08/test-automation-pyramid-review/

Best practices for enabling this automation to operate within an Agile environment were also developed. This enabled continuous testing, feedback and highly effective and fast responsiveness along with quality of delivery.

I will not go too deeply into this, but you can read more here if you like:

http://lisacrispin.com/2011/11/08/using-the-agile-testing-quadrants/

But all of these wonderful theories and tools could not work without a proper mindset: a culture of collaboration, trust and respect. It does not matter how well you define the process, the responsibilities, the tools, it’s the mindset that will make the difference.

You cannot test early if developers are not tuned to quality. You cannot get or react to early feedback if all testers want is to detect defects instead of preventing them. You cannot move fast if a developer does not understand the need to deliver working software in small chunks while maintaining direct communication with the tester. If they don’t understand the WHY, they will not become hooked to the practices. These are just few examples that demonstrate why the mindset needed to change in order to make this happen.

Let’s proceed to look at the Agile testing and DevOps mindset in part 3: