What makes an agile DevOps mindset? (Part 3 of "The Agile Testing Mindset in the DevOps World")

Inpart one I covered the basic terms of DevOps and the proper mindset.  The need to react to a fast and ever-changing environment, deliver ongoing changes to high quality working software led to the DevOps culture and framework that enables all of this to happen. I also discussed the crucial need for establishing the right mindset or culture, no matter what tool or processes we employ, in order to  achieve the goals of delivering quality working software in a complex and rapidly changing environment.

Inpart two I covered the changes that software testing went through during the last few years, which have led to the rise of the Agile testing mindset and to the development of DevOps culture.

In part three -  I will take a look at the actual mindset and culture we need to implement and cover real-life practices that will promote Agile testing and DevOps culture for delivering quality software, fast.

Well, having good tools, "good" processes and well-defined responsibilities are just one part of it. If we want to achieve effective testing in a DevOps world we need to establish the right mindset.

http://mappingcompanysuccess.com/2009/01/wordless-wednesday-a-really-bad-culture/

There are actually three basic things we want to achieve.

·         Remove barriers between silos

·         More quality

·         Fast progress and reaction

Those goals needs to be broken down into some practical instructions to become actionable. It will also not be enough to merely “say” these messages, we will need to follow them in practice and make sure to connect the dots whenever we encounter behavior that contradicts or fits the mindset we wish to achieve.

We wish to have a culture of:

•       No walls, no gates, no transitions …

•       Quality

•       Trust and collaboration

•       Early feedback

•       Continuous improvement

•       Leanness

Let's break some of these elements down into more practical instructions and take a closer look at the bricks that hold this entire culture together, including:

1.   Collaboration

Collaboration is a huge topic. Some of its meaning and significance is also relevant for other components of the mindset that appear in this article - after all, it's all about collaboration.

When we want collaboration we need to mean it. First we need to say the word itself clearly; COLLABORATION! We need every one to hear it and to act upon it.

Collaboration is a big word, it needs to be taken apart to actual behaviors, e.g.:

We want people to communicate directly. We want them to sit and talk. We want them to approach each other when they have questions and we want them to be pleasant and approachable themselves. And it should be OK. No one should get angry that a tester interrupted a developer with a question.

The organization must supply the right infrastructure for collaboration: encourage daily meetings, create spaces for communication, promote pre-planned meetings, planning ahead, and retrospective examination. Everything that brings people together to collaborate on the same tasks.

There are so many methods today that offer effective techniques for better collaboration.

For example: sitting together makes it easier for teams to collaborate. Placing developers, testers and others close to each other will make it easier.  When a tester has a problem he can reach out to a developer, just by asking.  Being part of the same team, sharing the same tasks and goals makes it easy to communicate on a relevant level without bothering the other,

Direct communication, sharing knowledge leads to faster production as well as a better understanding of what we need to move faster (tools, skills)

I hold the managers accountable and responsible for making this happen. They are trend-setters. They should lead, first by personal example and also by guiding, encouraging, and teaching collaboration.  It is unacceptable for an R&D manager to shout at Support staff for failing to handle defects. He must understand that the appropriate point of view is to focus on how we can help Support understand the customer’s defects  and take it from there to prevention. Maybe we can add better log analysis, maybe better monitoring or other related tools. He should also encourage direct communication and understanding between these two departments.

I can create collaborative tools such as monitoring and effective data visibility, but the tools are always second to the understanding and the actual practice of joint collaboration.

DevOps means that we are all collaborating together – PO, Dev, Tester, analyst, architect… There is no ego.

Imagine a Formula 1 team– without full collaboration and understanding no one can win the race and no one can get better at it. (I will return to F1 later on)

I was once working in a company that was located in an industrial area. We were always on the hunt for a good place to get a fast meal during our short lunch break. Our offices were surrounded by restaurants whose entire business targeted the few hours each day between 11:00-14:00, when they would feed the hungry tech workers. These places needed to be very efficient and be able to serve a lot of people, and as many as they could in a relatively short period of time. This was their bread and butter and they had to do it fast, provide high quality food and feed as many people as they could.

One of these places was so efficient at this, that when you entered the place in took only 1.5 min to be seated and served. Every 10 seconds a person was out the door.

You started the line by choosing which food you wanted. There was a running production line of two people who filled your plates. . From there you proceeded to the registry, where a lady would take your card and make the bill while offering quick packaging and drinks.

This was a very efficient production line.

The problem began when a customer who had no corporate meal card or was not familiar with the flow came in. This customer would keep the cashier busy by asking questions, paying in cash, etc. Treys of food started piling up near the cashier. As a customer it was very irritating to wait, all I wanted is for her to take my card, close the bill and let me get out of there.

But they were good, so good (Kanban-good) that when they noticed a 3-tray queue forming at the cashier’s, one of the plate-fillers would leave his station and efficiently handle the waiting customers, help the cashier and get back to his station once the pressure resided.

In addition to an effective understanding of flow, this is a good example of collaboration. Similarly, when we have a bottleneck, a developer can switch for a while and test the application just because it will get us faster to delivery.

2.  Testing is everywhere

Instead of being a step at the end of a development cycle, testing becomes something that happens everywhere. Everyone needs to understand that we are no longer compartmentalized.

What does ’everywhere’ mean in this context?

It means that it starts from the first line of code to the smallest functionality, through integration, regression and also in production. It never stops.

In order to do it everywhere, we need skilled testers who know their automation. Testers can no longer be unfamiliar with technical issues. They are collocated with R&D, sharing the same knowledge and goal to get things done.

But it's not just that, it's also means that if we want testing to be everywhere we need testing to be done by everyone.

The tester cannot be the one who writes every test, sometimes it’s easier and cheaper for a developer to do it. A developer that refuses to write tests is a problem. We will create the right tools to do so, but the understanding that everyone writes tests everywhere is a message that is crucial for continuous integration, delivery and deployment.

The mindset is that I am not waiting for someone else to test. If needed I will do it myself.

3.   Build something that has the quality of our brand written all over it 

Quality is non-negotiable.

 It's not something that the tester owns or protects with his life. If we mess up with quality, we can’t blame the tester.

Everyone is accountable for quality and everyone needs to do everything to maintain it.

Statements like " we should expect a massive amount of defects since we are a startup and that’s just how these things go" are unacceptable. We need to find the way and the tools to prevent this.

You cannot write untested code. Having problems testing it? Let us know. We will create the right tools but the information and flags needs to be in everyone’s minds.

No more "I am a developer, I am not doing a tester’s job" – you are doing a developer’s job by making sure nothing that is of low quality leaves your workstation.

4.   Start thinking of QA as an enabler rather than a gatekeeper

Instead of a gatekeeper holding things back from going to production, the tester becomes an enabler, helping good code rapidly flow into production. He has his hands all over the entire flow. Engaging with developers as well as with the Ops teams, perhaps performing smoke tests in the production environment, helping to identify monitoring needs, identifying more precise needs and better techniques for automation - everything that enables a better and faster flow to production.

We must banish the assumption that a tester should not be involved in production. Testers should understand the business aspects as well as the technical and product aspects, and should use this knowledge to enable afast and high quality flow to production.

5.   Always-working software 

Working software is one of our primary goals. If it doesn’t work, it’s worthless. We should all strive to create working software, not only testers.

Working software means coding + testing.

In every point of the development flow, the software should be in a "working" state. From the first line of code. This is a mindset that is hard to maintain. We will build the tools that enable the developer to check their code at any time to make sure it didn’t break anything.

The business should have the ability to change whenever it desires, to access the code at any time, change it and fix it to address a customer issue and quickly move on to the next assignment without big regression cycles and rollbacks, and without throwing away valuable code that  still didn’t reach “working status” and might need much additional work alter on, or without any hassles.

We need to instill this approach in everyone who is part of the production cycle, since changes will inevitably be required and the ability to react fast is crucial.

I remember working in a company where we used to have this big management meeting every Thursday, wherein we would report the development status to management. I have to repeat this: 90% done is not done. "We finished all the code, now all we have to do is test it" is a dangerous statement. We all know that everything can turn into "non-working" in one minute. And it usually does.

Working with small chunks with the right safety nets will help keep this mindset in place: continuous integration, red build approach, proper visualization, direct communication regarding problems and more.

A red build policy is crucial. Nothing is added to the build once it is red. It runs every all day with all tests being applied and no team, leader, manager or developer should ignore this.

We fix the bugs and then continue.

It’s like in a car factory. If a machine produces a faulty carburetor, the entire production line stops. Why? Because it is a faulty item and it will cost us. It can cost us as much as a recall of an entire line of cars or severe damage to our reputation. The costs of fixing and disturbing the entire production line are very high, and hence it is crucial to ensure everything works during production. This is a common approach in the automobile industry and in other industries. But somehow it is harder to grasp in software development.

Tools are created to spot these areas of faults, but it is the people’s responsibility not to ignore them.

I have seen many organizations ignoring the red build policy for various reasons from "oh, it’s a regression bug“ to “oh, it’s an old bug".

Keeping good quality depends on keeping this policy in mind.

The zero defects approach is also critical. We stop and fix on the spot. We don’t ignore defects and leave them to the next version or to end-state integration.

6.   Lean thinking 

Lean thinking is a huge topic, we already set some references points for Lean in some of the previous points. But to maintain it, we expect everyone to start thinking in terms of production line flow. For example:

•       Reduce siloes 

•       Remove waste

•       Identify bottlenecks

•       Value driven thinking

•       Continuous integration, Don’t wait for the “integration hell” period

Meaning: identify siloes that slow us down, identify unnecessary handoffs between people, and identify bottlenecks or waste.

As for waste, we define waste as anything that does not add value to the process. For example, in a furniture factory, producing tables without the ability to ship them to the customer creates inventory waste. Why is it waste? Because I need to pay for storage, it can become unusable over time. It can become damaged along the way.

Defects are another type of waste – why? Because I need to fix it, maybe recall fulfilled shipping orders, and replace it. It costs money.

Any waste in the development cycle?

Testers that are waiting for a build to test; extensive documentation that no one reads; extensive documentation that needs to be updated due to requirements, design or code changing towards the end (inventory).

Any bottlenecks?

A manager whose approval is required before anything moves forward; software that is waiting to be tested; code that is waiting for review that no one takes care of due to lack of time and meanwhile no code is being release.

In one company I worked in there was a testing bottleneck inside a Scrum team. Software did not ship out because "it was waiting for testing". After a few discussions with my colleagues we decided to just pull the tester out of the team. No tester = no bottleneck. It is a radical step but it sent a message: no waiting. The team needs to learn how to balance their work in a way that less code is being written, more is being tested and code is being shipped out rather than piling and waiting.

 In Agile and DevOps culture we look at the entire production line, we do not calculate the cost of each unit separately, but instead look at the entire value flow from beginning to end.  It does not matter that a developer costs more than a tester if the entire line of production can't produce working software due to lack of testing. What matters is that we have a piece of code that is getting older, harder to maintain and harder to test and repair later so we better get it moving and release it.

We expect the team to take ownership and promote stuck tasks, to identify and solve bottlenecks and to suggest the solution.

There's a lot more to Lean then just that. But the point is looking at everything as one production line, identifying bottlenecks, removing waste and continuously improving for better performance.

Only people who take active part in the work can identify the production line’s actual problems. They usually know the solution.

7.   We are all in this together

The first step will be to open up the lines of communication between all teams, not only between Dev and Ops but also between Test, Dev and Ops so we can all deliver E2E.

It means that we are all accountable for E2E delivery.  As a developer I am in charge of testing my code. I even report on the testing status in the Daily. I am accountable. When we work together so closely and things are continually being developed and tested it's easy to take E2E ownership because I know what's going on, there are no siloes.

In this environment you will see that the testing environment is the developer’s "problem" and concern, while the production environment is the tester’s concern.

The tester is not the sole person responsible for bugs in the end product, we are all responsible for the problem and the solution.

Imagine you are calling a help desk service for your cable TV. And imagine you accidently reach the wrong person, and the representative asks you to call again, or to call another service, they leave you empty handed and frustrated instead of helping you with your issue, regardless of the fact that it is not their “job” to solve it. This is not a mindset of "we are all in this together"; this is a mindset of “it’s not my problem, I am closing this service call".

‘We are all in this together” attitude generates feedback regarding tools we need to implement, what we need to test and why. From the former example we may be able to move faster if the service rep had the option to direct us to the right person. We may even monitor the time it takes and the amount of times it happens so we can think of a better solution in the future.

In a rapidly changing environment, we need frequent information available to as in different areas. We need everyone to be aware to the fact that we are all in this together and losing one of us may cost us dearly.

This is a true story: I remember one  offices of a major car rental company located in one of the biggest cities in Israel. The headquarters marketing department decided to go with a promotional offer that seemed likely to generate a lot of revenue for the company. The numbers looked amazing and would highly improve the figures we present to the public. What they did was to rent cars for the weekend for only 30$~ for the duration of the entire weekend. The promotional offer was aimed at a targeted audience of Orthodox Jews. Why? Since they don’t drive during Sabbath anyway, the cars would be safely parked around the city and returned after the weekend, and money will be entering the cashier without the cars even moving. Needless to say that what mattered was the actual revenue.

It took a lot of field employee pain and a lot of time, meetings and shouting till they realized they are losing money… And big time... But since they didn’t discuss their proposal with anyone, and were not aware of the entire production line, and were not fully cooperative with other departments, or didn’t know how to properly assess the costs of their actions, and were only concerned about the bottom line income, they didn’t know what was coming for them. Instead of earning more money the company lost big time.

The local rental offices suffered from a severe parking problem.  So in order to keep up with the promotion the cars needed to be shipped from other nearby cities around to the small local office (and this costs). Parking spaces were very limited due to the crowded nature of the city, so cars started getting parking tickets. The crowd that reacted to this attractive sale was huge so the office needed to hire another secretary to deal with this crowd every weekend for many consecutive weekends, at least at the beginning, in order to relieve the load on existing personnel. They also needed to hire another employee to deal with the municipal authorities in order to to pay or cancel all these parking tickets (usually to pay). Dealing with parking tickets and a crowded office became a regular event at the end and beginning of every week, when a large amount of cars was being rented or returned. 

Regular flow of customers' began to deteriorate since the small office could not keep up the pace and service suffered. Now, you do the math. Money was being lost.

8.    Shared ownership

Shared ownership means what's yours is mine and what's mine is yours.

Like brothers, your room is mine and my room is yours… ho... Just kidding. But this is what it feels like when we start mentioning the topic.

What was mine? As a testing manager the entire automation operation was mine. Testing mangers used to raise this talented child called `automation`, with talented skilled automation developers placing the framework and building tests. How many times have we heard developers complaining that the quality of code in automation is subpar, that it needs engineering review? I heard it from time to time. And most of the time they were right. The separation from developers was not very helpful for automation code engineering and quality.

I also experienced the control testing mangers had over the entire automation operation to the point where in one organization developers created their own automation tool just to shorten the cycles and were given a slap on the wrist for doing so without the authorization of the "BIG" testing manager.

Well, no more. Automation belongs to everyone. There are places where the actual coding decisions and management are under shared development-testing leadership.

It is bilateral. Any developer can activate, write, or review automation code.

This is heartbreaking for testing managers.

But, when a developer start dealing with automation and makes it work, he sees how hard and complicated it is.

But the other way around is also something we need to accept. A tester can change product code. Writing automation code sometimes requires special coding standards that will allow this entire cycle to run. Sometimes the code itself needs to change. There is no need to wait for a developer to make the changes that the testers need to continue building automation. Testers can change code. Yes. I saw this happening in Scrum teams and it works fine. As long as we are all on the same page, working towards the same sprint goals, meeting daily and discussing technical issues, code changes done by testers become trivial, a non-issue.

9.   Early feedback 

Early feedback means that testing, rather than being the last step of the process, which creates a gap in quality analysis (too late, at the end, when it’s expensive to fix or change), becomes a means of providing early feedback throughout the entire process. Remember, we test everywhere and all the time. But not only the testers, everyone is expected to work toward providing early feedback.

Our goal as testers is to look for feedback to provide all the time and provide it in a way that allows for early corrections and preventive measures to be implemented.

Testers provide feedback from business all the time, as they are the hands legs and eyes of the business inside the Scrum team. They do the same in everything that is being developed.

This mindset leads to developing and testing small chunks of software with frequent integration and frequent tests. Why? Because when the feedback comes earlier it's easy to relate to, easy to fix, easy to detect or easy to retest.

This mindset is the reason we will place this log instead of the other, it’s the reason we will automate one process instead of the other, just because we need this feedback earlier than the other. This mindset will lead us to continuously monitor system performance, and not leave it to the end of development for a specific phase of performance testing. Why? Because assuming we are continuously building and testing every day, any deviation from expected performance is immediately detected and the relevant code was added just a short while ago.

10.          Visibility

Visibility is a powerful tool. Correct usage of visibility can create the expected behavior. We expect everything that can point us in the right direction or provide early feedback to be plainly visible.

Let’s illustrate the value of visibility with an example: Pretend that in your town, there’s a problem of speeding drivers. And as you are a responsible driver yourself, you know that’s dangerous.

To solve the problem, police set up hidden speed cameras in strategic locations. According to them, this is meant to catch dangerous drivers who exceed the speed limit, and so reduce the number of accidents. But as it turned out, drivers didn’t actually drive any slower, as they didn’t see any visible deterrent (hidden speed cameras, remember?)

So was there any value in setting up the cameras? Apparently not. Our taxpayer money was spent on buying and installing them, but the actual driving speed didn’t go down. So there was a challenge (speeding drivers) and a quick reaction (hidden cameras), which didn’t actually produce any value.

But was there ANOTHER value in setting up the cameras?

Yes, and you can probably guess what it was. With the hidden cameras in place, the police department (and the town council) made buckets of money from collecting fines.

The problem, of course, is that’s not what we were looking for. We wanted to reduce the number of accidents by making people drive slower. THAT was the intended value.

When the hidden cameras were taken down, and VISIBLE cameras were set up, people started slowing down. And that reduced the number of accidents.

See? Value, and the value we were looking for.

The following is an example of changing driving speed by means of clear visualization:

The following is an easy way to control large crowds:

Control your inventory easily:

Control your child?

And here is how it looks like when you deal with DevOps. Everything is visible, and the relevant information that needs to change is presented:

11.   Continuous improvement

We expect everything to grow and improve. This means our ability to move fast, our skills, our insights and our performance. How? By adopting the mindset and tools that allow us to continuously improve.

Performance matures over time, it is not born excellent. Experience is learned but only when we aim to learn from past events and apply our conclusions to future events.

Retrospective, for example, is one of the tools used to improve. We examine our performance and deduce the next actions toward improvement.

This mindset should emphasize failure and mistakes as something we learn from. We will not tolerate blaming. Testers are not to blame if a bug was found and developers are not to blame if a mistake was made. This is easier said than done. People will often quickly turn to anger and blame, and hence attention and constantly reminding this point is crucial.

When we work in small chunks, it's easy to detect mistakes earlier, it's easy to fix them, it's cheaper to fix them and easy to learn from them and correct them. Therefore stopping the race from time to time for the sake of retrospect (or having kai-zen events) is a helpful practice to adopt.

Growing and improving can only happen in a nurturing environment. Respect for each other’s role, trust and collaboration should be built.

Let’s go back to F1. They race 20 times a year. Only 20 races. Each race is "to be or not to be". To reach the point of an excellent performance requires a great deal of team collaboration, openness, respect and a lot of continuous improvement. They didn’t start from the point of being able to stop for 2 seconds in the pit stop, they grew to that point by continuously improving their ways. The way they change a wheel, the way they stand, the colors they highlight, the people that need to be present in a pit stop – everything is examined and improved. 

You can see for yourself in the next two short videos, how they looked like back in the 50s and how they look like today… and how they got there.

 Have you heard of Team Sky? The won the Tour the France. Why? Because their manager’s approach was to retrospectively examine each race and continuously improve by making changes in small steps. They got to the point where they were taking their pillow with them to a race since it makes them perform better the next day. This is how champions get to be the best.

Who is the tester then?

•       Enabler

•       In every one of us

•       Everywhere

•       A skilled developer

•       A skilled business representative

•       Automation everywhere

•       Manual tests are saved for exploratory purposes

What is the culture?

•       No walls, no gates, no transitions …

•       Quality

•       Trust and collaboration

•       Early feedback

•       Continues improvement

•       Lean

Old world vs. new world DevOps approach and mindset.

One last thing:

What do you need to make this happen?

•       Management’s active support and appropriate mindset

•       Understand the fact that we need to react to a rapidly changing environment

•       Always be on top of technological changes

•       Highly skilled testers

•       Highly skilled developers

•       Hire team players

•       Start even without the mindset fully developed – because mindset is something that grows over time.

•       Hire a Supernanny (Agile coach). Do you know the Supernanny? Well , she is always around, advocating the right culture and coaching, teaching or mentoring on the right culture for desired behaviors

http://greatergood.berkeley.edu/article/item/powerless_parent