Building an Agile Testing Process

As organizations mull the transition to a more Agile development process, the focus is often on development or test automation tools like service virtualization that support them.

But Mary Thorn, Director of Quality Assurance at ChannelAdvisor, and co-author of the forthcoming book “The Three Pillars of Agile Testing,” argues that organizations need to think about implementing an Agile testing process to make Agile processes work well. I caught up with her recently and she explained what this means and how testers and QA personnel need to take a more proactive role in the development lifecycle. 

Service Virtualization: How does ChannelAdvisor leverage Agile processes as part of its software development?

Mary Thorn: ChannelAdvisor is an ecommerce company that provides retailers the ability to put their items on over forty leading marketplaces. It has several Scrum teams along with a few Kanban teams to address the large number of change requests that come in for particular applications. The monthly Scrum cycle is long when the number of change requests is high, and the Kanban teams help to allow a weekly release cycle for more frequent changes in the marketplaces.

What is involved in managing an Agile testing organization?

I provide the mission, vision and roadmap for our testing organization.  I also help Agile coach some of the teams in making sure they are planning and gathering requirements efficiently.

The QA teams needs to make sure they have the right requirements so they can test effectively. They also have to pair with developers to make sure they get what they need. I am trying to coach them to be agile champions so they have what they need to be successful at their job.

What are the best ways to determine how well an Agile practice is really doing?

At the end of the day, e-commerce is changing all of the time. Agile software development is the only way we can compete in the market. We support 40 marketplaces, and each one changes about once per month. That is about 480 changes for the existing marketplaces. In addition we add five to 10 additional marketplaces per year and we get about fifty customer requests per month. This does not include the new functionality we want to add. The only way you can handle that much change is to be Agile. We determine success by how much throughput we push through with high quality.

What are the key elements of continuously improving an Agile practice?

We want to make sure we deliver the right customer experience and want to make sure we improve on that. The basics of Agile include having retrospectives. Some organizations stop doing these after a while because they feel their process is good enough.

We have kept the tradition of doing retrospectives religiously because it closes the feedback loop on how well the process is doing. We can see where we had to rewrite functionality because the requirements were not well thought out or because of other problems. The retrospective provides you all of the information you need to close the feedback loop.

We also look at bug reports, support tickets, and customer feedback and make appropriate adjustments. One source of feedback is not necessarily better than the others. But the type of information provided in some individual reports is sometimes better than others.

What role does testing play in rolling out a successful Agile practice?

Testing plays a huge part in a good Agile process. At the end of the day, there are three places that testers play a huge part in Agile:

  1. The first area is automation and building a good framework so that things can be changed quickly and then tested.
  2. The second pillar is the craft of QA and testing. A lot of people in Agile do unit tests but don’t see the need to do regression testing or integration testing. You are moving so fast that you cannot test everything and have to take calculated risks. Introducing risked-based testing is critical and the biggest part is knowing what you are not going to test, and being OK with that.
  3. The third pillar is cross team collaboration for processes like code reviews and pairing. This involves making QA the best communicators on the team, and ensuring that the whole team is brought into the Scrum or Kanban process. QA is working in parallel with developers, product managers, and engineering managers to make sure that everyone is aware of the testing involved and making sure it is the right testing, and to ensure they are building the right things first.

A lot of time, people approach testing like a waterfall in the Scrum process. I try and get testers and developers to collaborate and work together. It is also important that QA be paired with people and have face-to-face contact so they can get bugs fixed quickly. This means that you are not coding one day and fixing bugs four days later. This involves getting QA to work in parallel with development.

What is involved in making the transition from a traditional test manager to an Agile test manager?

The biggest thing that changes is that the test leaders now have to partner with Engineering managers, Product Owners and Scrum Masters. You partner with them to make sure they are bought into your testing vision and initiatives because these are the people that will have to influence these initiatives to get done on the Agile teams. You move away from traditional project management of testing resources and now empower your testers to pick up their own tasks. So your role changes to more of a test leader who sets the vision and roadmap for the testers on the Scrum teams and less of a day to day do-er.

Also I feel there is benefits to Agile test leader to also become a champion for Agile process. In waterfall days the test manager typically policed the waterfall process. In Scrum that is typically the Scrum master, for most companies, and like in our organization the Scrum master is a member of the team and usually a Tester. So it is important that the testers understand the Scrum and Kanban frameworks so they can properly administer it.

What do you believe are some of the useful models for building great Agile test teams?

A few things are basic:

  1. At minimum, the tester to developer ratio needs to be one-to-two or one-to-three. One-to-four rarely works because of all of the automation that needs to get done.
  2. Second, you need some kind of test automation framework.
  3. Third you need a vision of where you want the test organization to go. This can include a road map of where you want to go and making sure that it gets executed in your model. At the end of the day, you have a QA leader who decides the plan, and they have a vision of where they want the organization to go.

It all centers around the three pillars of Agile testing that includes the automation, software testing practices, and cross team practices pillar. That is my scorecard, and how I measure my team.