The rise in tools like service virtualization for automating software testing can reduce the amount of work required to generate new tests. On the surface, this may lead to the impression that organizations can replace testers with developers more proficient at automation. But this is likely to lead to more problems in the long run, argues Hans Buwalda, CTO at LogiGear, a software testing consultancy. He explains how organizations can strike the appropriate balance between testers and developers.
What do you see as the trends driving increased test automation?
Buwalda: Systems are getting more complex, large scale, and ambitious. We are also seeing an increasing amount of attempted automation of the tests. A lot of customers worry about the amount of time it takes to retest the system. That is something that better automation can help to address.
Another trend is agile. An agile method like Scrum depends on automation. In addition there are factors like the coming of the cloud and the growing use of virtual machines, which allows you to set up instances with different configurations quickly and then bring them down.
There is also a kind of trend that I don’t agree with, of replacing testers with “developers in test” in which developers are coming in and taking some of that testing over. There is an argument that people make that developers are better at automation. But I don’t see automated testing as primarily a technical challenge. It is more of a test design challenge.
How would you distinguish software automation from test design?
Buwalda: When most people think of automation, they think that you need to be a smart developer to do it right since they are good at refactoring and hence test automation. One of the problems is that developers do not like to test. They might be happy to set up a complex framework of automation, but once it is done they are not interested in using it to write the test.
There are hundreds of books written about testing, but not all testers actually read them, and apply the testing techniques described in them. Instead I hear people talking about automated testing as if it was a development task and then they forget the testing part of it. For example if you test a transport application, you want to know what happens if I try shipping to Alaska.
What can organizations do to shift left testing in the development process, if this is not a strength of developers?
Buwalda: They need to make testers a part of the design and development process. In other words, if you have a sprint team in Scrum, you need to make testers part of the sprint team. We are doing that now, and it works well because testers can create the tests and the business test can be made available as a high level test earlier.
If you go into a sprint, a tester can create a lot of tests using keywords early on, that then can be automated afterwards. You don’t have to wait until after the application and UI become available. You can worry about that later. When you put testers in the sprint, they have a different point of view than developers. That does not mean that developers cannot be testers. It is just if you play the role of a tester you have to ask the “wrong” questions, like what happens if I hit that button that I’m not supposed to hit.
What are some of the problems that can come from using developers to create the tests?
Buwalda: We had a customer that came from a situation in which there was a large off shoring team creating test kits for them. They had about 40,000 scripts built using one of the major testing tools. But when you have that amount of scripted tests you have a lot of scripting software to maintain, and at some point it simply did not work anymore. When there was a new iteration of the program more time was spent on the test automation tool than on the actual testing. It was a major system with three to four releases per year. That became a maintenance problem because the new version of the systems would have a lot of differences on the user interface, and thus the tests were hard to maintain.
We came into that customer and started creating the tests using keywords. We paid close attention so that the keywords could relate to business tests and domain tests. The keywords would describe the test at a higher level without showing the details of the interaction of the system under test. This made the test a lot more manageable.
How does keyword testing work?
Buwalda: Keyword testing means you write the test as a sequence of keywords with arguments. You try to keep the test and logic of the test out of the programming language in an easy to read format. Our product TestArchitect uses a spreadsheet format with test lines in which we have what we call “action keywords” with arguments. This could be something like a “login” with user ID and password as arguments.
We separate the test into business domain tests and “interaction tests”, which test user interaction. That distinction between business tests and interaction tests leads to a more maintainable and reusable set of tests. In most cases, you only have to maintain individual keywords to accommodate changes of the application under test
We use an interpreter to execute these tests. For example, for the business level action like “rent a car” the interpreter can recognize the keyword and call a corresponding function. There is also have a mechanism to create higher level actions from lower level actions. This way you can use existing actions, like “enter” in textbox, to compose more domain specific actions like “rent a car”.
How can this process be leveraged to improve integration testing?
Buwalda: I see a strong trend towards non-UI testing, in which some of the automation moves away from the user interface. If you talk about a web service for example, you would test that directly rather than through the client application. A test like that would go after the individual functions of the web service, RCP, or whatever platform you are using. It would leverage the interpreter and for example use actions to take care of emulated responses. If it is a service you want to test, you would send messages and get replies. If you want to test components that use that service, you can emulate the replies of that service. The key of our approach is that you always want to focus on what you want to test, and then write down the test action in just enough detail to reflect that scope.
What do you see as the primary distinctions between developers and testers?
Buwalda: When I talk about “developers” and “testers,” I am of course using simplified stereotypes, not necessarily the actual people. There are for example plenty of developers that can be perfect testers. But testers need to approach the application from a different point of view. The developers point of view is to create systems with many components that interact with each other using predefined behavior. A professional tester will use testing techniques, like decision tables or state transition models, to find out what could go wrong. They want to think of sequences of events and situations that a developer might not have thought about. Developers are more about composing something, whereas testers are more focused on dissecting it. If you go into a company and walk into a room with developers or a room with testers, you may find people with different mind sets and different skills.
The company might want to hire developers to do testing because they believe it is difficult for testers to access interfaces like for services. With methods like keywords however, testers can focus on the business logic and then the automation engineers can come in and automate that test logic. This allows the testers to create interesting new tests.
When I come into new customers, what I notice often is that tests are boring. They follow design documents step by step and do not try hard enough to actually break the system under test. If bugs were easy to find, they would not be there. The normal business for the system under test usually works. You don’t have to do a lot of testing to know that you can rent a car. What you need to look for as a tester are the borderline cases and a combination of sequence of events and situations that the system might have trouble with.