For answers, I turned to Sandeep Shukla, a professor in the Bradley Department of Electrical & Computer Engineering on the Arlington, Va., campus of Virginia Tech. Shukla, whose curriculum vitae includes a stint as a staff design engineer at the chip maker Intel Corp.
Shukla and I spoke via email. This is the first in a two-part interview.
Given how much testing embedded software typically goes through, why do buggy technology products still reach the market with such frequency?
The answer is in the enormous complexity of such software … Formal techniques [are not easily doable] for checking all possible paths and making sure all correctness properties are checked, unless the software itself has a nice, repetitive structure or is compositionally built from pre-verified components.
However, even those are not foolproof, [because] when you compose, the composition has properties that are not easy to predict from the constituent components. There are many possible interactions.
Testing can only test a fraction of all possible execution paths, and thus cannot uncover bugs that happen under special circumstances. Also, the methods of formal verifications are expensive, not only computationally but (also because) hiring engineers with training in the formal methods is very expensive as there are not many in supply.
How can software testers work better with developers, both to contain problems in code and make software work better?
Testing should be done by developers as well at the unit testing level – and to some extent, composition testing level when their part of the software is composed of units. Also, developers should have training in formal specification, property driven test generation, or some model checking and static analysis techniques, and those should be integrated in the development process so that the developers can do a whole lot of those for their part of the system.
The testing people should be trained more in using formal approaches, and also tools must be available to them to easily use some of the advanced methods – rather than just random or targeted testing.
Also, as the developers develop a set of properties that have been proven on their components, they should provide those to integrators and testers, and then [show] how to compose those, and also develop properties of the composition should be learned.
When I worked for Intel as a formal verification engineer for the Itanium, we had a hard time talking to logic designers (akin to software developers in software companies) because they always thought that their design was defect-free. When we raised an issue, they got very annoyed and did not like to sit with the validation people to explain the design complexity.
There is a need for cultural change – where testing/validation engineers should be considered on par with developers. When many of our students interview for a development position and do not do so well in development, they are often offered a testing position. This is a bad practice because it gives rise to an impression among developers that they are superior.