A Simple Return on Investment Calculation for Service Virtualization

Business decisions are run by numbers, and these numbers are often compared over time. One of the most important values for making decisions is Return on Investment, or ROI, and it can be a very powerful tool to track value.

I do this all the time when I see percentage discounts at the supermarket. It’s a way to put a relative measure to things that would otherwise take more effort without yielding much benefit. I’m perfectly happy taking a few seconds to work out in my head that one can of soda costs roughly $0.50 instead of taking out a calculator or notepad and determining the exact cost of $0.5491666 repeating. The difference is negligible in this and similar situations where and rounding and estimating get me close enough since speed and simplicity are the priority.

I can hear you thinking, “Great! So, I can find the best deal on sugar water, but how can I use this to show my manager all the great work my team is doing with service virtualization and continuous delivery?” Well, management loves numbers, so let’s give them some!

We’ll use some rounded numbers for our example. I’ll walk you through an example calculation and explain the assumptions, but if you’re like most people and would rather poke your eye out than do your taxes, feel free to skip to the link to the ROI calculator in the post script.

SV-ROI-1

 

Engineering cost assumptions

Let’s start by calculating the average cost of one engineering hour. You can of course calculate separate software developer or QA engineer costs, but I’ll combine them for simplicity. We can assume their average salary combined is $100,000 and they work eight hours per day. We all know there are 365 days per a year, but excluding weekends and holidays there are only about 250 working days per year (don’t get me started with the Gregorian calendar, leap years, and working nights and weekends to launch a major release). The rest of the time you get to spend with friends and family, mowing the lawn, and ferrying your children from one game to the next.

SV-ROI-2

Done! Not too bad, huh?

So far, we have:

Hourly cost per full-time software or QA engineer $50/hr

Bug fixing costs

OK, so, what about the average cost to fix a defect prior to release? How’d you get that number? Well, there’s no “right” answer. There are several studies and published reports and of course we have recent industry surveys. The numbers vary between sources but not by as much as you might expect.

Obviously, the cost of a bug goes up based on how far down the Software Development Lifecycle (SDLC) the bug is found since it must go back to the beginning to be fixed, hence the “shift left” saying that is all the rage these days. Interestingly, the cost per bug also goes up as quality improves since less bugs are found (but this is not a reason to skimp on quality!). And don’t forget that some bugs take longer or more people to fix than others, so you see this isn’t cut and dry.

OK, then. What do we know? Well, I’m glad you asked! There’s a widely-cited study from the Systems Sciences Institute at IBM claiming that errors cost about seven times as much to fix if found in production compared to pre-release testing (Dawson, Burrell, Rahim, & Brewster, 2010). Barry Boehm explains in his book Software Engineering Economics that an error in the wild is somewhere around 10 times costlier to fix than if found during testing (Boehm, 1981). Furthermore, CA’s research into public sources has developed an industry average cost ratio per defect at each stage of testing.

This gives us an average of being eight times costlier to fix a bug in production than during testing. Putting real money behind our example, if a bug found before release takes one engineer 10 hours to fix ($50/hr x 10 hours), the cost to fix the error is $500. If this same error were found at a customer site, it could cost $4,000 to fix. This is because troubleshooting takes time from customers and multiple technical support, development, and quality assurance engineers as it restarts the SDLC.

SV-ROI-3

Let’s add this to our list:

Cost to fix a defect before release $500
Cost to fix a defect after release $4,000

 

Are you starting to see why testing early and often is so important? It’s more cost effective to test sooner rather than later!

Putting it all together

Now, after just two rough estimates, we can compile a simple ROI assessment! Let’s use SV to “shift left” and assume we found half the bugs reported in production before release when using SV compare to not using SV. We’ll add to our assumptions that we release a new software version quarterly and calculate the total savings per year. We might also say that we will find more bugs pre-release because of more thorough testing and increasing the number of test cases, coverage percentage, or otherwise, but we’re trying to be simplistic. Use your own numbers and try a few different iterations and see what you come up with! Are you with me?

Our assumptions:

Hourly cost per full-time software or QA engineer $50/hr
Cost to fix a defect before release $500
Cost to fix a defect after release $4,000
Number of defects found before to release 400
Number of defects found after release 100

SV-ROI-4

 

SV-ROI-5

Wow!!! Can that be? We reduced costs in defect remediation due to shifting left, improved staff productivity due to accessibility, and reduced overall time to market due to earlier release!

Did we really save the company $175,000 last year?

YES! Yes, we did! Roughly. Now go show your manager and ask him or her to hire two more developers with the savings so you can improve your ROI even further. Multiply the savings by the number of years you’re using SV. Don’t forget to subtract hardware and third-party licensing costs. Happy saving!

Please note, any ROI analysis is only as accurate as the data you put in. Check out our SV Benefits Calculator or use the ROI Tool in the DevTest Enterprise Dashboard to run your own ROI analysis. You can also contact your CA account representative for a more thorough analysis by our specialized team.

 

DISCLAIMER:  The numbers/percentages used to calculate savings are representative of data found across CA Service Virtualization customers and various industry studies/reports. The values expressed are not a guarantee of achievable results and will vary depending upon your current infrastructure, people, and processes.

REFERENCES

Boehm, B. W. (1981). Software Engineering Economics. Prentice-Hall.

Dawson, M., Burrell, D., Rahim, E., & Brewster, S. (2010). Integrating Software Assurance into the Software Development Life Cycle (SDLC). Journal of Information Systems Technology and Planning (Vol. 3).