App development teams do a lot of things right, but one thing they often struggle with is accurately predicting how their products will be used in the real world. They posit educated guesses based on prior experiences, about a group of common use cases, obstacles, errors, performance profiles and so on.
Well, we all know that bad things happen when you ASSUME, right? They’re not assuming because they’re bad people, but because you have set deadlines and budgets and they have no way of testing for every case. They have to make choices.
This has been happening in the software world for years. We develop an app on a set of assumptions that turns out to be completely wrong, or at the least falls apart as the world around us changes. For example, you test an app to stand up under customer demand X, but in reality demand turns out to be 100X. Or, you model for a dependent third-party system — let’s say a system that must provide your app with product availability information — to respond in time Y, but that response time is really 100Y.
This happens all the time, and for more reasons than we can count here. There’s simply no way to predict every possible outcome.
That’s the key virtue of Service Virtualization: The guessing games end, because your team has the ability to test for an unlimited number of use cases, response times, web traffic, on and on. The possibilities are virtually endless, and the results are better software, faster, and at lower cost.
This excerpt from the book Service Virtualization: Reality is Overrated, helps explain why testing this way makes so much sense:
Informing Performance from Production
In order to keep software projects moving in preproduction, developers and testers make guesses about what happens in production. They might try to guess what users are most commonly doing with an application that is consuming their component’s logic, estimate the web traffic on Black Friday, or guess how fast the SAP financials system might respond to a query. The industry of software really needs to innovate and become more scientific here.
With SV, we can capture and simulate the performance and response rate of the back end, as well as measure a profile of the requests that we would typically make of it — the load patterns, scenarios, and types of data we are pushing toward the back end.
When we “shift-left” for quality, we want to do so for performance as well to ensure more reliable outcomes at much lower repair and adjustment costs. SV creates the environment for performance testing, but its response behavior is also informed by system logs and other tools such as load testing and Application Performance Monitoring (or APM) tools that can continuously export useful data from production for this profiling.
This process is called Production Data Mining (or PDM). A great example would be from a major electronics retailer that was planning to upgrade the Order Management System — from OMS version 2.0 to OMS version 2.1 — for its many stores’ use during the upcoming holiday season.
Instead of making estimates, it uses SV to capture the performance profile of the underlying systems on an actual business day — the busiest day of the year, in fact. Then as it builds the new OMS 2.1 release, it can be certain to take into account a range of possible response times (and possibly timeouts) from those systems.
As you see in the figure below, SV can capture a a livelike replica of a dependent system and then throttle the typical reaponse times that are likely int he real world, when systems are under certain loads. The front-end load can also be similarly captured and profiled for very accurate simulation of traffic for a given time period.
In addition, why not also capture the front end of that scenario, using the observed load pattern from Customer Service requests on that day and feed that data into a load testing solution as a realistic simulation of that front-end interaction? Using PDM with SV provides a real “performance sandwich” around your new OMS 2.0 in development. We gain an early and very efficient way to prove performance of the new system, at a fraction of the infrastructure cost of setting up real environments.
Adding Another Wrinkle
There’s one more interesting wrinkle we can exploit here, as bottlenecks don’t just occur in our own System Under Test (SUT). By observing data of when and how downstream components break or run out of capacity in production, we can set that value in their corresponding VS like a “redline” on your engine, where it goes into the danger zone when pushed too hard. With this behavior in place, you will already know if your application is likely to push a dependency out of its comfort zone and account for it.
Using PDM and SV together to understand performance thresholds allows us to become even more proactive about understanding how to improve end-to-end responsiveness.