36 Hours… This is what it took to push our code into production.
Yes, I was up for 36 hours without sleep to push our code into production. The install that was scheduled to finish within a few hours, extended beyond what we had imagined. Our install was dependent on the install from two other teams, which were in turn dependent on each other.
The stars and planets did not align right that day, due to which, the first team that went in found issues and had to fix them. The second team who installed also had issues and had to fix them. Hence, for a really long time, we just had to be on the call waiting for the other teams to install. I had to resort to food, lots of caffeine to keep myself awake.
I later found out, that this happens on a regular basis every time there is a big install. After further examination, the reasons why this happen were plenty. Here are three I’d like to explain further.
Not Enough Testing
I don’t think there is ever such a thing as enough testing. Often, the team is under pressure to deliver poorly tested code. The developers and testers then spend sleepless nights often praying that the code behaves right in the production environment. But there is never a replacement for proper code testing.
It is not like the developers and testers don’t want to test, but usually they are faced with a plethora of challenges such as system unavailability, not having the right data, lack of testing for negative scenarios, or the inability to create boundary conditions, among many.
The ideal scenario is to have high system availability, the right kind of data and all of the other missing parts ready to go whenever they are needed. But this is sort of a testing utopian view of the world, right?!?
The solution is service virtualization. Through service virtualization, we can build virtual services, which simulate the behavior of the real service and provide high availability and the ability for teams to configure the kind of data they need. Thereby, letting the teams test in an uninterrupted fashion. This will enable them to deliver code which is properly tested and be more confident about the code they are delivering.
In all fairness, this 36 hour scenario occurred even before we ever heard the term DevOps. I know of teams who still go through something like this scenario even today.
There are organizations who have either not adopted DevOps or are still in infancy with respect to adoption. When this happened, an install process was very manual and the pace and accuracy of install often depended on the person who did the install. Often, we had a number of issues despite the checklists that were in place. When an issue happened, we used to chase people down to get approvals and get them to move the code between the environments. This was very time consuming, costly and had terrible consequences like sleeplessness.
Today, with automated deployment tools and CI/CD, we are able to bring down the deployment times and reduce human errors. This meant that once you found an issue in production, the fix could be moved to a lower environment and it could be tested, promoted between environments and deployed in an automated fashion, thereby making it much faster and less painful.
Service virtualization plays a critical role in the DevOps lifecycle by providing an environment against which, the builds can be validated as they are promoted between environments. The beauty of virtual services is that, the same virtual service that has been created for use by developers for unit testing can be leveraged for functional test (by adding more data / making it data driven) as well as for performance testing (by setting appropriate think times, service capacity etc.).
Finding Defects Later in the Cycle
A sort of a side effect of not having stable environments is finding defects later in the software development cycle. By the time, the systems are relatively stable and some testing gets done, the application would be in UAT / pre-prod phase. Hence, if they find defects, it is much costlier to fix them, since the code has to move through all of the previous phases to be addressed.
Fixing defects is much cheaper if they are found earlier in the testing phase even as early as dev phase. This can be achieved through service virtualization by such things as having a more stable environment to test, with the ability to configure data to test negative scenarios and boundary conditions.
Overall the use of service virtualization can help reduce delays, cost and provide better confidence to developers and testers and have them spend less sleepless nights, unlike what I had to do.