To Avoid Being Bitten, Developers Run to SWAMP

In order to help improve the security of complex software development projects, the U.S. Department of Homeland Security (DHS) has funded the creation of the Continuous Software Assurance Marketplace, or SWAMP.

The project provides a one-stop shop for application developers to test out the security of their software early in the development process. Barton Miller, chief scientist at SWAMP and professor of computer science at University of Wisconsin, Madison explains how this free service promises to improve software security. What are the goals of  the SWAMP project?

Barton Miller: The SWAMP project has several goals. The highest-level goal is to bring together software assessment tools, applications and system programs and provide an environment where it is easy to provide a large variety of software assessment tools for many application packages.

As a software tools developer, you can get results and compare your results with other tools. Tools developers can run against a suite of about 500 test cases to get a reference point as to how the tool is performing. We have also developed a bunch of reference applications with known flaws built into them. These are either real application that have been precisely analyzed or specifically generated applications like the Juliet Test Suite. Tool developers can leverage these reference packages, and the automation to help ease the testing of their tools.

Educators teaching security classes can have immediate access to a whole suite of analysis tools.

Software assurance researchers can study the database generated from the various tools used in the SWAMP.

People dealing with software supply chain issues can identify the security of libraries they intend to use as part of their software development process. What are the expected deliverables or services and timeline from SWAMP?

Miller: We currently have the ability to upload, C, C++ and Java code and run it against a variety of open-source tools. The SWAMP provides a way of merging results from the tools into one nice graphical interface that unifies the reports.

We are getting ready to release tools for scripting languages like Python, JavaScript, PHP and Ruby. As part of that we will have automation and internal structures for handling the scripting languages that are interpreted rather than compiled.

We will also have plans to release commercial tools onto the SWAMP by the end of the year. How do you automate the process of security testing?

Miller: When you run an assessment, you will get a combined merge result that includes the best information for all of the tools we have. This multi-tool approach has advantages. One is that you can more easily use tools for identifying specific kinds of vulnerability. Bringing a tool like that to market is hard because few organizations want to bring another tool into their software development workflow for another report. With the SWAMP, you need to learn just one workflow, and when a new tool arrives, it does not change the way a programmer has to use them.

If you are a tool developer, you can bring the tool into the SWAMP. Commercial software packages vendors have done work on automating the process of assessing new code. This is not a thing that a lot of the open source tools have done. Tools developer can bring a new tool to the SWAMP and quickly get a software package that automates much of this process. How do you can you make these services available in a way that respects the competitive advantage of security tools vendors?

Miller: The initial tools deployed in the SWAMP are open source. We also have several commercial tools in the pipeline. The swamp can create visibility into the use of these tools. The commercial tools will run side by side with the open source tools.

On average, commercial tools are stronger, and every tool is good at different things. There is no one tool that covers everything. We are trying to create an environment where you are not as interested in listening to each instrument as to the whole orchestra. We are trying to bring together a merged combined result. It may be that the combined results of open source tools will outmatch the commercial tools.

A few security tool vendors see the value of providing more visibility for the user of their tools. There will be a lot of companies that will not upload code to the SWAMP because they cannot release it from their facility due to security concerns. The SWAMP allows them to see and experience commercial tools before they buy them.

There is also the advantage that running commercial tools in the SWAMP makes it easier to integrate these tools with others. We are setting up licensed service management that will allow people to come to the swap with tool licenses using a bring your own license model. This will allow a developer to take advantage of the multiple best of breed tools as part of a single software assurance workflow. How does the SWAMP help to improve the configuration of various security tools?

Miller: We spend a lot of time trying to figure out good settings and get feedback from the users so the tool gives effective messages that don’t overwhelm the users. When you run one of these tools there are a lot of options and filtering of output. If you turn on a tool and get every report, there tends to be a lot of noise. This is true of commercial and open source tools. We try and minimize any report that will just be stylistic and not have information that is essential to you. What are the plans for doing dynamic analysis of software?

Miller: Dynamic analysis features will be coming next year. This means a variety of different things. Over the next couple of months, we will come up with a vocabulary to name these in a clear way. The first ones on the SWAMP will deal with extra checks for memory buffering and storage problems.

Other dynamic tools run on a separate process and connect as a client to the Web server and attempt a range of queries to stress the Web server. How does the SWAMP fit into the Software Development Lifecycle?

Miller: Continuous assurance is a natural extension of continuous integration. The idea of continuous integration is every time you do a major update, you do a regression test against the rest of the code. The idea of continuous assurance is that every time you make an update, you do an assessment run. As you do these commits and builds you get results back from the assessment run.

There are several things we are doing to make this smoother. Every time you reach a stage where you are going to commit code to the repository, you can trigger an assessment run. Something we are working on now is a way to pull code automatically from the repository so as you commit to the repository that will trigger a pull from SWAMP.

The other thing we are doing is integrating with the various IDEs. We have done a prototype with BlueJ, and will be integrating with Eclipse and IntelliJ soon. We have demonstrated the ability to push a button on the IDE to trigger an assessment. It is important we inject this assurance at each development stage, and we are trying to apply the automation to make this process smooth. Do developers have to pay to use the SWAMP?

Miller: We currently have a five-year funding commitment from the DHS to stand up an open facility, and we are a year and a half into that. Right now any developer can come in and upload a package, and we provide the hardware, software, and staff to help. In year six, the DHS could renew their commitment for another term, or we could move to a paid consortium mode where we get companies to fund us. There is also some talk of a fee for service model. What are the main challenges with analyzing possible vulnerabilities that can arise from integrating software and services together?

Miller: We talk about weaknesses and vulnerabilities. Weaknesses are sometimes called flaws or bugs, such as using a number wrong or over-writing a buffer. A vulnerability is an exploitable weakness. Static analysis tells you weaknesses in the code, but cannot tell you to what level a hacker might be able to attack you.

The integration problem is important, and there are a number of approaches for identifying them. One is that we want to make sure the common components like open sourced libraries and commercial libraries get assessed before development starts. In many cases, organizations use these libraries to create this incredible software supply chain problem. We want to make sure as many of those packages as possible are actively assessed in the SWAMP. Right now we have created over a hundred curated packages that include Apache Web servers and open source libraries.

Another piece is that there are bugs that only show up when you combine pieces together as a whole program. There are not many tools that simultaneously analyze all of the code in a program, and it is a challenging problem for compiler technology.

There may be two components that look OK when separate, but when you look at the interactions between them you find flaws. We now allow you to see how each component fairs, but will bring in tools to see how components work together. How does software assurance relate to software quality?

Miller: Software quality is about maintainability, clarity, and reliability. There are a lot of components to software quality, and assurance is just one of those. Continuous integration is a software engineering process of frequently checking in code for regression testing at certain intervals in the development process. Continuous assurance is a process for analyzing the security of software in the same manner.