EXPERT ADVICE
Vulnerability and Penetration Testing: What's the Difference?

We don't typically stop to think about it, but we all need feedback in order to succeed. Imagine, for example, trying to cook a meal without periodically tasting it to see if the spices are right. When it comes down to it, trying to do anything -- even the simplest of tasks -- is much more difficult without empirical validation ("feedback").

Of course, when it comes to IT, it's sometimes very challenging to obtain the kind of feedback that we need to be successful.

Since most IT shops aren't profit centers, the measurements and observations that our business partners use to help "steer the ship" (such as profit/loss information) aren't always applicable to us. Add to that the fact that some areas within the IT purview are harder to test than others, and you have areas that are extremely difficult to validate performance-wise.

Help From Outside

Probably some of the hardest things to test within IT are issues related to security -- more specifically, vulnerability management tasks such as patch management and system hardening. Not only can it be damaging to attack systems in production, but many IT shops don't have personnel with experience to perform these attacks in-house. After all, we usually hire technical resources on their ability to keep system running -- not their skill in breaking into them.

As a consequence, many firms look to outside partners to help in this area. However, if an enterprise turns to an outside vendor to help out, how do they know what type of service they should ask for? Do they need a penetration test? How about a vulnerability assessment? Add to the mix that some vendors and consultants use these terms interchangeably, and many IT folks can find themselves in some very confusing discussions.

Both of these services fulfill two very different roles and help provide "feedback" on two very different sets of things. The goal of this article will be to lay out what each of these two services are, what they're useful for, and what enterprises might want to consider when favoring one over the other.

Vulnerability Assessment

A vulnerability assessment is a service designed to analyze the hosts in scope and find areas where attack might be more likely to occur, without necessarily exploiting the issues located. Specifically, a vulnerability assessment will typically involve investigation of the machine to determine whether current patches are applied, whether the system is configured in a manner that makes attack more difficult, and whether the system exposes any information that an attacker could use to gain leverage against other systems in the environment. Most vulnerability assessments will use a number of commercial and proprietary tools to minimize false positives, to provide action items on how to close the risks located, and will make suggestions about things the IT shop can do to make sure that the issues located don't resurface.

The inherent advantage of a vulnerability assessment is that the enterprise is looking at large number of systems and getting feedback on each of them in turn. In other words, at the end of the process, the enterprise will ideally have some idea of the risk of attack for each of the systems surveyed using known attack methods and techniques.

The inherent disadvantage of a vulnerability assessment is that since the actual attacks are not performed, it can sometimes be difficult to simultaneously test incident response procedures and/or other mitigation controls that might be in place in the event of a successful attack (e.g. logging and auditing methods).

Generally speaking, vulnerability assessment is a useful activity for shops that want to evaluate the processes/controls that they have in place for patch management, for secure configuration of hosts and, to some degree, security associated with system administration processes. To maximize the value of the information obtained from a vulnerability assessment, it is useful to specify to the vendor or internal team doing the work what types of feedback you are looking for and what metrics are important to your organization.

For example, if you have conducted a vulnerability assessment in the recent past, it can be useful to get data about how many vulnerabilities are "new" (meaning, not in the previous report); on the other hand, if you have multiple environments in use (e.g. UNIX and Windows) with different administrators and procedures, you may be interested in the breakdown of vulnerabilities in each environment.

Penetration Testing

A penetration test is a service designed to simulate an attack on systems within your or a partner's environment. While there might be a number of parameters that can change to determine how the attacks are initiated and conducted (for example, you might choose to limit the date and times when attacks are done to minimize impact to production), the defining characteristic of a penetration test is that individuals will be actively attacking the systems in scope using the same or similar methods to what an actual attacker would use.

While this might involve some type of analysis of each of the hosts in scope (for example, to look for vulnerable hosts to attack), a specific host-by-host analysis of each of the machines in scope is not always communicated back to the client. Some vendors may choose to make this information available whereas others might not.

Penetration can be done in a "black box" manner, where no information is provided to the testing team, or specific information might be given to the testing team to give them a "jump start" and make sure that they are testing the right things.

The advantage of a penetration test is that it demonstrates how critical topics like patch management are to the organization. For example, nothing "makes real" the importance of patch management than a screen capture of company data being viewed by an attacker due to a missing patch. On the downside, you prove only that your organization is subject to attack and you only find out one way (out of possibly many) that an attacker could exploit your systems to gain entry. While you do have the opportunity to test incident response and logging/auditing features of the environment, you only test them in a narrow subset of the environment.

Generally speaking, penetration testing is useful in the later stages of a vulnerability management process to validate that nothing has been overlooked. It's also useful to demonstrate the impact of potential issues. To get the most out of the testing process, it is useful to scope the test carefully based on what you are most interested in getting feedback about. For example, if you are most interested in whether attackers can get access to production data via a QA environment, it might help reduce costs to provide addresses of those systems rather than selecting a black box approach and hoping they find those systems.

At the end of the day, both vulnerability assessment and penetration testing are useful, but they are not interchangeable. Which approach to use should be selected carefully and with an eye on how you will use the results. Like anything, it is important to communicate the objectives with a high level of clarity and detail to the individuals doing the work. Having this information available will help them make sure that what they deliver best satisfies your ultimate goal.

Ed Moyle is currently a manager with CTG's information security solutions practice, providing strategy , consulting and solutions to clients worldwide, as well as a founding partner of Security Curve. His extensive background in computer security includes experience in forensics, application penetration testing, information security audit and secure solutions development.