Latest AV Weapons for Disarming Software Miscreants

In the early days of antivirus protection, all vendors used basically the same approach. Antivirus software scanned a computer's memory and all the files on the hard drive, and then compared them to a database of signatures that matched known malicious code.

The only real difference among antivirus software vendors was in the ability of their researchers to find new malicious code before their competitors did. How rapidly and how often vendors issued signature updates also differentiated good antivirus programs from the better ones.

Over the last few years, virus writers have taken their malicious code delivery methods to new heights, and that has forced security firms to adapt.

A Way Around

Malicious programs such as viruses, worms and trojans are now able to slip into computers protected with current signature-based protection for hours or days before researchers find them and develop removal instructions for a new signature database update to subscribers.

Known as "zero-day vulnerability," this weakness has led to the development of antivirus protection that looks at code behavior when a portion of a program executes.

Depending on the vendor, an antivirus solution will use signature-based catalogs, behavior-based monitoring or a combination of both methods.

"We know that virus writers test their codes against signature-based detection. Behavior-based methods are a necessary trade-off. Signature-based methods are still more effective with established infections. Behavior-based methods treat the newest types of infections," David Finger, product marketing manager for Trend Micro (Nasdaq: TMIC), told TechNewsWorld.

Actions Speak Louder

Behavior-based protection offers advantages over signature-based antivirus protection. To understand the differences, consider the analogy of a bank robber, suggested Brian Foster, senior director of products management for Symantec's (Nasdaq: SYMC) endpoint security division.

Law enforcement investigators may be able to use fingerprints left at a crime scene to identify a bank robber and track him down. However, having those fingerprints on file is not enough to prevent a robbery from occurring, he said.

While a bank robber is standing on line at the bank, the fingerprint information is useless in detecting his intentions. Behavior-based technology works to recognize the preliminary actions of the would-be bank robber and notify bank guards to remove him from the line.

Having the criminal's fingerprint does not offer protection until after the robbery occurs. So, antivirus vendors need another way to identify bad behavior and stop it before malicious code does harm, explained Foster.

False Results

One problem with behavior-based antivirus protection is the potential for a false alarm. A signature-based detection method matches up with a close match to a known code behavior, so such false positives do not often occur.

Yet that is not always the case with behavior-based detection systems. Depending on the type of behavior-based engine a vendor uses, detected behavior within a running program could be viewed as bad and stopped. This false positive then interferes with the program functionality the computer user intended.

"Lots of applications do legitimate things that are seen as bad -- for example, disk formatting," said Foster.

Different Behavior Approaches

Symantec has used a behavior engine strategy it calls "Sonar" since last year within its Norton AntiVirus product, according to Foster. It looks for examples of bad behavior such as outbound SMTP (Simple Mail Transfer Protocol) mail, activity that uses a one-pixel focus and key logging.

Security firm Sophos uses a new type of behavior-monitoring called "behavioral genotype." Sophos, which offers security products for enterprises rather than consumers, uses a single scanning engine for all customers' content, whether it be e-mail , programs or network traffic.

"This is a different approach from Symantec and McAfee, for instance, which use different monitoring approaches for different types of applications," Ron O'Brien, senior security analyst at Sophos, told TechNewsWorld.

Defeating False Positives

Sophos believes its behavior genotype engine gets more reliable results by eliminating the causes of false positives. Other vendors use procedures that push a more rapid identification of malicious code but often result in false positives, according to O'Brien.

Sophos solved that problem by verifying the potentially bad behavior and comparing it to legitimate code, he said. This reduces the occurrence of false positives by scanning pre-execution of the code.

"Behavior genotype looks at code and goes beyond code level to see behavior," O'Brien explained.

Playing in a Sandbox

One common strategy security vendors developed is giving behavior-based engines the ability to look at code executed in a controlled, real-time, restricted area. This is known as a "sandbox environment." Essentially, a key part of the behavior-monitoring technology is to use host intrusion protection (HIP), explained Ed Metcalf, senior product marketing manager for systems security at McAfee. This provides a run-time behavior analysis that allows programs to run while monitored.

These HIP programs use a sandbox environment to analyze behavior. If the behavior is suspicious or malicious, the HIP can block and clean up the partial installation within the sandbox.

Three years ago, McAfee integrated advanced behavior-based methods into its standard VirusScan antivirus product for desktop protection.

"The newest threats were getting through, so we added buffer-overflow blocking spyware behavior," he said.

Within the Rules

A series of rules and behavior-based policies configured into McAfee's scanning engines can block malicious programs from executing from a temporary directory or open certain ports to outside servers, Metcalf explained. Both methods are common ploys of virus writers.

McAfee combines a host intrusion protection system (IPS) with signature- and behavior-based methods into a single platform plus firewall. The security firm uses the same philosophy for both network and consumer desktop solutions.

The separate engines inspect every packet. An auto quarantine feature automatically shuts down badly behaving hosts.

"McAfee uses both approaches, signature-based and behavior-based. Network-intrusion and host-intrusion protection use multiple security engines," added John Vecchi, director of product marketing for network security at McAfee, "including signature-based methods plus new designs for behavior-based."

Without the Sand

While most antivirus vendors are now using some form of behavior-based technology, no single system is at play throughout the security industry. Although most vendors use the sandbox concept to create a temporary protected zone while monitoring behavior, Sophos takes a different tack.

Sophos does not use a sandbox environment. Instead, it looks at the smallest behavior traces without code having to execute even partially, according to O'Brien.

"This level of scanning is not available elsewhere. This approach eliminates lengthy downloads of program updates and the performance delays that often cause system managers to wait until the network is less busy. These delays pose added security risks," explained O'Brien.

Another Approach

Symantec developed a different method to protect against malicious intrusion for its enterprise customers. It is called "generic exploit blocking" or "vulnerability-based protection."

"When a vulnerability is announced, Symantec analyzes what needs to exist for an attack to occur. Then the security program analyzes network traffic for that list of known traits or characteristics," Foster said.

Bottom Line

The death of signature-based antivirus protection has been greatly exaggerated, according to computer security experts. Signature-based scanning still plays an important role in the detection and remediation of threats.

However, signature-only solutions are no longer enough. The most effective protection against the quickly evolving threat landscape is a layered security solution that integrates behavior- and signature-based protection technologies.

"Signature-based ultimately will ensure cleanup and removal" Foster summed up.