Testers Find Major Open Source Packages Reliable

The four anchors of the open source world -- Linux, Apache, MySQL and Perl/PHP/Python, or the so-called LAMP stack -- proved most reliable in a study released Monday by Coverity, a San Francisco-based maker of source code analysis tools.

With the release of the study, which is part of an ongoing US$1.24 million project funded by the U.S. Department of Homeland Security (DHS), Coverity said it is establishing a new baseline for quality and security in open source software.

"There hasn't been an effective yardstick to date for generally measuring software quality," Coverity Vice President for Product Services Ben Chelf told TechNewsWorld.

Better Quality Control

Other evaluation techniques have been tried, he explained, such as cyclomatic complexity, which links quality to code complexity.

A Coverity precept is that quality should be tied to actionable defects in the code, Chelf continued. Instead of telling developers, "your code is too complex -- make it less complex," the company would rather give them a set of bugs that need to be fixed to make their code better.

After examining some 17.5 million lines of code in 32 open source software projects, Coverity found the average bug rate in those applications to be 0.434 defects per 1,000 lines of code. However, the LAMP stack was considerably better than average, with 0.290 defects.

Defect rates for the programs in the study ranged from 1.237 for Amanda -- a backup system that allows the administrator of a LAN to set up a single server to backup multiple hosts to a single large capacity tape drive -- to 0.051 for XMMS, a Unix media player.

Many Eyes, Fewer Bugs

Low defect ratios found throughout the projects analyzed by Coverity attest to the effectiveness of the open source model, according to Russell Nelson, vice president of the Open Source Initiative.

That model uses the "many eyes" approach to building software, where many developers review a program's source code in a process similar to a large-scale peer review.

"Peer review is working just as we said it would," Nelson told TechNewsWorld via e-mail. "The more popular a project and the more people looking at its source, the more potential problems are averted."

Significant Improvement

Other prominent figures in the open source community also praised Coverity's work.

"Coverity's static source code analysis has proven to be an effective step towards furthering the quality and security of Linux," said Andrew Morton, head maintainer of the Linux 2.6 kernel.

"Coverity's Prevent [software] is an invaluable tool that we've now been able to integrate into the FreeBSD Project development process with nightly source code scans," added Robert Watson, president of the FreeBSD Foundation. "Coverity's contributions have significantly improved the quality of FreeBSD source code base, which is greatly appreciated by both FreeBSD developers and users."

Problems Other Than Bugs

While Coverity's software may be good at catching bugs, that doesn't necessarily translate into making open source programs more secure, which was one of the project's selling points for DHS.

There are bugs that cause security misbehaviors, but there's a whole set of other things, unrelated to bugs, that result in security breeches, noted Jack Danahy, CTO and founder of Ounce Labs, a maker of security software in Waltham, Mass.

"There's a whole universe of security problems that are outside what's going to be checked for with a quality tool with some security smarts," he told TechNewsWorld. "We see more exposure of things like private information because of those other types of problems."

Hackers Hate Bugs

The relationship between software bugs and security vulnerabilities can be confusing to developers, according to Roger Thornton, CTO and founder of Fortify Software of Palo Alto, Calif., which makes a program to analyze the security of application code.

"I've seen code that was really poorly written with a lot of bugs in it that also had a lot of security vulnerabilities," he told TechNewsWorld, "and I've seen code that was really well written from a bug point of view also have a lot security vulnerabilities.

"That's because you can be an expert programmer," he continued, "but if you're not expert in the techniques to hack code, it doesn't matter. You're still going to make the same security mistakes that a sloppy programmer makes."

Ironically, Coverity's war on bugs may benefit hackers, Thornton added.

"A hacker that's going to use your program to launch another program on a machine," Thornton often tells developers, "[doesn't] want your program to crash while it's doing that.

"So, most bugs -- hackers don't like them either," he said.