1 Million Linux Kernels Booted for Vast Botnet Simulation

Researchers at Sandia National Laboratories have laid the groundwork for an unprecedented simulation of a large-scale botnet after booting up 1 million Linux kernels as virtual machines.

Sandia computer scientists Ron Minnich (foreground) and Don Rudish (background) have successfully run more than a million Linux kernels as virtual machines.

(click image to enlarge)

They now are waiting for completion of a new, faster and more capable supercomputer at the Livermore, Calif., lab, on which they hope to run 10 million kernels in a simulation of the open Internet -- complete with Web and mail servers, as well as simulated users clicking on simulated emails, getting simulated infections, and joining a simulated botnet.

A kernel is the core component of the operating system that passes instructions between hardware and software. To make the unprecedented achievement of running 1 million kernels as virtual machines, researchers stripped out support for extraneous devices like Bluetooth and wireless connectivity.

Managing a Challenge

They still had difficulty keeping up with all of the virtual machines, said Sandia computer science researcher Don Rudish, who worked on the experiment.

For instance, the Ethernet switch on the lab's supercomputer, called "Thunderbird," wasn't designed to recognize one million MAC addresses online.

"After 100,000, the whole network came to a crawl," Rudish told LinuxInsider. "Just looking through 1 million lines on a text log takes some time."

While the experiment was a success , Rudish said researchers didn't entirely solve the issue of monitoring the vast network, even after working out ways to visualize some of the data and reduce the amount of information flowing to them.

Virtual Botnet

They hope to solve that in the future by using botnet behavior to help control the network, Rudish said.

Unfortunately, researchers still don't know much about how botnets actually work. So, they're planning to use the lab's new Red Sky supercomputer, currently under construction, to create a 10 million kernel system and introduce botnet software into the system to see what happens, Rudish said.

Other researchers have simulated the behavior of botnets in computer models, but little is known about how they really operate. Sandia's experiment will be different because it's much closer to an actual real-world application with what will look to the network like 10 million computers online at once, he said.

"It's implemented in software, so you can say it's a simulation, but it's a much better one in that you're running real code, real TCP stacks," Rudish said.

Some of the computers will be programmed to act as Web and mail servers, others as simulated users with a percentage chance to click on incoming "emails" -- some of which will download botnet software to infect the virtual machine. That machine will then take on one of several roles in the botnet: storage server, Web server, or aggressor seeking to further propagate the botnet's control.

The experiment should give researchers more insight into how botnets work and how to combat them, Rudish said.

Project Cost

It's difficult to calculate the cost of the 1 million kernel experiment because so much of the technology that went into it was developed for other purposes, explained Rudish.

However, direct costs ran about US$100,000, he said. The Department of Energy's Office of Science, the National Nuclear Security Administration, and Sandia funded the project.