Lab Quickly Clamps Down on Slammer Worm

Some quick thinking by Lab network systems engineer Greg Bell helped protect LBNL, ESnet and other DOE sites from the fast-spreading “SQL Slammer” worm recently.

While the Lab had contained the situation within two hours after discovering the attack the evening of Friday, Jan. 24, other organizations weren’t so fortunate. Customers of major banks were unable to use ATMs over the weekend, a major credit card company couldn’t provide users with access to their account information and the nation’s largest residential mortgage company was knocked off-line for four days.

“Fortunately, our team did what we have come to expect from them - they performed well beyond any reasonable expectation,” said Sandy Merola, the Lab’s Chief Information Officer and head of the Information Technologies and Services Division. “Their quick action helped protect both the Laboratory and other DOE facilities, even as other institutions were brought down by this denial-of-service attack.”

According to Internet Security Systems, a vendor of a popular vulnerability scanner, the worm infected an estimated 247,000 systems. At the Lab, only a dozen systems were infected, and members of the Lab’s Computer Protection Program and ESnet (DOE’s Energy Sciences Network) credit Bell with putting up the first line of defense here.

Bell, who works for the LBLnet networking group, was at home studying for a certification test and working online the Friday night when Slammer first hit. A self-described “email addict,” Bell decided to log in and check his messages around 9:50 p.m., just minutes after the worm began its attack across the Internet. When he couldn’t connect over his DSL line, he became concerned and then dialed into the remote access server. That connection worked, so Bell decided to investigate the problem with the Internet link.

“I looked at the border router traffic and saw there was a fire hose of malicious traffic going in and out,” he said. “Luckily, the worm had a simple pattern and I was able to block the port it was addressed to. Then our team swung into action and we worked together to contain the situation.” Bell’s action prevented more Lab systems from becoming infected and kept the Lab from spreading the worm even farther.

According to Jim Mellander, the Lab’s cybersecurity incident response manager, the worm acts in a chain-reaction fashion: an infected system randomly scans for other vulnerable systems, and infects them. The newly infected system then repeats the pattern, which created an explosion of network traffic. The result is that systems are so bogged down that they can no longer function, resulting in a denial of service. The Slammer worm uses Microsoft database products created with Structured Query Language (SQL) to scan for the same products on other systems, then infects them.

Fortunately, Mellander said, the worm does not damage or disable the system it infects, but they need to be taken off the network to stop the spread.

That’s what Bell did to the 10 infected systems after putting the router blocks in place and alerting the Lab’s computer protection staff, as well as ESnet operators, about the situation.

Computer protection staffers Mellander and Gene Schultz quickly began researching the worm and also ensuring that all Lab systems, including those at the Joint Genome Institute in Walnut Creek and in Washington, D.C., were blocked and protected.

They continued working late into the night Friday, and then again the next day. By 1 p.m. Saturday, all essential Lab network services were restored.

Working in parallel with the LBNL group was a team from ESnet, the network operated by the Lab to provide high-speed networking to researchers at national labs and universities around the country and overseas.

Around 10:30 p.m. Friday night, an ESnet operator noticed that system routers were generating unusual log messages.. “We knew something was going on and we had heard Greg was working on a similar problem,” said Mike Collins, group lead for ESnet’s Network Engineering Services. “We had never encountered anything like this before, but concluded the worm was causing the problems and we started putting up blocks.”

A virtual team was quickly assembled and went to work. ESnet blocked all traffic coming in from “peering” networks through the port addressed by the Slammer worm.

“Once we stabilized the network, we checked the traffic volume at the sites we serve and noticed that several still had large amounts of outgoing traffic, so we put the same block on them and traffic levels returned to normal,” Collins said. “The important thing is that the network stayed up, though traffic for some sites was slowed down, and we were able to provide some protection to other sites.”

In all, the Lab effort also tapped the expertise of numerous other employees, including Mike Bennett, Joe Burrescia, John Christman, Jim Gagliardi, Chin Guok, Cedric Hui, Jim Leighton, Chris Manders, Roberto Morelli, Kevin Oberman, Dan Peterson, Ken Pon, Joe Ramus, Mark Redman, Ted Sopher and Clint Wadsworth.

Schultz of the Computer Protection Program, who is co-authoring a paper with Mellander and Dan Peterson on the worm attack, attributed relatively minor impact on the Lab to increasing awareness of the need for heightened cybersecurity, especially among the computer support staff who scan for vulnerabilities and ensure that they are patched during the course of their work. Had that not been the case, Lab employees may have arrived at work Monday to find their networks down, their computers unusable and even some experiments out of action.

“It could have been a whole lot worse,” Mellander said.