Navy's Open Source Security Project Shines

Welcome Guest. | Log In| Register | Membership Benefits

October 08, 1998 (12:33 PM EDT)

By John Borland,

An open source security program created by a team of Navy programmers is proving to be one of the most successful high-tech network burglar alarms online.

Late last month, the Navy released an unusual warning -- attackers were probing military computers in ways that had previously gone unnoticed, coordinating efforts around the world to keep any individual series of probes virtually invisible.

Analysts had finally noticed the potential crackers' coordinated probes using the Navy's SHADOW, or Secondary Heuristic Analysis System for Defensive Online Warfare, intrusion-detection program.

"It was partly dumb luck," said Stephen Northcutt, the Navy's lead analyst and programmer on the SHADOW team. But the software's sensitivity to subtle attacks, combined with the number-crunching power of statisticians associated with the project, let Northcutt and his team of analysts tease evidence of the probes out of a mass of apparently innocuous network logs, he said.

The SHADOW software is one of a growing number of intrusion-detection tools on the market, designed to pick up and help analyze attempts to break into computer networks instead of simply functioning as a passive firewall-style siege wall.

Most of the major commercial-security vendors, such as Axent, Internet Security Systems, or Network Associates, all provide intrusion-detection programs, with support and service teams that can help analyze possible attacks.

SHADOW is different in this respect. It is freely distributed online. Like most open source programs, there is some documentation, but no official support -- although there is a huge community of programmers who have looked at the code and have written improvements and continue to tinker with the way it functions.

The software itself is the product of more than two years of work by a team led by Northcutt. The code was initially released to the public last May, and revised later in the summer after a slew of comments and criticism from outside developers.

It consists of two parts. Sensors sit outside a network firewall, monitoring normal and potentially illicit attempts to enter the network. An analysis system sits inside the firewall keeping a log of activity, and periodically putting this information in front of a human security analyst.

In the months since its release, the program has been picked up and used by several major financial institutions, universities, local government systems, and divisions of large companies that don't have budgets for commercial intrusion-detection programs, Northcutt said.

"It's very good at doing some things and not so good at others," said Allen Paller, chief researcher at the SANS Institute, a network-security research and education organization. The program can be initially difficult to use, since it requires users to program their own filters to recognize attacks or probes not included in the original documentation.

But the program's open source birth and evolution has made it strong and extremely sensitive, Paller said. "The real strength of this process is [the program] has been beaten on."

Northcutt is a proponent of pushing the open source model even beyond the development of code, at least in the security field.

Most intrusion-detection programs function by picking up unusual events -- malformed TCP or domain name system queries, handshakes between servers and clients that don't look quite right, or other signs of computer probes and attacks. SHADOW and other commercial trip-wire programs do a good job of picking up things they recognize, Northcutt and other security analysts said. But new attacks -- such as the coordinated probes spotlighted by the Navy last month -- require considerable expert analysis to spot.

"Attackers have been sharing very well inside their community," we have no equivalent to the underground magazines and other communication channels."
-- Stephen Northcutt
U.S. Navy

That's where the open source model comes in, Northcutt said. Intrusion-detection analysts can function best if information about different attacks is widely and freely distributed. The Navy site that distributes SHADOW publishes much of the information it uncovers, and distributes new filters that recognize new attacks and probes. This kind of open, widely shared information is critical for stopping crackers, but must happen on a wide scale, he said.

"Attackers have been sharing very well inside their community," Northcutt said. "We have no equivalent to the underground magazines and other communication channels."

Paller agreed. His organization is one of several that sponsor workshops where security professionals can share their experiences with their peers. SANS also runs a security-oriented mailing list with nearly 55,000 subscribers, many of whom served as SHADOW reviewers.

"Unless we get communication lines going, we can't keep up," Paller said. "Otherwise, we don't have a chance."