CIOs Uncensored: Security Smarts

Feb 22, 2008 (07:02 PM EST)

Read the Original Article at

"Thank goodness it wasn't us!"

We can't help it. Every time another nasty cybersecurity failure makes headlines, our eyes roll heavenward and we breathe a sigh of relief. Yet, while we have great empathy for the CIO at the enterprise that just got nailed, we know there's a bullet somewhere with our name on it.

Not just one bullet--millions of them. At Pacific Northwest National Laboratory, we deflect more than 3 million attacks on our Internet firewall every day--10% of the connection requests. During the same time, our e-mail system rejects more than 1.2 million messages from disreputable sources or because they're detected as spam. That's nearly 97% of the e-mail being sent via the Internet to the laboratory. And it's getting worse daily.

InformationWeek Reports

PNNL is a U. S. Department of Energy Office of Science national laboratory that's working to solve complex problems in energy, the environment, and national security. Our 4,000 staffers conduct fundamental research in the chemical, biological, materials, environmental, and computational sciences, and translate new discoveries into practical solutions to some of the most vital challenges facing our nation.

Pacific Northwest
National Laboratory

ESTABLISHED: In 1965 by the U.S. Department of Energy

STAFF: Approximately 4000

BUSINESS VOLUME (2007): $761 million

RESEARCH: Energy, environment, national security, and fundamental science for government, academia, and industry

PATENTS: 1,557, U.S. and foreign
That kind of work attracts a lot of interest, not all of it good. Attackers range from hackers to organized crime to foreign governments. The motives are economic, national security, or simply the challenge of attacking a government facility. The targets include intellectual property, customer data and other proprietary or business information, and personal information such as employee Social Security numbers.

Concurrent with the need to protect our sensitive information assets, we must also allow for appropriate and authorized sharing of data, scientific instruments, and computing resources with scientists around the world. Collaboration is an imperative of modern science, and IT enables effective and efficient partnering without regard to time or place. In addition, we have to ensure the availability of computing resources, including one of the world's largest supercomputers.

Cyberattacks are constantly evolving and increasingly sophisticated, making it impossible to get ahead of the perpetrators. To counter these attacks, the lab has deployed a defense-in-depth strategy with seven layers of protective measures. Each successive layer is designed to protect information and computing assets from attacks that get past the layers above.

Photograph by Mark Roberts


What you protect, how you protect it, and how much you spend protecting it are business decisions to be made at appropriate management levels. A risk-based approach to cybersecurity investments works best. For example, identifying and grouping systems that have similar protection requirements can simplify and reduce the cost of providing appropriate levels of protection. The most expensive controls need only be deployed to protect the most sensitive information and systems.

Like many companies, we have an extranet enclave that hosts our publicly accessible servers. But unlike the norm, we also subdivide our intranet into enclaves. Our three internal security enclaves are based on the sensitivity of the information resources resident in each and the assessed threats to those resources.


We use traditional network-layer firewalls to manage access on the Internet perimeter and between intranet enclaves. Application-layer firewalls scan and eliminate known malware attacks from extranet Web services and Internet mail before they reach a server or user's workstation. At this point, those 4.2 million attacks a day are filtered from reaching the inside of our network, reducing the problem to a smaller number of potential threats and concerns. All firewall rules have an associated security plan that describes the purpose and mitigating controls that are in place.

PNNL's seven layers of cybersecurity safeguard some of the world's most advanced tech research

PNNL's seven layers of cybersecurity safeguard some of the world's most advanced tech research

Requiring strong passwords is such a no-brainer that it's easy to ignore. PNNL follows the recommendations of the National Institute of Standards and Technology. We require a minimum of eight characters, using mixed cases plus a numeric or special character. User passwords are changed at least every six months, privileged access passwords more frequently.

There's an ongoing debate about the efficacy of computer-generated random passwords versus user-selected passwords. Computer-generated passwords are more secure but harder to remember, so users write them down. User-selected passwords are easier to remember but often easy to guess. If you let users select their own, you can test for easily guessed passwords using commercial or open source tools.

Two-factor authentication has been standard procedure for all remote and wireless access to the PNNL network and information resources since 1988. We employ a token-based system that requires the user to know something (a PIN) and to have something (the token) in order to gain access to our network. It's a small inconvenience, but one that provides enormous added protection against keyboard loggers and session replay attacks.


In medieval times, the European nobility constructed great castles to protect their wealth. They cleared fields around their castles, dug moats, erected rock walls, and installed thick wooden doors. But these protective measures were effective only if they were maintained properly. If the fields became overgrown, they provided hiding places for invaders. If the moats were allowed to dry up, or crumbling mortar not replaced, those barriers became ineffective. And if they forgot to raise the draw bridge or bar the door ...

The importance of effective configuration and patch management of all devices on your network is similar, from border routers to data center servers to user workstations and even networked printers. "Effective" means automating patch and configuration updates using tools such as Microsoft Windows Server Update Services, adopting "least-user privilege" policies to reduce the risk of users introducing security flaws, and continually monitoring the network for vulnerabilities.

While some companies deploy "locked-down" systems, our environment requires a level of flexibility to meet customer requirements. This may include allowing employees to install and configure software for bona fide business needs, such as a particular software package for a research project. PNNL's configuration management efforts strive to provide appropriate mitigating controls around those systems that need to use nonstandard configurations. For instance, we've deployed capabilities that allow users to raise their privileges temporarily in order to install software.

But this flexibility raises risks of vulnerabilities creeping back into the network. To guard against that, the lab monitors the network for workstations that have unpatched software or unsafe configurations.


Our e-mail and Web application firewalls filter an enormous number of virus and other malware attacks, but that's hardly sufficient. Malware can enter your network though other channels--a Web site or a USB thumb drive, for example. Workstation and server virus detection is necessary to protect against those threats. Ours is configured to automatically scan files whenever they're opened, copied, moved, saved, or otherwise accessed. Virus signatures are updated on all workstations as changes are released by the vendor, and disks are scanned regularly to search for any previously unknown viruses lurking on users' systems.

As protection against the insider threat, and to reduce the risk of malware brought into the intranet on a single workstation from spreading to other systems, we supplement our network-border and application firewalls with host-based firewalls. These play an important role, not only in intrusion prevention but also in detection. Host and border firewall logs and data from other sources are captured and combined into a central log-analysis database. These data are analyzed in real time to detect intrusions that have known signatures, such as malware that infects a workstation and then "phones home" to certain IP addresses. The data also is used in forensic analysis of intrusions using unknown attack vectors, allowing responders to identify, isolate, and clean up compromised systems.


Sometimes the crusty outer layers of defense are simply missing, such as when employees are away from the office with company laptops that contain potentially sensitive information. PNNL requires that all mobile computing devices use full disk encryption. This includes not only laptops but smartphones, PDAs, and USB thumb drives as well.

There you have it. Three outside layers of defense, and three inside layers. But wait--I promised seven layers.


One of the best investments the lab makes is our aggressive cybersecurity awareness program. Users are the weakest link in any network security effort. People make mistakes and can override controls. Conversely, people are your most effective tool for identifying problems.

New threats may not yet be recognized by technical controls, and making users aware of these threats is essential to any effective cybersecurity defense. This is particularly true for threats that seek to bypass other barriers through social engineering, such as phishing attacks or attacks that lead a user to a Web site that contains malware. Few Nigerian bank-type scams get through our e-mail spam filters, but it's much more difficult to filter attractors aimed at the intellectual interests of our scientists and engineers.

In the past two years, we've seen increasingly sophisticated "spear phishing" attacks. Messages are made to appear authentic by taking officials' names, logos, and other information from public Web sites. The messages are personalized to the recipient using information from published research papers, grant proposals, bibliographic searches, and other public sources. Appealing to their research interests, the reader is encouraged to click on an embedded link to a Web site that ultimately attempts to download malware. If the malware exploits an unknown vulnerability--what we refer to as a "zero-day" attack--our intranet can be compromised.

The good news is that the lab's user-awareness program has had a measurable impact. Witness a recent phishing message generated by a security review team. PNNL had a less than 1% response, compared with a 15% or greater response typical in other organizations.

Most cybersecurity training, if there is any at all, is a one-time, read-content-and-answer-questions course. PNNL has gone beyond that by developing a multifaceted awareness program that includes interactive online training as well as paper and electronic topical awareness campaigns.

The course is conventional in that it walks the learner through several pages of content about such topics as malware awareness, appropriate uses of computing resources, and password management. Unlike common read-content-and-answer-questions methods, though, an animated robot named Cybot conducts the interactive online class. To ensure that the content is understood, the training incorporates exercises that present practical, realistic situations. For example, to recognize spam or phishing messages, Cybot takes the learner to a mock e-mail in-box containing several messages. The learner chooses to open or delete messages just as with a real e-mail in-box. Staff report that they actually enjoy the session, which lasts about an hour.

All new hires complete the training as part of their initial orientation. A mandatory annual refresher reinforces the original messages and keeps all staff up to date with changes in the cybersecurity landscape.

Paper products consist of fliers, newsletters, posters, and postcards. The fliers or newsletters are conventional; the posters and postcards aren't. Postcards with a graphical image on one side and text on the other are "mailed" to every staff member at work. Matching posters are placed in copy centers, lunchrooms, and other common areas. Topics pertain to such issues as phishing, peer-to-peer software, and cleaning data off hard drives before transferring equipment.

Electronic communications add to the mix with directed e-mails and all-staff newsletters. E-mails and newsletter items have included such issues as phishing, spam, not sending sensitive information to shared printers, and peer-to-peer software.


The beneficiaries of the laboratory's cybersecurity program are our clients and partners, whose information we're entrusted to protect, and PNNL, which is able to attract and retain these customers by demonstrating a trustworthy computing environment. While it's difficult to measure the impact of a cybersecurity event, there's no doubt that it's expensive to perform the forensics, repair, and cleanup necessary to return a compromised system to the network. Worst of all, the impact on an organization's reputation can be incalculable. You can spend an infinite amount of money and you'll never be perfectly secure. But taking a smart, risk-based approach that includes defense-in-depth tactics tailored for your organization will reduce the risk to the information with which you are entrusted.

-- Jerry Johnson is the CIO of Pacific Northwest National Laboratory.

Continue to the sidebar:
Action Plan: Pacific Northwest National Labs Recommends Regular Check-Ups