Oct 31, 2003 (06:10 AM EST)
IT Vendor's Staff Battles To Save Headquarters From California Fires

Read the Original Article at InformationWeek

Staff at ADCS, a San Diego IT services firm, spent a day and a half personally battling fires threatening the company's headquarters.

Led by facilities manager Bryan Wilkes, seven facilities staff fought the fires all day Sunday, Sunday night, and most of Monday. They used garbage cans full of water, hoses, and fabric to wet down the building and surrounding parking lot and grounds, and put out embers as they landed. The fire lapped right up to the building parking lot, about 100 feet from the building itself. Flames shot more than 30 feet in the air.

"It was scary, but it was sort of exciting at the same time," Wilkes said in an interview on Thursday. "We were doing everything we could to make sure nothing would happen to the building. At one point, you could look around to the west, to the east, to the north, to the south, and there was fire all around."

Only two staff were injured; both received minor burns that did not require hospitalization.

Within a few hours of Wilkes' arrival on the scene Sunday morning, the fires were getting very close to ADCS. "We could see flames and feel heat. The winds picked up, we probably had 50-to-60 mile-an-hour gusts," Wilkes said. That was around noon on Sunday.

"I don't know the exact time there was the first explosion and the neighborhood starting losing power," Wilkes said. He saw neighboring buildings lose power; power fluctuated at ADCS but it remained on. Wilkes said he still doesn't know what the explosions were.

Wilkes saw three electric poles catch fire. One of them eventually burned through and fell, which pulled down another one. "There was another explosion. We watched the voltage go down the line--there was a huge spark--and then the power went down," he said. The electrical poles were about 200 yards from the office building.

The ADCS building stands on a ridge overlooking a valley filled with residential homes. When the electric poles went down, Wilkes could see the power go out at all the homes in the valley.

The adventure started for Wilkes hours earlier.

"I got a call Sunday morning at 8 o'clock from the owner's wife," he said. ADCS is owned by Brent Wilkes, Bryan's uncle, whose wife is Jina Wilkes. She asked him to turn on the news, where he saw the fire was threatening the area of ADCS. He immediately called everyone in his department and asked them to report to the building.

ADCS has about 100 staff located at its headquarters in Poway, Calif., an inland suburb just north of San Diego. The company is in multiple businesses: It does document conversion and warehousing, scanning in paper documents, extracting the information from the scans, and cross-referencing the information in usable form. Its biggest project was archiving 1.2 million engineering drawings for the Panama Canal in 1999. Sister companies that have been spun off from ADCS include a marketing company, Group W Media; Mirror Labs, an independent research and testing laboratory; a catering company and transportation company.

ADCS's offices are in a nondescript high-tech office building that stands shoulder-to-shoulder with similar buildings, each surrounded by about 100 feet of parking lot. The street is the same as thousands like it all over suburban California. The building is low and wide, with lots of stone and mirrored glass on the outside, and a fountain in front that was switched off on Thursday, presumably to help preserve water for the firefighting emergency.

When Wilkes and his staff arrived on the scene on Sunday, the first thing they did was turn on all the sprinklers to wet down all the vegetation. They had brought in weed whackers to cut down vegetation in case that became necessary--it wasn't. "We basically sat and watched a while," Wilkes said.

Brent Wilkes was stuck at his home in La Jolla, another San Diego suburb south and west of Poway on the Pacific Ocean. Firefighters had shut down the roads from La Jolla to Poway as the fire had come between the towns. In Scripps Ranch, a town between La Jolla and Poway, 300 houses were destroyed by the fire.

Jina Wilkes had already arrived at the office when Bryan arrived. While Bryan and his staff protected the building, Jina went back and forth from the building to nearby firefighters to get updates, and she and the facilities staff kept in touch on Nextel mobile phones.

Natasha Jones, an administrative assistant for the facilities department, was also on the scene, monitoring news broadcasts and providing updates to Bryan Wilkes and his team. She sat in an area the company calls "the Pavilion"; it's a huge open room, where chairs can be brought in to seat 300 people. On Thursday it was vacant, except for a single folding table and a couple of chairs, where Jones had sat watching the Pavilion's built-in video system. One wall of the Pavilion is occupied by a square of video monitors the size of a screen at a drive-in movie theater, usually used to deliver business presentations, but now being used to carry all the available news channels simultaneously.

Cliff Rittel, ADCS's director of information technologies, monitored the company network and data center via remote access.

The company has locations in Panama, Hawaii, and Chantilly, Va.; the corporate headquarters hosts mail and telecom and other IT functions on 14 racks of servers, with 32 terabytes of data and 20 Web sites. "We were concerned about going down because that would knock out our whole business," Rittel said.

The data center runs on a central uninterruptible power supply system from APC. Batteries can support the data center for two hours. During the crisis, the APC recorded 54 separate power fluctuations, either surges or powering down, but the APC system handled them all. The company has a natural gas-powered generator in case the county electrical supply is cut off, but San Diego Gas & Electric cut off the natural gas flow soon after the power started.

"From an IT perspective, our big accomplishment was we had zero downtime," Rittel said.

The company mirrors 4 terabytes of current data to its Chantilly, Va., location and a Level 3 data center in downtown San Diego, and stores three weeks of data at an Iron Mountain repository.

While Bryan Wilkes worked at ADCS, his neighborhood at home was evacuated twice. Wilkes lives in Escondido, about 25 miles from ADCS.

"I got my dog out and some personal items. If I was going to lose one place, I'd rather lose my house than this building, because there's more people affected by this building," Wilkes said. "If I can't work, I can't pay for the house."

His wife, who is pregnant, was in San Francisco visiting a sister.

"She didn't like my being here," he said.

Wilkes attended school to get training as a firefighter, but never followed through with that career.