Apr 27, 2005 (11:04 AM EDT)
Trend Micro Takes Financial Hit From Faulty Virus Update
Read the Original Article at InformationWeek
Trend Micro is still suffering financial and customer relations repercussions nearly a week after issuing a faulty virus definition file that caused customers' PCs to slow to a crawl. The Tokyo-based security firm's stock has taken a hit. An investment firm has downgraded the stock, and the company acknowledges the incident may affect corporate earnings in the second quarter.
The problem began last Friday afternoon when Trend Micro pushed out a flawed virus signature update that caused computers, primarily those running Windows XP SP2, to lock up as the software consumed 100 percent of the processor's cycles.
"This trouble was caused by insufficient work in compatibility testing of the product with the operating system before it was released," Trend Micro said in a statement.
After acknowledging the flaw, the chief executive of Trend Micro, Eva Chen told Reuters in Tokyo that she expected the gaffe to impact second quarter revenues and results. "There will be some impact, mainly in Japan," she reportedly told the news wire service. During a press conference, she said it was too soon after the event to provide details on second quarter financials, and said it might be as long as a month to generate final figures.
Investors didn't wait that long. Trend Micro shares have fallen off $2.50 since Friday. Adding to the company's struggle this week, Standard & Poors placed the company on its CreditWatch service "with negative implications."
"The additional costs needed to cope with the problems and possible need to compensate customers heavily damaged by the defect may have an adverse effect on the company's financial profile," Standard & Poors said in its alert.
While at least one customer has sought reimbursement for time lost, Trend Micro has said that it has no plans to compensate customers affected by the bad signature file. As of Tuesday, it had received no information about possible legal action, or news of corporate customers planning to pull out of contracts.
"We've been in touch with many of our customers," said Michael Sweeny, a spokesman for Trend Micro's U.S. office in Cupertino, Calif. "We're actively outreaching both through our channel partners and directly. We're very concerned that our customers know about this problem and the solution."
Even so, some customers remain hot about incident.
"We're caught in a hard place," said Scott Larsen, the IT manager for a Centennial, Colo.-based travel firm that was sidelined for nearly eight hours while diagnosing and then fixing, the flaw. "We'd change vendors if we could. But we just renewed our contracts with Trend Micro 30 to 45 days ago."
Larsen's case dramatically illustrates how one IT manager struggled with the problem. On Friday, about 5:00 MDT, Larsen was called in by his company's help desk to troubleshoot the call center's 40 Windows XP SP2 computers, which had suddenly stopped responding. "All the machines had locked up and weren't responding," Larsen said. "The entire call center floor had gone offline."
With no idea what had gone wrong, Larsen and his support team did basic troubleshooting, but found nothing amiss. Only after looking at the event logs and pinning it down to the Trend Micro updated pattern file, was Larsen able to figure out the source of the problem.
"And then, even after Trend issued an updated pattern file, the CPUs of the affected PCs were so busy that they wouldn't take the update." Using trial and error, and some of the suggestions by this time posted to the Trend Micro Web site, Larsen and his staff were able to delete the offending files and reboot the machines so that the newer, safe update could be installed. But it was a one-at-a-time process.
Larsen said it took almost eight hours, until 12:30 a.m. Saturday, to get the call center's machines up and running.
In a letter e-mailed to Trend Micro's chief executive, he claimed that the downtime had cost his company "thousands of dollars in lost business" during the nearly eight hours when travel agents couldn't complete transactions.
"With no proactive contact from Trend, no support from Trend (your support lines were closed), and no notice on the Trend site that could be found without 20 to 30 minutes of searching, there was no way to know that Trend had, in fact, issued a pattern file that completely makes the computers it is installed on useless," said Larsen.
Larsen also complained that although he subscribes to Trend Micro's e-mail alerts, he didn't receive anything related to the issue until Monday morning, nearly 60 hours after the event.
Trend's Sweeny acknowledged that an alert didn't go out until Monday, but defended his company's timing. "We posted information on the Web site Friday, first on how to deal with the problem manually," Sweeny said Wednesday. "But we wanted to be able to describe the problem more fully before we sent out an alert. We didn't want to describe a problem we didn't have a solution for."