October 04, 2007 (IDG News Service) — SAN FRANCISCO — The federal government pulled the plug on the Web domain name used by the state of California on Tuesday, setting into motion a chain of events that threatened to grind government business to a standstill within the state.
State IT staffers were able to fix the problem within a few hours, narrowly averting disaster, but the situation shed light on what observers are calling a shocking weakness in the state’s IT infrastructure.
The story behind the shutdown, and how the U.S. General Services Administration (GSA) came within hours of shutting down the California state government’s Internet presence, is a complex one but, as with many stories on the Internet, it begins with pornography.
In early September the Transportation Authority of Marin, a 10-person agency charged with managing transportation funding in California’s Marin County, discovered that the servers that handled the agency’s Web and domain name service had been hacked and were being used to create links to pornographic Web sites.
Domain name servers are used to translate the domain names we type into our browsers into the numerical IP addresses used by computers. Together these Domain Name System (DNS) servers form a database to tell all of the computers on the Internet how to find each other. In the case of the Transportation Authority, there was one authoritative server responsible for telling all other DNS servers where to find computers operating within the tam.ca.gov domain.
The agency spent a frustrating two weeks trying to get its Internet service provider, StartLogic Inc., to resolve the problem, said Dianne Steinhauser, executive director of the Transportation Authority of Marin. Then, in mid-September, it delegated domain name server authority for the Transportation Authority’s domain to the ca.gov name server, run by the state’s Department of Technology Services, she said. That meant that the state’s servers and not StartLogic’s were now responsible for keeping the authoritative domain record for tam.ca.gov.
Unfortunately, it also meant that if an outside observer believed that the DNS server responsible for tam.ca.gov had been hacked, he might have assumed that the ca.gov name server was compromised as well. And that, apparently, is where the trouble really began for the state government.
On Tuesday, at around 2 p.m., the federal organization responsible for managing the .gov top-level domain pulled the plug on the ca.gov domain, according to Jim Hanacek, a public information officer for California’s Department of Technology Services.
The “ca.gov domain was removed as a valid address by the federal General Services Administration, who has an office that oversees the use of the .gov domain,” he said.
Only the GSA knows for sure why this was done, but Hanacek said that the problems that had been experienced by the Transportation Authority of Marin were behind the move. “Our understanding… is they were seeing signs of some redirects over to pornographic sites and that is what caused them to shut down that domain.”
A GSA spokeswoman did not dispute California’s account of what happened, but she said that her agency was looking into Tuesday’s events and “will be able to provide an update once the details are gathered.”
Security experts expressed amazement Wednesday that the federal government would undertake such a drastic move without first trying to resolve the problem with the state. “That’s hard core, given how much stuff there is under ca.gov,” said Cricket Liu, vice president of architecture with DNS appliance vendor Infoblox Inc. “Maybe they thought there was some sort of imminent threat.”
Within hours of the GSA’s move, the state began working with the federal agency to reverse the damage caused by delisting the ca.gov domain from the world’s DNS servers, Hanacek said. Although there were some isolated reports of state Web sites being inaccessible or e-mail not going through, the disruption caused by the event was minimal and things were back to normal by 5:30 p.m. on Tuesday.
Because it takes time for DNS servers to update their records, California managed to avoid disaster. But if Hanacek’s office had not taken action within 24 hours, Web access and e-mail to all state agencies using the ca.gov domain would have been cut off, Liu said. “It would have been crippling.”
Hanacek said that the state is working with the GSA to make sure that this type of event doesn’t happen again. “I’m sure that we’ll have some processes in place to make sure that the right parties get advanced notification of a significant change like this.”
