the film forum library tutorial contact

Blackout's Precise Sequence is Elusive to Investigators by Rebecca Smith, Staff Reporter Wall Street Journal - August 26, 2003

Investigators probing the roots of the massive Aug. 14 blackout must first build a precise chronology of what went wrong when. The problem they are running into, however, is that the affected utilities' clocks aren't all synchronized.

"The clock problem is a major holdup for the investigation," said Ellen Vancko, spokeswoman for the North American Electric Reliability Council, the industry group that is working with the Department of Energy and others to determine the reasons for the nearly two-day blackout. Ms. Vancko said the council, based in Princeton, N.J., hasn't been able to produce a chronology as a result.

The clocks on tens of thousands of pieces of electrical equipment scattered throughout the Midwest and Northeast aren't all synchronized, so their time stamps can't be relied upon to build an accurate chronology. Furthermore, there is a good possibility that electrical-frequency fluctuations during the period leading up to the blackout may have also thrown the time pieces off.

Investigators say it is essential that they get the time sequence right or the result will be an incorrect analysis of what happened. That could lead to incorrect recommendations for corrective action. It could also shift liability to companies or organizations that aren't responsible for the events that triggered the cascade of outages.

Several organizations have produced timelines with their best guesses about the unfolding of events, including an account by International Transmission Co., which runs Detroit's transmission system. But none of these is comprehensive. At best, they are snapshots of what took place in small areas of the total region hit by the blackout, which affected 50 million people in eight states and parts of Canada.

For many industries, establishing precise time measurements isn't all that important. But the electric industry is different. Problems in electricity transmission can develop in seconds, and spread at lightning speed. If the finely tuned system falls out of balance, it can cause protective devices -- such as circuit breakers or relays -- to isolate expensive machinery from damaging fluctuations.

One of the primary duties of grid managers is to function as time keepers. When clocks stray too far out of alignment on pieces of equipment in the field, grid operators order them to be reset. They also time-stamp information as it comes into central control rooms and computer systems that are used to manage the grid's voltages and regulate generation output.

PJM Interconnection, the grid operator for the mid-Atlantic states, keeps a central clock that is set to the federal atomic clock in Colorado. It is accurate to a thousandth of a second. So data coming into PJM's computers or instructions leaving its control center in Valley Forge, Pa., are all time-stamped with great exactitude. But there is still a problem reconciling data as the information moves between utilities and grid-control areas.

This is especially true in the Midwest, widely believed to be the origin of the blackout.

Utilities in that region still run their own pieces of the high-voltage system. They often rely on information that is recorded electronically and comes from equipment that may be reset only occasionally.

Once a reliable chronology has been created of the Aug. 14 event, investigators are confident they will be able to determine the blackout's causes. Powerful computers are used to run simulations that first try to recreate the event. Then changes are made in how the system is set up to try to bring about a controlled outcome. The goal is to figure out what needs to be done in the future to assure that the electrical system doesn't again become unstable.

Engineers always fear that a system could have a relapse before they are able to diagnose the problem and take corrective action. The grid collapse that occurred on Aug. 10, 1996 -- spreading throughout the West and lasting as long as nine hours -- was preceded by near-collapses on July 2 and July 3.

Investigators "were still trying to figure out what happened in July, when along came the August event," said Jim Detmers, vice president of operations for the California Independent System Operator, created two years after that power failure. He was part of the analysis team. In the end, more than three million simulations were run of that event.

Bill Mittelstadt, an engineer for the Bonneville Power Administration and a member of the federal task force investigating the Aug. 14 blackout, says he is confident investigators will be able to determine precisely what happened. He believes a rough sequence of events "with the most important features" will be ready in a week or so.

Right now, the task force is trying to get the time element nailed down. "You don't have to be off very far to get things in the wrong order," he says. "All the time recorders have to be reconciled."