|Photo 1: A view of the failed iso phase tap.|
By Bruce Hack, Managing Member, Crown Electric Engineering & Manufacturing and Mark D. Haag, Principal Design Engineer, Nuclear Services, Ameren Missouri
At 11:19 p.m. on July 26, 2013, Callaway Nuclear Plant operation technicians performed the monthly swap-over of the dual redundant isolated phase bus (iso phase bus, or IPB) cooling fans. This monthly procedure insures that the run time between the redundant systems is equalized.
|Photo 2: The failed Phase B IPB tap.|
At 11:33 p.m., 14 minutes after the swap over, massive electrical faults caused protective relaying to initiate trip signals to the switchyard main generator output breakers, the unit auxiliary transformer feeder breakers, the 13.8 kV service busses, the main generator field circuit breaker and the main turbine. All four reactor coolant pumps and all three circulating water pumps tripped due to the momentary 13.8 kV busses under-voltages. The reactor tripped from 100 percent power as a result of the turbine/generator trip. Extensive damage to the plant’s iso phase bus at the iso phase tap point for the unit auxiliary transformer and at the main generator neutral connection box had occurred.
At 11:37 p.m., the plant fire brigade was dispatched in reaction to a smoke condition permeating throughout the turbine building coming from a cable’s burning insulation and a nearby oil collection pan fire.
|Photo 3: The faulted Phase tap.|
At 11:49 p.m., the shift manager declared an “Unusual Event” and began the process of making appropriate notifications. Plant operators continued their event response to the reactor trip by employing the prescribed emergency and operating procedures.
At 12:56 a.m. on July 27, 2013, a significant water leak was identified at the condensate polishers due to a rupture disk failure creating flooding in the turbine building. Reactor operators isolated all the condensate polishers from the control room. A report from the field identified one to two feet of water in the condensate pump pit.
|Photo 4: The damper blade lodged in the iso phase maintenance access point.|
At 1:01 pm., the control room operators notified the NRC of the closeout of the “Unusual Event.”
Workers at the Callaway Plant assembled a forced outage response team to manage assessment, repair, and establish recovery activities related to the event. The root cause team defined an area to sequester all damaged parts for evidence while establishing the scope and parameters necessary to ensure that obtaining evidence, pictures, assessments and inspections needed for the root cause analysis were exhaustive and implemented.
|Photo 5: The damaged damper blade.|
The plant’s sub-system that was damaged by the event was the main generator IPB system, which includes all its spur auxiliary tap bus and the main generator neutral connection box. Iso phase bus is the super highway that transports the plant’s high amperage electrical power from the plant’s main generator to the main step-up (GSU) transformer. The IPB system operates at medium voltage levels transporting 33,000 amps of current when operating at full power. Lower amperage iso-phase tap connections to the main iso phase bus provide power to the plant’s auxiliary transformer and excitation transformer. Iso phase bus systems are normally static, robust, low in maintenance requirements and highly reliable. In cases where power plants generate output currents around or exceeding 20,000 amps, it is common to employ forced cooling for the iso phase conductor. This forced cooling creates a composite system less static than their self-cooled counterparts. Forced cooled iso phase bus duct systems incorporate large heat exchangers (often water cooled, forced air) to remove the heat associated with the I2R losses from the large currents flowing in the IPB system. Fully redundant fans, cooling coils and associated dampers make up the cooling skid assembly, providing 100 percent mechanical backup against failure or during periods of scheduled maintenance.
|Photo 6: The damper as it was found after the incident.|
Chilled forced air from the cooler enters the “B” phase of the iso phase bus duct and splits so as to bi-directionally flow toward the main generator terminal box and towards the main GSU transformers. Air crossover plenums can be located at each end-of-run point to allow the “B” phase cool air to split and return the airflow to the coolers’ suction side through the IPB’s A and C phases. The returning air then passes over the water-cooled heat exchanger to remove the absorbed heat and moisture. Then the process starts anew.
|Photo 7: Parts of the damper.|
Callaway’s cooling skid heat exchanger assemblies employ so-called “back-draft” dampers located on the fan discharge. One of the back-draft damper blades had broken free and entered the “B” phase of the iso phase bus duct. As the blade flew through the IPB’s “B” phase, it caused repeated short lived phase-to-ground electrical faults. Ultimately, the condition cascaded into a massive electrical fault of 180,000 amps rms at both the IPB seal off bushing for the auxiliary transformer’s “B” phase tap as well as at the main generator neutral connection box. This final massive short circuit led to a turbine/generator and reactor trip.
The damper blade breaking free was problematic enough, but a single fault to ground should not initiate such a response in iso phase bus. What happened?
|Photo 8: Arcing caused by the loose damper blade.|
To assist in assessing the damage and then emergency fabricate the iso phase bus and its ancillary components, Callaway’s forced outage response team reached out to Crown Electric Eng. & Mfg. LLC of Middletown, Ohio, a company rooted in the old Westinghouse iso phase bus plant in Cincinnati. Crown Electric had previously assisted Callaway, and the team knew that Crown Electric possessed a full on-site services division offering direct user support that is tied back to its engineering and IPB manufacturing capabilities.
|Photo 9: Pitting caused by the damper blade.|
Results of Root Cause Evaluation
The root cause evaluation was performed using multiple techniques. A forensic examination of the failed components was made to determine the sequence of failures and the potential physical causes. Detailed inspections were performed throughout the iso phase bus ducts for additional evidence to support or refute various theories. Photographic evidence was combined with design drawings to allow the team to draw conclusions based on the positions of equipment and the damage that was caused by the event. A fault tree analysis was completed to determine the most likely causes of the mechanical failures of the iso-phase cooler backdraft dampers, the main generator neutral connection box, and the “B” iso phase bus duct above the unit auxiliary transformer. The extensive damage at each failure location rendered definitive conclusions about the exact failure modes impossible. Therefore, all potential causes were uncovered and corrective actions chosen in such manners as to address them in their entirety.
|Photo 10: The failed bushing.|
Organizational and programmatic factors that led up to the failure that caused the unit trip were conducted via a review of the history of the operation of the iso-phase cooling system and the main generator neutral connection box
Since the plant’s original construction, the air flow rates from the iso-phase cooling fans were higher than the design ratings of the iso-phase cooler backdraft dampers. The elevated flow rates through the backdraft dampers increased the risk of damper failure. Callaway Engineering believes backdraft dampers should have a more robust design for this intended application. One example of the damper’s weak link was that plastic retaining clips served as bushings to allow operation of the damper blades.
|Photo 11: Another angle of the failed bushing.|
Back in January 2005, flow rates were estimated to be as high as 40,000 cfm. Air flow measurements were taken in May 2007 and found to be 34,000 cfm. More accurate data on the specifics of the system’s operation would have likely pointed to the need for corrective actions to be taken in a more timely fashion.
Callaway Engineering considers a number of initial plant parameters could have had larger safety margins designed into sub-systems, including those within the main generator neutral connection box. These include increased air gaps, creep distance on the neutral grounding cable, providing a grounded shield on the neutral grounding cable, ferrous conduit parts, and removal of any sharp corners and exposed bolt ends which could contribute to the generation of corona. Additionally, the de-ionizing* screen filter that was installed between the generator output bushing enclosure and the neutral connection box was painted, eliminating its effectiveness. These collective factors produced conditions that were more likely to create an electrical fault if the neutral connection box was ever challenged by a high voltage potential.
|Photo 12: Another angle of the failed bushing.|
(*Forced cooled iso phase bus systems include a sub-assembly known as de-ionizing baffles. De-ionizing baffles are large stacks of thin aluminum metal plates spaced about an inch apart that allow the forced air to flow over their large accumulative surface area. De-ionizing baffles are grounded so that any ionized air molecules in the cooling air stream pass over the baffles giving up their charge to the ground plane. Callaway’s design placed a smaller version of this functional device between the generator output bushing’s enclosure and the neutral bushing enclosure. The effectiveness of this de-ionizing screen is considered to have been compromised due to inappropriate painting).
|Photo 13: The failed generator neutral enclosure.|
The air flow change caused during the monthly swapping of the fans carried the thrown damper blade into a position that caused arcing in the “B” phase of the IPB. The flying damper blade is suspected of intermittently causing short duration arcing, pitting and damage as it was blown around the iso phase duct during a 14-minute operational period. Arcing, pitting and associated damage was found periodically through a 40-foot section of the iso phase bus work. An oscillography from the 311/G generator protection relay captured one neutral voltage spike of 13.1 kV. This fault then clears itself as evidenced by the VN voltage dropping back to zero. The last momentary arc was caught in the pre-fault data initiated by the protection relay lockout. Neutral bus currents during these initial faults were limited to 12.8 Amps through the neutral grounding transformer.
|Photo 14: Another angle of the failed generator neutral enclosure.|
The dislodged damper blade that flew and arced along the IPB vaporized along its trajectory. Clouds of ionized gas and metal vapor – think comet tails – are winging through the IPB system, completing a round trip every 13.5 seconds. Over time this metal vapor cloud, in part, collects on the cool porcelain surfaces of both the “B” phase iso-phase seal off bushing for the unit-auxiliary transformer and at the main generator neutral connection box. These locations present cooler surfaces and are both points in the air stream system which are deadened zones not seeing full forced air flow. It is noted that there are no de-ionizers in the flow path between the arcing location and the unit auxiliary transformer connection location. Further, the neutral connection box under the generator had compromised de-ionizing baffles, which, even when new, were never designed to handle anything more than typical IPB system created ionization loads. They certainly were not designed for the amount of ionization created by electrical arcing of foreign material such as that developed by a dislodged and vaporizing damper blade.
Oscillography offers verification of the iso phase bus phase-B intermittently going to ground. This was most likely due to the continued movement of the arcing damper blade resulting in additional gas cloud generation and potentially resulting in blowing an arc down the ductwork.
|Photo 15: The failed generator neutral enclosure bushing.|
At some point, the damper blade lodged in the T-tap pocket of the seal-off bushing for the “B” phase IPB connection to the unit auxiliary transformer. The damper blade is now easily jumping out the bushings line to ground integrity. The compromised main generator neutral (in the neutral connection box) and the unit auxiliary T-tap are both co-faulting directly to ground. This ground loop completely bypasses the generator neutral resistor connection resulting in a solid line to ground fault on the generator 220 kA.
These massive electrical fault currents generated a subsequent devastating blowout pressure wave at both fault locations. The post-event inspection of the resultant damage at the generator neutral connection box and unit auxiliary transformer’s tap bus connection was eye opening. Not surprising, the pressure waves induced catastrophic failure of both sets of backdraft dampers. Multiple blades were dislodged; some well bent inwards towards the fan and some simply fell into the fan after breaking off. Four of the blades in the duct were found with minor arc burns from having gone down the duct towards the generator while it was spinning to rest and still providing some fault current. Two of these blades worked their way across the duct work and landed down the outlet of the off-line fan. Multiple off-line damper blades were found bent in, but none were ejected into the bus ductwork.
|Photo 16: The failed generator neutral enclosure bushing from another angle.|
At this point, the event is effectively over. As the turbine/generator winds down, the fault is no longer fed. The damper blades settled into their as-found locations with no more damage created.
The electrical faults that occurred resulted in a turbine/generator trip and a reactor trip. Such an event had the potential to challenge systems important to safety. The reactor trip was classified as “uncomplicated” and all safety systems performed properly. Repair or replacement of multiple damaged components associated with the secondary side of the plant was required prior to the safe restart of the reactor on August 16, 2013. No one was injured as a result of the electrical faults and subsequent plant event.
|Photo 17: Yet another angle of the failed generator neutral enclosure bushing.|
The Callaway incidence response team with Crown Electric inspected the iso phase bus to determine the extent of damage. Together, Crown Electric and Callaway detailed dimensions for all failed component parts and/or verified them from existing drawings. Crown Electric produced a replacement neutral bushing enclosure, the neutral copper, flex braids, replacement aux transformer tap bus and ancillary parts all in a few days, shipping serially as production was complete.
Each bus was thoroughly inspected and all identified foreign material was removed. The iso phase bus with all its support insulators was completely cleaned. Filters were replaced; bus insulators, bushings, dampers, and deionizers were provided and replaced as necessary. New backdraft dampers with modified catch screens were installed. Flow testing in the system was performed. The seal-off bushings were cleaned or replaced and tested. The iso phase bus was then Hi-Pot tested.
|Photo 18: Workers repairing the iso phase cooler.|
A Flawed Defense
The inspections that were intended as a barrier to identify blade degradation prior to failure of the iso phase cooler backdraft dampers were not sufficient to prevent backdraft dampers from being ejected into the B iso phase bus duct. The inspections were performed with a borescope and could not see enough detail to accurately identify wear of the components that were most likely to fail. For some of the potential backdraft damper failure modes, it would be improbable that there would be visible indications that could be used to predict failures.
|Photo 19: A painted de-ionizing filter.|
Iso phase bus is about the most rugged, reliable electrical sub-system to be found in any power station. Iso phase should be inspected regularly enough to insure insulator integrity and proper torque values of any bolted connections from bushings to flex braids. Bus and insulators should be cleaned on an appropriate schedule for that specific location to protect its voltage withstand capabilities. In the case of forced cooled IPB, it should be remembered that even though these coolers are attached to an IPB system, they are and should fall under the same general maintenance schedule and replacement schedule as any large motor, fan or cooler in the power station.
|Photo 20: A worker poses with new de-ionizing baffles|
Nuclear Power Internaional Issue Archives
View Power Generation Articles on PennEnergy.com