Hamilton Co. 911 is investigating system failure after calls went unanswered for 40 minutes
The Hamilton County Communications Center is seeking a third party report after its redundancy systems failed to operate properly in the wake of a weekend power outage.
The center reports calls from 27 unique phone numbers went unanswered during a 40-minute period before the problem was discovered.
"Within nine minutes of the power being restored, they had a complete list of all the numbers that attempted 911 traffic, and within nine minutes of the power being restored, they had made contact or at least called every single one of those numbers back to find out if if, in fact, they had an emergency or if they needed some types of services," Director Andrew Knapp reported Thursday to Hamilton County commissioners.
He says it appears none of the missed calls involved serious medical, emergency or police needs.
"We've done a lot of causal analysis, we're looking at the data that's coming in to us, and we've determined that our 911 service was impacted for a total of 40 minutes, which we consider completely and totally unacceptable," Knapp says.
The Hamilton County Communications Center is located at I-275 and Hamilton Avenue. It provides 911 call-taking services for more than 100 police, fire, and EMS departments across 42 political jurisdictions, not including the city of Cincinnati, which has its own 911 communications center.
According to Knapp, the center briefly lost power Saturday, March 11, when a power surge created a three-second outage. Employees in the building noticed a quick screen flash and the dimming of lights but the building never appeared to lose overall power.
The surge triggered the building's backup power supply called an "uninterruptible power supply," or UPS, to kick on. This is a type of battery bank that provides temporary, immediate bridge power until backup generators turn on.
"However, a key component is that (UPS) was damaged and it subsequently prohibited the system from transitioning off of the battery power. The battery depleted itself after about an hour, and we then lost most of our critical systems with the exception of our radio system," Knapp says, noting the center never lost contact with EMS units.
In short, the UPS thought the building had no power and kept running — and powering — some of the building's systems even though the building actually had power. When the batteries died, the building lost power a second time.
This second outage lasted 40 minutes.
The Communications Center has redundancy systems in place to route 911 calls to Cincinnati's 911 center if the county's system fails. In this case, Knapp says, since the UPS was depleted, calls were not re-routed.
"The nature of the partial outage meant that our workstations were down but our phone lines were still active. This resulted in the calls being stuck in a queue, if you will, within our system that we were not aware of," says Knapp.
While he says the center routinely tests its systems and backup scenarios, this incident exposed a flaw "that will be corrected."
What's being done
Knapp tells commissioners the center is engaging a third party independent contractor to help staff investigate what went wrong, how it can be prevented in the future, and possibly what additional redundancies or protocols should be implemented to prevent a repeat situation.
"We're working to identify the root cause of the service impacts, and to improve our plan to mitigate any future impacts such as this," he reports. "We routinely practice and test all of our backup scenarios and this one exposed a flaw that we had in our system that will be corrected."
According to Knapp, a board and relay in the UPS that malfunctioned have been repaired.
While the systems and redundancies are set up to function without human intervention, Knapp notes the center was fully staffed at the time of the outage and it was a veteran employee, Karen Denning, who caught the problem.
"Within about three to four minutes of the power surge that we experienced, she recognized the anomalies within our reporting systems that told us there was a problem and she made notification to off-duty staff very quickly. ... Although 40 minutes is a long time, we were able to minimize that. It could have been much, much worse had we not known that this was going on."
Knapp says the center will prepare an after-action report, too.
"As soon as we can determine the actual cause of what the failure was, we'll know better how to behave and react the next time."
He says one solution could be as simple as making a test call to make sure 911 traffic is routing as it should.