‍Santa Rosa's Municipal IT Department

With a population of more than 170,000, the City of Santa Rosa is responsible for providing high-quality public services to all its citizens. As expected in any modern organization, its IT team is essential in delivering these services.

‍

The team's duties include developing, leveraging, and implementing technology to help the local government be more efficient, transparent, and responsive. The IT department is responsible for the back-end technical infrastructure of the City of Santa Rosa.

‍

This includes:

• Backing up municipal government data and ensuring it can be restored in the event of an accident, malware attack, or natural disaster
• Maintaining and configuring the computer network for the city government, covering more than 1,280 connections across 60 locations
• Providing help desk support during regular business hours for city employees spread across 1,500 desktop units
• Supervising 1,200 government phones controlled with more than 50 Mitel MiVoice Connect switches throughout 25 locations

‍

"We operate at the enterprise level," explains Ari Piotrkowski, Network System Analyst at the City of Santa Rosa IT department. "The solutions that we support, including the phone system, are used by every department in the city, ranging from city council to public works."

‍

Facing a Natural Disaster

In October 2017, communities in Northern California’s Sonoma County and Napa Valley experienced devastating wildfires. The disaster spread quickly, burning 160,000 acres of land and destroying over 7,000 structures. Sadly, there were many casualties.

‍

In Santa Rosa, city employees and the community rallied to protect people and property. The Emergency Operations Center (EOC) was at the heart of this response effort. In the earliest stages of the emergency, the EOC staff comprised 37 people. This number was in line with the city's emergency management procedures. Among the EOC team were four call center agents and one supervisor who could respond to citizen questions and requests. This quickly proved to be insufficient.

‍

Once the hotline was publicized, the EOC received 3,500 calls within 24 hours. A backlog soon built up, leading to call wait times of up to 10 minutes and a 50% abandon rate. It became clear that more help was needed to manage the volume of calls they were receiving.

‍

FEMA advised the EOC that the hotline call center needed many more agents. The agency determined that a significant increase to 30 agents would be required within 24 hours to handle call volumes. And while there were plenty of volunteers, scaling from a team of four call center agents to 30 was daunting.

‍

In an emergency, every second counts, and the IT team knew that phone communications were crucial in exchanging time-critical information with at-risk citizens. So, the IT team prioritized equipping volunteers and reassigning city employees with the hardware and credentials they'd need.

‍

Alongside the call center team, the city also needed to be able to answer citizens' questions regarding evacuation zones. This demanded that Santa Rosa's Geographic Information Services (GIS) team of two people rapidly scale up and accommodate many volunteers, presenting the same challenge faced in swiftly scaling the call center team.

‍

“The city was a hundred percent focused on the safety of the community and preventing any additional loss of life and property damage," recalls Ari.

‍

Leveraging Brightmetrics to Guarantee Communications

Thanks to the quick efforts of city employees and volunteers, the EOC hotline proved effective and reliable. While initial wait times were averaging 10 minutes and the call abandon rate was 50%, the EOC cut the average hold time to less than a minute (a 90%+ reduction) and the call abandon rate to 0% within 24 hours. With the urgency and compressed timeframe of the crisis, this was a huge accomplishment and a great credit to the city employees and emergency volunteers.

‍

During this crisis, a significant factor in the ability of staff to meet the demand of the emergency phone line was the use of analytics. The city government had set up Brightmetrics sometime before to report on call center operations. Few team members guessed that Brightmetrics would be such a handy tool in this emergency.

‍

Providing Automated Metrics and Reports

Brightmetrics quickly adapted to monitor traffic and call volume on the EOC's public phone line. As a result, staff and volunteers had access to daily and hourly automated reports on key metrics surrounding the number of inbound calls, call length, abandon rates, and outcomes. Along with the time savings automated reporting represented, it provided critical intelligence for emergency decisions.

‍

With these reports, the EOC's leadership was able to increase or decrease call center staff numbers to meet demand. Consequently, consistent service delivery was guaranteed while ensuring that the center was not over-staffed. This freed up team members to handle other emergency tasks or meet call demand during peak hours.

‍

For example, the team saw that late-night calls were becoming increasingly sparse, so they could use a skeleton staff during nighttime hours. Most agents were reserved for daytime operations when call volume was highest. Scaling back night staff also helped ensure a steady rotation of agents who would be well-rested and alert on their shifts.

‍

Triaging Scarce Emergency Resources

This intelligence also allowed the EOC team to see the overarching downward trend in calls as the crisis progressed. As a result, the team could accurately plan when the emergency hotline could be transferred to its Sonoma County emergency team counterparts. By allowing Sonoma County teams to handle calls in off-peak hours, Santa Rosa's EOC could better use their scarce resources and assign team members to other tasks.

‍

While the initial idea of adding this load to the county's resources drew some concern, Ari and the IT team were able to use Brightmetrics reports to provide exact metrics regarding call volume. Because of this, the EOC team was confident they would not negatively impact the county's resources in attending to callers.

‍

"The county was worried they would be inundated with calls," Ari explains. "But our Brightmetrics reports showed exactly what was going on during those hours, and how many calls we sent to them."

‍

Furthermore, the EOC could use Brightmetrics to monitor network traffic and trunk utilization. At the initial phase of the fires, the EOC received 9,000 calls. Most of these were to emergency services like fire and police.

‍

Since these were high-priority calls, Ari, the IT team, and the EOC worked to ensure they would not be dropped. Brightmetrics allowed the team to monitor this usage and reroute certain trunk groups. Overall capacity remained adequate as they prioritized sources based on the top call destinations.

‍

Lessons Learned for Emergency Management

Ari, his team, the city employees, and the many local volunteers did an exceptional job in this crisis. They managed to find the equipment, build processes, and establish a communication center. Because of all this, they guaranteed the safety of countless citizens during a major natural disaster.

‍

Having learned from this experience, Ari and his team offer some advice for other municipalities that might someday face a similar emergency.

‍

1. Call Reporting Is Vital

While it’s not typically discussed during disaster drills, call reporting is critical to crisis management. Ari noted that call reporting became especially important for the call center team operating in the EOC to know precisely the call volume they were managing.

‍

Call reporting isn't just vital for allocating team members. Reporting can also help guarantee equipment's resilience during times of unprecedented demand. Brightmetrics reports gave Ari and his team a better understanding of the load on the city's Mitel MiVoice Connect system and how the equipment handled that load.

‍

When it comes to the delivery of call reporting during a crisis, automation can be critical. Ari and the team were particularly grateful for the regular, scheduled, and automated reporting provided by Brightmetrics. This enabled them to focus their time and energy on other time-critical tasks during the wildfires.

‍

2. Plan for Extra Equipment

One of the most significant challenges during the emergency was obtaining equipment. The city government did not have enough reserve equipment, such as phones, headsets, and laptops.

‍

While the IT team was incredibly successful and resourceful in acquiring equipment for every employee and volunteer, it would have been easier for everyone if they'd already had the inventory. Ari recommends keeping as many extra supplies on hand as possible. Regarding equipment you cannot stockpile, Ari suggests planning with your technology partner how they can assist you in an emergency.

‍

In addition to stockpiling and setting expectations with technology partners, Ari proposes creating contingency agreements with nearby communities and vendors. These agreements can provide one another with surge resources in an emergency.

‍

3. Practice Your Displacement Contingency Plan

The EOC setup was in a building that the city had retrofitted to withstand most natural disasters. However, fires are unpredictable, and no structure can be wholly hardened against them. So, the EOC team had to discuss the possibility of the fire encroaching on the building. They then had to create a contingency plan to decide precisely how they would handle the evacuation in that event.

‍

Ari recommends having a backup plan to save time during a future crisis. Teams should run drills in case an EOC is jeopardized. This will involve the challenging task of rapidly re-deploying communications infrastructure in a new location.

‍

A Deep Thanks to the City of Santa Rosa

Our team at Brightmetrics, based in Sonoma County, witnessed the 2017 wildfires firsthand. We want to extend our deepest thanks to everyone involved in the emergency management process.

‍

Brightmetrics is grateful for everything you did to help contain the damage and keep our community safe. Our team is humbled that we were able to play a small role in the communications process.

‍