Wildfires serve as warning to tech services providers: Are you prepared?

Uli Kirkegaard was working from his home Aug. 8 in Lahaina, on the western shore of Maui, Hawaii, when the alert came in: Wildfires on the island were spreading rapidly and the town had to evacuate.

The news was expected and it wasn’t the first time that residents had been asked to leave for wildfires. Kirkegaard and his family knew the drill well. They grabbed passports, a few clothes and drove two cars to a designated safer area. However the fires were spreading so rapidly, exacerbated by winds of nearly 80 miles per hour from Hurricane Dora raging south of the Hawaiian Islands, that he had no time to go to his office on the other side of Lahaina to grab some things. All Kirkegaard and other Maui residents could do was wait to see what might be spared—and hope.

This time was different. The fires raged throughout the day and night, destroying Lahaina and killing more than 100 people across Maui. “We thought we’d really seen it before, but this was so immense. Apocalyptic I’d call it,” Kirkegaard said.

Disaster survival list – for businesses, for you

Family and business communications plan

• Write down critical cell and landline numbers in case your phone dies.
• Know where your family members or business colleagues will meet if cell towers and landline phones go out.
• Designate an out-of-area relative everyone should contact if they can’t reach each other.
• For your business, complete a contact sheet for all workforce members, including their personal e-mail address, their spouse and partner contact info and an out-of-area contact you can call if you can’t reach your employee.

Water

• Buy cases of water or fill pots and pitchers with drinking water – one gallon per day per person.
• Store 10 gallons of clean water for cooking.
• Use food-grade water containers from camping stores.
• Replace your water every six months if you’re not using commercially bottled water.
• Never use water from toilet flush tanks or bowls, radiators, waterbeds or swimming pools and spas.
• Purchase water purifiers to filter contaminated water.

Food

• Store non-perishable food you don’t have to cook.
• Include food needed for special diets.
• Store non-perishable food for your pets.

Utilities

• Know how to turn off your power, water and gas services.
• Keep tools near your shutoffs for quick action.
• Never turn on your utilities until authorized.
• Always have a professional turn your gas back on.

 Cash

• Withdraw $200 to $300 in small bills so you can make cash purchases, as credit cards won’t work if power and communications go down.

Automobiles

• Fill your gas tank when you first hear a warning.
• Don’t drain your car battery charging cell phones or other devices.
• Keep an emergency kit in your car in case you are stranded away from home.

Cell phones

• Register your cell phone to receive calls from your county’s emergency management system.
• Install weather and disaster preparedness apps and configure for alert notifications.
• Use text messages as they use less battery and get through when calls can’t.
• Photograph the contents of your house or business now.
• Store emergency contact info for family members and employees, including alternate e-mail contacts and phone numbers for their close friends and relatives in case you can’t reach them directly.
• Charge your phone and keep it charged.
• Pack your car charger and home charger if you evacuate.
• Extend your battery by turning off WiFi and other services you aren’t using.
• Use solar chargers when possible.
• Learn how to set up your cell phone as a personal Internet hotspot.

Computers

• Back up critical data onto external hard drives or the cloud.
• If you have to evacuate, take critical equipment with you or seal it in garbage bags and store it in a high spot to protect against water damage.
• Pack your laptop charger if you evacuate.
• For businesses, do a test-restore of your backup files to ensure that the backups will work during an emergency and all critical data is backed up.

Generators

• Use gasoline or propane generators, as communities may turn off natural gas service prior to a disaster.
• Plan your needs to be sure you can power heating, refrigerators, lights and television for seven days.
• Store gasoline in safe containers with fuel stabilizer. Replace stored gas every six to 10 months. Generators can use seven to 10 gallons of gasoline per day.
• Buy a siphon to borrow fuel for your generator from cars and mowers.

Evacuation

• Have a backpack or overnight bag pre-packed with clothes; energy and meal replacement bars; emergency blankets; sanitation and hygiene items; photocopies of identification and credit cards; special-needs items such as prescription medications, eye glasses, contact lens solution and hearing aid batteries; and items for infants, such as formula, diapers, bottles and pacifiers.
• Follow instructions to shut off water, gas and electricity before you leave.
• Know your evacuation routes before you go. Print out directions from your county emergency management website.

Common sense

• Evacuate if your local government issues an order.
• Don’t drive through flooded streets.
• Stay away from downed wires.

Don’t panic

• Practice your disaster response so everyone knows what to expect.
• Keep your cool for your own safety and your family’s.

Source: CompTIA

The next day, the road into town was still blocked by police, so Kirkegaard parked and walked two miles to check on his home. It was gone. “We saw complete devastation. We walked around in the pile of ash and rubble. Nothing was left,” Kirkegaard said.

And the nightmare wasn’t over. It still wasn’t possible to get to the other side of the town and he had a grueling wait of another full day before he was notified that the commercial building that housed his MSP business, Pacific Tech Gurus, was completely gone too.

Just like that, his entire world was in disarray. His home, employees and customers, and their families—everyone in Lahaina—lost everything.

“I felt tremendously sad. If it were just us, or a few people, that would be doable for the community to deal with. But the entire old Lahaina Town is gone—an entire community,” Kirkegaard said. “It’s just overwhelming right now, but we have to narrow our view and focus on each task at hand, one step at a time. There is no other way,” he said.

What Do You Do? Where Do You Even Start?

Natural disasters are a constant threat to almost any community everywhere: Earthquakes, hurricanes, blizzards, tornadoes, fires. Businesses spend a lot of time and money developing strategies for how to respond to a disaster, but the truth is it’s easy to become complacent. It won’t be that bad. We know what to do. It doesn’t happen here. Until it does.

MSPs [managed service providers]  across Hawaii are scrambling, tasked with helping their customers recover while their own lives, and those of their employees, are in chaos too. Compounding the problem is the confusion and feeling overwhelmed. Everyone responds at different speeds and their needs and availability could change at any turn. And it’s just the beginning. The initial recovery may be the “easy” part. Rebuilding can be much more challenging.

In the immediate aftermath, it’s difficult for MSPs serving Maui to plan anything long term because they don’t know what customers are going to do. Much of Pacific Tech Gurus’ customer base was restaurants and hotels that cater to tourists. Rebuilding entire resorts could take months to even begin and years to complete. And where do you even begin? Kirkegaard himself is working from a relative’s house, but that can’t last forever. Rebuilding will begin, but Maui could look very different than it did before the fires.

“We can build new IT infrastructure, but I can’t build a big footprint myself in someone else’s shop on a temporary basis. We’re filing insurance claims, but where do we stand coming out of all this? We need permanent housing for my family and where do we look for office space? Do we look to a region where there’s less risk of fire but we’re farther from clients, or do we look to be closer to where our clients were—if those clients even come back?” Kirkegaard asked. “It’s hard to evaluate. What I do has somewhat to do with what our customer base does.”

The immediate goal is to get customers back online and get them access to email, payroll, accounts receivable and other applications to help determine next steps.

“We’re trying to mentally keep things going. The challenge right now is to figure out what we need to do after major disruption,” Kirkegaard said.

Widespread Impact Wreaks Havoc

The impact of the wildfires is being felt across the Hawaiian Islands, even by MSPs not in the direct fire zone.

Jason Stone, president and CEO of Tech Partners Hawaii, whose business is based on Maui but has an office in Honolulu, on the island of Oahu, was able to back up his Maui customers’ systems and minimize any potential damage

But the hurricane-force winds also knocked out the internet on Maui early on Aug. 8, preventing offsite backups from happening. The fires didn’t reach Lahaina until later that afternoon so many of his clients—hotels and other resorts that rely on tourism—suffered catastrophic damage.

“The only silver lining was that because there was no power or internet, most clients were not working in their offices, so the (Aug. 7) backups were up to date,” Stone said.

Tech Partners Hawaii had a disaster recovery (DR) plan in place, but it didn’t account for two things, according to Stone. First, the unfortunate pairing of the fires and hurricane-force exponentially escalated the threat in multiple locations. Second, restoring client data from the cloud took much longer than expected due to bandwidth and other issues—everyone using their systems at once. Many of the MSP’s DR plans relied on restoration to local hardware and those companies lost their entire offices—including backup and data recovery appliances, Stone said.

“We had never anticipated losing that many systems at once in catastrophic office disasters and had not kept enough spare BDR appliances on island to handle the number of restorations,” he said.

Related Article: Business Tips to Survive a Natural Disaster (Part 1: Managing Employees)

Preparations Should Begin Today—and Never Stop

Hawaii residents are pretty familiar with large fires. Historically, sugarcane farmers burn sugarcane crops to remove leaves and harvest the remaining stalks that contain the sugar. That’s gone on for more than 100 years, said Derek Gabriel, co-founder and CEO of Ignite Solutions Group in Honolulu, Hawaii. “It’s not uncommon for people who grew up in Hawaii to know the smell of fire,” Gabriel said.

The layout of Maui’s hotels and resort properties are also much different than high-tourism areas on the U.S. mainland, Gabriel said. Resorts are more spread out, as opposed to vertical structures with smaller footprints. Because Hawaii is also built more on rock than dirt, that presents additional challenges when building out IT infrastructure for businesses and homes.

“Swear to God, we would go to Kauai and have to climb into coconut trees to affix an access point,” Gabriel said.

Meanwhile, the Tradewinds blow from northeast to southwest and the leeward side of the islands tends to be very dry. “I can drive west and see cactus on Oahu. It’s quite the juxtaposition,” Gabriel said. “People are not strangers to brush fires but up to this point, they were never paired with these winds. That’s why it became a horrible situation. And people were lackadaisical. It was always ‘the fire department will put it out.’ Add in that our state is grossly underprepared for this, we’re in the middle of the Pacific and we have two weeks of food on the island, so we rely so heavily on ships and planes. We’re really lucky that the fires didn’t touch bigger critical infrastructure like airports, major harbors.”

Cyber Threats Lurk Everywhere

As companies look to rebuild IT infrastructures, even temporary networks in homes or other remote locations, it’s critically important to include cybersecurity in any discussion, said Gabriel. Hopefully, lessons learned from going remote during the COVID-19 pandemic won’t be forgotten and shortcuts won’t be taken.

“Is a building burning down a cyber incident? No. But cyber and IT go hand-in hand. Please, please, please build cybersecurity into an incident response plan,” Gabriel said.

For one, while the physical damage may be over, there’s still a lot of virtual damage that can be wrought in Hawaii, warned Gabriel. There’s already evidence of fake fundraisers and phishing scams as a means to get into personal and business networks.

“I believe the worst is yet to come, from a cyber perspective. I’ve registered domains to take them out of circulation. Whenever there’s money, there’s people trying to get it. We’re doing a lot of awareness stuff, saying watch out for scams. They’re going to come for your money and that will add insult after injury,” he said.

But it’s also important to develop and test incident response programs as you rebuild customers’ networks from scratch.

“The biggest mistake I see is that a vast majority of MSPs are SMBs themselves. And they don’t walk the talk,” Gabriel said. “I can think of a couple of vendors that help us secure, audit and verify our systems. I don’t know many MSPs that have a third party verify if they’re doing it right. There’s a lot of MSPs that go by the grace of God in operation. They’re not prepared for this. The number of natural disasters is not going to slow down. They’re going to increase and you have to be prepared. Make connections in your community. It’s easy to ask for help from someone you know.”

MSPs can also reach out to organizations such as CompTIA’s Emergency Response Team, which is a volunteer-led initiative to help fellow MSPs in need.

Lessons Learned and Looking Forward

Even businesses not taken by the fires could face long roads to recovery.

“We also have clients on Maui’s west side that were not directly affected but have no power or internet. We’ve been working with some local partners to pair Starlink connections with solar battery/charger units that they’re rolling out,” said Stone. “After the initial recovery of client data, providing connections to those in need has been our number one priority. We were able to source around 30 Starlink devices and coordinate with local non-profits to provide free internet access to those that need it.”

Power is slowly being restored but it could be up to three months before all homes and businesses have power, according to Stone. “Internet is another story. LTE and mobile data services are spotty. The large providers have brought in mobile systems to provide emergency connectivity,” he said. “I’ve heard everything from six to 24 months to restore normal residential and business class internet from our local providers.”

In hindsight, staying calm and staying in communication are critical to managing through a catastrophic situation, said Stone.

“We worked with many people and businesses who wanted to help those directly affected by the disaster. There was a lot of overlapping effort and confusion early on. We’ve tried to be very strategic with our resource allocation so that we can do the most amount of good,” Stone said. “Communication is key. As soon as I heard about what had happened, I emailed all our clients with our commitment to them, what they could expect from us, letting them know if they needed anything (tech related or not) we were available for them. We continued to stay in contact and reiterated we were there for them. For those clients we knew had lost offices or were affected, we reached out directly via phone to first, confirm they and their loved ones were okay, and second, to ask if they were ready to move forward and if so how to do it.”

Other MSPs did the same. About 90% of Pacific Tech Gurus’ clients have cloud backups, compared to just 20% two years ago, Kirkegaard said. The MSP had convinced many businesses to backup offsite, but that didn’t offer much help because there was no power or internet.

“Customers in the fire zone lost all their hardware, it’s just dust and melted metal,” Kirkegaard said. “We’re running numbers with a few companies now to see if they want to replace a $15,000 server or go to the cloud instead. The problem with cloud here is that there is a latency to connect to the Continental United States. Eighty milliseconds is not snappy enough for people doing bookkeeping, saving large PDF files or using file-based management systems.”

Initially there were limited IT supplies on the island too.

“We went to Costco and got laptops and SSD drives. We can spool their data set into new machines to get them operational. But we’re still waiting for companies to make a decision to continue. Do they think they are going to survive as an entity? They don’t have a building and if you’re a restaurant or a hotel that’s a problem,” Kirkegaard said. “It’s hard to think when your life is in shreds. We got all new clothes donated and my wife couldn’t even recognize the laundry in the machine because it didn’t look familiar.”

Still, Kirkegaard and the other MSPs are confident they will prevail and Maui will return despite the current dire circumstances.

“For Maui’s west side, the short term is going to be rough. The rest of the island is still business as usual. The world did not stop. We want to be a part of the rebuilding process and help Lahaina and all of Maui heal from this horrible disaster,” said Stone.

“We’ve had fires before. We thought we’d go back to the office, grab the last image on a USB drive, that it would be minor. But then the wind came, and the humidity went to 60 to 50 to 40, down to 28, with 94 degrees Fahrenheit. It was so unreal, but so very real at the same time,” said Kirkegaard. “When the time comes, we all become construction workers—there is no other way.”