Winter Storm Update: Exchange Headquarters ‘Was Moments Away’ from Losing Critical Connectivity
Exchange Headquarters was reopened to associates Feb. 22—except for some areas of the first and second floors—thanks to round-the-clock work by associates in the IT and Real Estate directorates to repair the damage.
Some work was herculean, as associates dealt with an estimated 150,000 square feet of flooding on the second floor.
“Some computers sat in 2 ½ inches of water,” said Chief of Facilities Management Moises Pinedo. “It’s been an all-night affair for some, but our teammates’ dedication helped us make significant progress.”
IT associates Ric Murphy, Cody Van Haaften, David Drake and Shinobu Agarie connect computers after the storm hit to allow associates to be able to work from home.
After several waterlines and pipes burst the week of the storm, IT associates feverishly worked to disconnect equipment from power sources and relocate it to dry staging areas. The storm also disrupted the ability for associates to remote in from home, but salvageable equipment was relocated so servers could get up and running again in just a couple of days.
IT associates found any open, dry space they could, and then reconnected hundreds of computers by hand. A makeshift rack was set up in the Data Center that included about 550 computers, all of which had to be checked for damage and then reconnected.
Facility Management Office team members installed a portable, 5-ton cooler to cool the additional load generated by the computers.
About 300 more PCs were set up in a room on the first floor. In the middle of a 100-year winter storm that paralyzed the state, associates scoured hardware stores across the Dallas-Fort Worth area to find enough power strips and other necessary supplies to run the makeshift setup.
“Our team expended a tremendous amount of energy, but they didn’t stop until the job was done,” IT Infrastructure Support Manager Valerie Ledford said.
In rooms that housed batteries to power equipment during outages, water nearly reached the top of the pads where the batteries sat.
Massive, temporary air ducts were installed on the first and second floors to keep fresh air flowing at Headquarters.
“We were one drop of water landing in the wrong place away from total darkness,” Data Center Facility Maintenance Supervisor Cody Van Haaften said.
While IT was busy trying to keep things connected, HQ FMO associates and contractors worked to limit water damage and make the building habitable.
Dehumidifiers were installed on the first and second floors, which reduced humidity levels on the second floor from 77% to 50% and lower in the hallways. Massive, temporary ducts were constructed on both floors to keep fresh air circulating.
“We have assessed the amount of moisture in the walls and have identified sections of walls that need to be removed,” Pinedo said.
Only the thin, outer layer of the walls are being removed so the dehumidification process can work properly, Pinedo said.
Pinedo said working alongside IT associates was a gratifying experience, even if the task was daunting.
“I don’t know how many hours they worked, but their efforts were massive in keeping the Exchange going,” he said. “It brought us closer together and gave us some ideas on how to strengthen collaboration further.”
Ledford was equally grateful to RE.
“The Data Center is the heartbeat of the Exchange, and we were just moments away from losing it,” she said. “So many Exchange associates made sure that didn’t happen.”
To see more images, see our Flickr album.
Great job to EVERYONE! Thank you for your dedication and hard work to keep us going strong!
Thanks, Melissa. Appreciate your support of our great Exchange teammates!
Awesome save guys! Thank you!
Thanks, Arcena. Associates from several directorates truly went above and beyond to keep the Exchange operational!
Thanks for all your hard work!
Appreciate your kind thoughts, Richard. It took a team effort and a lot of hard work, but our associates truly rose to the occasion!
I truly Thank God that the right people were capable of getting our building back to as much normal as possible. Thanks to all who worked so very hard to “fix us up” !!
Indeed, Rhonda. Thank you!
Fantastic job Team!
You guys truly showed your dedication and determination that makes the Exchange family proud!
You are awesome!
Thanks for your support, Yolanda!
I am vary impressed and grateful for what was accomplished in such a short period of time.
Thanks, Harry. It took round-the-clock efforts and a refusal to quit, but these associates shined!
Woo hoo Great Job thanks for all you do..Team!!
Thank you, Cheryle. The team did magnificent work!
GREAT JOB GUYS!!!!
Thanks for your support, Elaine!
Thank You to all for your untiring efforts to keep things going! I’m sure it wasn’t easy as some may have had to deal with additional problems in their own homes as well! Thanks Again.
“Untiring” really is the key word here, Thenson! They were rushed into work in the middle of the night and worked around the clock to make things operational again.
Your team’s ability to quickly absorb and filter what needed to be done in a crisis situation was commendable! I was amazed at being able to get our payroll out on time! Sincere, heartfelt thanks to all of you!
Thanks, Julia. It really was amazing how quickly our associates got Exchange operations back up and running!
Awesome job by FMO and the IT teams. Well done!!!!
Thank you, Denise! The Exchange is a great example of how working together pays off!
Awesome guys, really a herculean job and kudos to each and everyone. May our Lord God Almighty be with you and yours always. Stay safe, stay healthy.
Thank you, Regalado! The work our Exchange associates did in mitigating the damage was simply incredible!