What a strange five-year trip it has been...

I originally published the following summary as part of my weekly CTO status report in December 2021.

October 24, 2021 marked my five-year anniversary with Adventist Health. As I am increasing fond of saying, I am an old dog. In a forty plus year career in Information Technology, I have had the opportunity for a dizzying variety of projects, opportunities, and challenges. The last five years feels unique. A summary of my first five years with Sutter Health can be found here.

In the wake of the Lodi Cerner go-live in September, I got the job as the Chief Technical Officer (CTO) for Adventist Health in late October of 2016.

Karl Nebel was named CIO in early November of 2016.

In early January of 2017, Jennifer, Karl and I were scheduled on a flight to Portland to meet with OHSU. Karl didn’t make the flight. When we landed and I turned my phone back on, it exploded. Matt asked me to grab Jennifer, find somewhere quiet and call him. Karl had passed away that morning.

Much of 2017 was filled with a struggle to clean up a bushel of technical issues in the aftermath of the Lodi Cerner go-live. Unfortunately, there are still some lingering Lodi legacy issues.

Matt Fink was named CIO in May of 2017.

Negotiations with Cerner about the ITWorks engagement started in the second half of 2017.

In fall of the 2017, the Chief Information Security Officer (CISO) melted down in a meeting with Matt and me. I walked him out of the building as he shouted obscenities at me.

The enterprise data center had been in Creekside 1 for more than sixteen years. With the plans for the new Roseville building not including a data center, the enterprise technical teams spent most of 2017 meeting with data center colocation vendors and touring possible sites.

The ITWorks deal was announced in January of 2018.

In March of 2018, many user accounts were compromised as part of large-scale phishing attack. The credentials were used to change the paycheck routing codes in Oracle. It is a dangerous electronic world.

The ITWorks arrangement was implemented in April of 2018. This was the fifth outsourcing arrangement that I have been involved with in my career.

In June of 2018, after months of fighting with Microsoft, they finally agreed to a graduated payment schedule. Almost a month after the previous enterprise agreement expired, Adventist Health renewed the Microsoft EA.

At the end of June of 2018, we published an IT Playbook. While other Adventist Health business units had playbooks for years, this was first for IT. I am proud of the work that we did on this playbook.

In August of 2018, the teams started moving over 1,000 physical and virtual servers and about 1.6 petabytes of storage from Creekside 1 to a colocation data center. This project took just over six months.

In November of 2018, we responded to the Camp Fire and the loss of the Feather River Hospital. Sitting in the command center in Creekside 3 with Bill, I called one of my oldest friends whose brother was a retired Cal Fire battalion chief living in Paradise. I told him about the fire and asked where his brother lived in relation to the hospital. My friend said that he lived on the north end of town and was sure that he would be fine. His brother and sister-in-law literally barely escaped with their lives...

In December of 2018, as part of a best of vendor strategy, we moved the email archive from Proofpoint to Microsoft.

In March and April of 2019, there was a mad scramble to complete the technical infrastructure for the Roseville building. This culminated in moving more 1,300 people into the new building in five days in late April.

In June of 2019, as part of a best of vendor strategy, we moved workstation imaging, patching and application packaging from Systematic Altiris to Microsoft System Center.

As we were settling into the new building, the Rideout Cerner project went off the rails. To get it back on track, we purchased, configured, and installed more than 1,000 workstations and hundreds of printers and other peripherals in about seven weeks in late June, July and early August. To support this effort, I spent almost two months living in a cave in Marysville. Like Lodi, we still have some lingering technical issues at Rideout that we haven’t found the resources or capital to address.

Starting to take a breather from the Rideout Project, Adventist Health took the covers off Project Green in late September of 2019. In less than five weeks, we built out the technical infrastructure for a 350-employee office in Kansas City. This included workstations, printers, phones, network infrastructure and call center software.

In November of 2019, we started to have serious about leadership discussions about the need for a reduction in force and an increase in the span of control. These discussions stretched for several months.

In December of 2019, as part of a best of vendor strategy, we started moving the anti-virus from McAfee to Cisco Amp. In total, the multi-part shift from a best of breed to a best of vendor strategy for technical and security products cut more than $2.5 million from IT operating cost budget.

On March 11, 2020, the decision was made to send everyone to work from home. We scrambled to outfit and send 1,300 employees home on Thursday, March 12. The Roseville building has largely remained vacant for more twenty-one (21) months.

At the beginning of April 2020, Adventist Health rolled out the Hospital@Home using an accelerated six-week project plan. This included building command centers in Hanford, St. Helena and Ukiah.

In April of 2020, we did a reduction in force. This included the elimination of several senior IT leadership positions and a major restructuring of the reporting structure of the markets.

Starting in May of 2020, working with Sean’s web Team, we moved everyone to Microsoft Teams. I am frankly not sure how we would have survived the last two years without Microsoft Teams as a collaboration tool.

The enterprise technical teams worked through June and July of 2020 to harden the infrastructure to support more than 4,000 employees working remotely.

In August of 2020, we supported the evacuation and shutdown of St Helena Hospital and clinics followed by the reinstall and repopulation when the mandatory evacuation was lifted.

In September of 2020, the St. Helena Hospital was evacuated again for the Glass Fire. We scrambled to restore network services to the clinics and relocate services.

In October of 2020, we were faced with a ransomware attack at a small hospital. Over the course of just over a week, we rebuilt their entire technical infrastructure from scratch.

In November of 2020, we torn down the 350-person office space that we had built for Project Green in fall of 2019.

In December of 2020, we supported the reopening of St. Helena Hospital.

In the first quarter of 2021, we finally completed a multi-year journey to upgrade our popsicle-stick house of user authentication technology and implement multi-factor authentication for all users.

In April of 2021, Tyler Whetstine was appointed CIO.

In May of 2021, Adventist Health signed a five-year Microsoft Enterprise Agreement renewal. I am frankly not as happy with this renewal as I was with the previous agreement in 2018. I believe that this negotiation foreshadows the kind of struggles that we are going to have managing operating cost increases with cloud vendors.

Dwight Morgan passed away in July of 2021.

In late July of 2021, IT did another reduction in force. I have literally had to perform layoffs for more than thirty years. Emotionally, these hit me hard. With the layoffs and the elimination of open positions, there are now ten fewer managers, directors and administrative directors reporting to me than there was a little more than two years ago.

Starting about the second quarter of 2021, Adventist Health experienced an unusual number of Cerner application bumps and outages. The number of incidents and the length of several outages has put an enormous strain on everyone.

In my experience, the last eighteen months have been unique regarding getting capital for technical infrastructure lifecycle. It has taken voodoo to move capital equipment requests forward, e.g. standing in my front yard swinging a dead chicken over my head.

The PMO and the market and enterprise technical teams have done some amazing work at Mendocino Coast, Tulare and Delano over the last three years. Applying the acquisition wedding cake, we have worked to avoid the Lodi and Rideout mistakes and provide a solid technology and security foundation.

In November of 2021, I traveled to a market for the first time in almost exactly two years.

It is hard to believe that there are lots and lots of employees that I am responsible for that I have not seen in person since March of 2020. While I feel that our IT teams have been effective and productive working remotely over the last twenty-one plus months, I feel like I have lost some of the connections that I had with other Adventist Health leaders. The new building was great for running into people on the stairs and the cafeteria or dropping in to visit with leaders in their offices. Those connections and relationship are important for IT and have been harder to maintain virtually.

I am sure that I have missed a few significant events.

What a strange five-year trip it has been...