A nurse friend was recently describing a system at her hospital that summed up what I call the complexity trap. She works with seriously ill patients that need to be monitored 24/7. Humans used to do this continuous testing and assessment, but over the decades a collection of machines was gradually introduced to save labor and reduce costs.
.
.
As the number of devices proliferated a special device was installed that gathers data from all the separate bits of equipment and synthesizes the information. Having concise real time diagnostics saves lives in the critical care unit. It also saves money by allowing fewer staff to achieve the same level of service. Unfortunately the company that manufactured the machine doesn’t make it anymore and no longer offers maintenance or technical support. Evidently the product wasn’t profitable.
.
.
Her team has been keeping the system running for years using a combination of hacks. None of the individual work-arounds are all that difficult, but the steps need to be followed exactly and they aren’t immediately apparent to the uninitiated. And that’s where the next set of institutional complexities comes in.
.
The Covid situation has greatly increased costs while diminishing revenue for hospitals. The old economic model involved elective procedures that subsidized other aspects of patient care, but that arrangement has crashed and left a giant hole in the budget. The best practice is to have an established team of people work cohesively together over time. But the oldest most experienced (and most expensive) hospital staff have taken retirement and remove themselves from the chaos.
.
.
The administrative solution has been to find younger less experienced replacements that are significantly cheaper than established seasoned staff. In theory these new people will be brought up to speed quickly, but in practice there aren’t enough older mentors to train them effectively. On top of this, the new hires are often rotated in and out of different hospitals on short term contracts so both individuals and whole teams are in a perpetual state of flux.
.
.
The eccentric work-arounds to keep medical equipment functioning combined with the lack of institutional memory create the ideal conditions for something to fail. This is the kind of failure that no one person will actually be responsible for, but once something goes wrong there will be a hunt for the culprit. And very few people will even understand why things went south because they’ll be unaware of the bigger picture. It’s easy to see some novice employee taking the fall for what is actually a systemic disaster waiting to happen.
.
.
This kind of situation is probably playing out in all our institutional structures right now. Oil refineries, nuclear power stations, commercial aviation, water treatment plants, financial organizations… Covid didn’t create our present fragility. Instead, it exposed it. As a society we’ve spent decades pushing various envelopes. We’ve leveraged up enormous unrepayable debts, allowed our infrastructure to quietly deteriorate, and distracted ourselves with political nonsense on all sides. We’re weak and vulnerable to the next crisis events that are hovering on the horizon. In the long run we’ll rise to the occasion and rebuild our institutions – kicking and screaming. We’ll have no choice in the matter if we want to survive. But for the next little while we’re just going to have to suck up a whole lot of failure.
Granola Shotgun 
Scapegoating some hapless employee when something goes wrong has been corporate standard operating procedure for several decades now. As another blogger so aptly put it: employees are treated like tools, and if something goes wrong or that tool breaks, simply throw it out and grab another from the toolbox. Don’t bother figuring out why that something went wrong. Which figures because Americans especially have never been systems thinkers. Small wonder so many quick-fix solutions are counterproductive.
The loss of institutional knowledge is a huge unacknowledged problem. Tech and engineering types waste incredible time reverse engineering legacy systems lacking documentation, where the original designers walked (or were pushed) out the door. Subtle but consequential mistakes get made by replacements not fully understanding the nuances. Again, this is largely a managerial failure that pretends employees are interchangeable parts with identical knowledge and abilities.
Johnny, are you doing all right? You’ve been quiet lately. Hope all is well.
Still alive and well. Life is keeping me busy these days.
Good to hear. Take care! I’m looking forward to your next post.
WordPress “upgraded my user experience” and I don’t like my cheese moved. I’m only half heartedly attempting to navigate the new system.
Just reading the title, I assumed this post was going to be about sale-and-leasebacks.
Anyway, working in IT myself I know very well how (in IT terms) ancient systems are being kept alive long after the manufacturer stopped supporting them. The systems I work on though are very unlikely to cause any physical harm in case they fail.
Hopefully it’s safe to tell this story and not cause anyone trouble as it’s been nearly 35 years.
When I was running a business, in the 1980s, one of my customers was an aerospace engineer. We had many discussions about aerospace topics in particular the Space Shuttle, which he had done design work on. This was in mid-1985 and he described the problems with the solid fuel booster rockets and how they would often ‘burn through’ and that eventually that would cause a flight problem/failure.
Also he described how, when a new project was being drawn up, that occasionally one of the ‘old hands’ would look at the design and say, “You can’t do that because…..” I asked what was going to happen when those ‘old hands’ retire. He said, “We’re going to be in deep sh—!”
Perhaps coincidentally about nine months later the Space Shuttle Challenger spectacularly disassembled on lift off killing the crew. It happened just the way the engineer described the year before. You might think that NASA would have learned something from this, but if you read the details of the inquiry commission, especially the contribution of Richard Feynman, you will find that they did not.
As Feynman said, in part, “nature cannot be fooled”.
I am reminded of my mother’s stay in a Seattle area hospital. Things kept going “ping” which was disconcerting (and keeping my mother awake). I flagged someone down and asked what was wrong with my mother. It was because the batteries were low…
“You do not rise to the level of your goals. You fall to the level of your systems.”
James Clear
Or, from the horse world, “you do not rise to the occasion, you default to the level of your training.”
You gotta cell I can text you some pics? Wanted to send you pics of some tiny shops I’m doing for vendors at the Truck Stop! Mike
Sent from my iPhone
>
I work for a large organization that is heavily software dependant, to the extent that perhaps half our employees are in some form of IT. Keeping complexity from ratcheting so high that it becomes impossible to understand or maintain is a constant, probably ultimately losing, battle.
Vendor software becoming unsupported is a constant threat – but the home-brewed stuff isn’t maintenance free either.
Consider it a jobs guarantee.
In the late 1980’s we lived in a small Canadian city that had two hospitals which were about three miles apart. The quickest way to transfer medical records from Hospital A’s computer to Hospital B’s computer was for Hospital A to print out the records, which were then faxed to Hospital B. On arrival at Hospital B, the records were keyed into the hospital’s computer system.
At least this system used technologies (paper, fax) that were readily understood by almost everyone and were robust. As long as printer paper was available and the telephone system did not fail, the system worked although it was prone to keying error.
The phrase I always used has been “institutional knowledge” . The organization itself maintains it so long as that organization exists in a structure that supports its continuity. I’ve seen it fail a number of times during my career as personnel come and go. Often it seems it’s based on one of two people putting in just a little more effort to improve a task or process. And that gets ‘picked up’ as the doctrine; when one the people leaves (or even just medical or vacation) the wheels come off the process. I suspect western civilization will fail because an un-labeled valve won’t be turned at the appropriate time, LOL.
People with gray hair (which covers the outside of a head full of “institutional knowledge”) are expensive, according to the sales and beancounter types. Their production and contribution can’t always be quantified in the “metrics” that those folks live by, so out they go. Sometimes with “retirement packages”.
True story: several people I know pretty well used to work for a major pharma company. For decades. Then three years ago a PhD scientist-CEO was replaced by an MBA CEO who’d come up through the sales ranks. He wanted R&D (the heart of a drug company) to meet “Wall Street Targets” of 20% of sales…no more. He launched an early-retirement/buyout for folks in their traditional pension plan within a few years of, or past, the eligibility point. So 50- and 60-somethings. Several thousand well-paid employees took the offer.
A couple of things happened: the local United Way took a huge hit; the corporation took hundreds of millions in “non-recurring charges”; and a bunch of those people went to work for smaller biotech and contract services companies in pharma. Some have done extremely well, including folks I know.
The company lost a bunch of institutional knowledge, as the younger early retirees who went elsewhere have no remaining loyalty to help their former (mid-career) colleagues. The ones who actually retired probably do informally advise… but their culture changed, to the regrets of to those (mostly Boomers) who could have left but stayed.
The stock price is higher without all that “overhead”, which in the end, is what matters to a wealth-driven CEO.
Perhaps the biggest failure was the massive allocation of resources to those elective surgeries, to extend life or independence for a short time, while reducing funding for basic public health, possibly ending people’s lives years early.
What I think needs to be asked any medical intervention is “what is the break even point,?” best case, worst case, average case.
Surgery, drugs, etc. generally do damage make you less healthy and reduce your quality of life. At some point you hopefully recover, and then are better off than you had been. As you get older, however, the interventions cease to be worth it. There just isn’t enough time left to overcome the damage, even if it works, and risk.
It sounds like the average break even point is getting worse. Older generations point to the i-phone and Starbucks and claim later born generations have it lucky. Life expectancy, the ultimate statistic, says otherwise.
One of the dirtiest little secrets of the medical profession is that far too many treatments have never had their efficacy verified by rigorous double blind studies. Some treatments amount to little more than a hunch (lets see if this works). Several studies estimate up to a third of medical interventions are useless, if not counterproductive. And don’t forget the estimated 10% of patients harmed or killed by medical errors.
Johnny, when you write about our medical ‘system’ I am reminded of another John Cleese bit. He was being interviewed about Fawlty Towers and where the ideas for the series came from. Cleese related a location trip the Pythons went on and the hotel they stayed in where the manager was worse than the fictional one on F T. The proprietor said at one point that the hotel “would run so much smoother without the guests.”
It’s similar with hospitals. The management regards the patients as a, barely, necessary irritant and things would run much better without them. As long as they could figure out a way to get people to pay them every month without having to do anything.
Oh NO, wait. Isn’t that called “medical insurance”?
In addition to Dr. Cleese asking for the “machine that goes PING!” he also declared that they had the facilities “to perform a complete cash-ectomy, removing all funds from the patient!”
Johnny I’m with you on the near-to-failure complexity of things. Seen this for my whole life. Things are held together with tape or perhaps a spring pulling on something and many machines have notes on them. “DO NOT TURN UP PAST ….” “IF SMOKE COMES OUT SHUT OFF IMMEDIATELY!”, “THIS IS THE CORRECT OPERATING SEQUENCE!” and my favorite, “DO NOT USE!” on a machine in use everyday.
What was most, at first, astonishing to me was how most ‘drank the flavor-aid’ and found this to be normal. Everywhere I have been there were a few, with small halos, that kept things going. With lots of tweaks here and there.
When the inevitable happened management was always aghast. “How could you let this happen!” when they had been told of the death of this or that system long ago.
There was always a lot of hair pulling about the money that was needed for new equipment and as Johnny said, a search for who got the last out in the bottom of the ninth inning. Because they “lost the ball game!”
Thanks for another fascinating post, Johnny.
Funny, the situation in this post echoes my own wacky work-arounds in moving scads upon scads of otherwise unreadable files from an old PC to a MAC, and also in salvaging the content and URLs of a ginormous webpage made in a discontinued program on said PC. The headaches inspired me to buy a refurbished 1960s-era typewriter. Though I now primarily rely on my MacBook Pro laptop, Word and WordPress, I actually also use my typewriter just about every day. By Jove! No software to get updated or in any way outdated! EMR and distraction free! No wifi needed! Yes, I can get ribbons, and I have access to a local typewriter repair service, should I need it. So far so good. And so appealing less complex.
Back to your post. It sounds to me like the timeline is about like my computer files and website issues. 20 years, and boom, mega problems. But with patience, Advil, and a Charlie Chaplin movie or three…
Meanwhile, I’ll aim to stay out of the hospital.