The Machine That Goes Ping

19 thoughts on “The Machine That Goes Ping”

  1. Scapegoating some hapless employee when something goes wrong has been corporate standard operating procedure for several decades now. As another blogger so aptly put it: employees are treated like tools, and if something goes wrong or that tool breaks, simply throw it out and grab another from the toolbox. Don’t bother figuring out why that something went wrong. Which figures because Americans especially have never been systems thinkers. Small wonder so many quick-fix solutions are counterproductive.

    The loss of institutional knowledge is a huge unacknowledged problem. Tech and engineering types waste incredible time reverse engineering legacy systems lacking documentation, where the original designers walked (or were pushed) out the door. Subtle but consequential mistakes get made by replacements not fully understanding the nuances. Again, this is largely a managerial failure that pretends employees are interchangeable parts with identical knowledge and abilities.

        1. WordPress “upgraded my user experience” and I don’t like my cheese moved. I’m only half heartedly attempting to navigate the new system.

  2. Just reading the title, I assumed this post was going to be about sale-and-leasebacks.

    Anyway, working in IT myself I know very well how (in IT terms) ancient systems are being kept alive long after the manufacturer stopped supporting them. The systems I work on though are very unlikely to cause any physical harm in case they fail.

  3. Hopefully it’s safe to tell this story and not cause anyone trouble as it’s been nearly 35 years.
    When I was running a business, in the 1980s, one of my customers was an aerospace engineer. We had many discussions about aerospace topics in particular the Space Shuttle, which he had done design work on. This was in mid-1985 and he described the problems with the solid fuel booster rockets and how they would often ‘burn through’ and that eventually that would cause a flight problem/failure.
    Also he described how, when a new project was being drawn up, that occasionally one of the ‘old hands’ would look at the design and say, “You can’t do that because…..” I asked what was going to happen when those ‘old hands’ retire. He said, “We’re going to be in deep sh—!”
    Perhaps coincidentally about nine months later the Space Shuttle Challenger spectacularly disassembled on lift off killing the crew. It happened just the way the engineer described the year before. You might think that NASA would have learned something from this, but if you read the details of the inquiry commission, especially the contribution of Richard Feynman, you will find that they did not.
    As Feynman said, in part, “nature cannot be fooled”.

  4. I am reminded of my mother’s stay in a Seattle area hospital. Things kept going “ping” which was disconcerting (and keeping my mother awake). I flagged someone down and asked what was wrong with my mother. It was because the batteries were low…

  5. I work for a large organization that is heavily software dependant, to the extent that perhaps half our employees are in some form of IT. Keeping complexity from ratcheting so high that it becomes impossible to understand or maintain is a constant, probably ultimately losing, battle.

    Vendor software becoming unsupported is a constant threat – but the home-brewed stuff isn’t maintenance free either.

    Consider it a jobs guarantee.

  6. In the late 1980’s we lived in a small Canadian city that had two hospitals which were about three miles apart. The quickest way to transfer medical records from Hospital A’s computer to Hospital B’s computer was for Hospital A to print out the records, which were then faxed to Hospital B. On arrival at Hospital B, the records were keyed into the hospital’s computer system.

    At least this system used technologies (paper, fax) that were readily understood by almost everyone and were robust. As long as printer paper was available and the telephone system did not fail, the system worked although it was prone to keying error.

  7. The phrase I always used has been “institutional knowledge” . The organization itself maintains it so long as that organization exists in a structure that supports its continuity. I’ve seen it fail a number of times during my career as personnel come and go. Often it seems it’s based on one of two people putting in just a little more effort to improve a task or process. And that gets ‘picked up’ as the doctrine; when one the people leaves (or even just medical or vacation) the wheels come off the process. I suspect western civilization will fail because an un-labeled valve won’t be turned at the appropriate time, LOL.

    1. People with gray hair (which covers the outside of a head full of “institutional knowledge”) are expensive, according to the sales and beancounter types. Their production and contribution can’t always be quantified in the “metrics” that those folks live by, so out they go. Sometimes with “retirement packages”.

      True story: several people I know pretty well used to work for a major pharma company. For decades. Then three years ago a PhD scientist-CEO was replaced by an MBA CEO who’d come up through the sales ranks. He wanted R&D (the heart of a drug company) to meet “Wall Street Targets” of 20% of sales…no more. He launched an early-retirement/buyout for folks in their traditional pension plan within a few years of, or past, the eligibility point. So 50- and 60-somethings. Several thousand well-paid employees took the offer.

      A couple of things happened: the local United Way took a huge hit; the corporation took hundreds of millions in “non-recurring charges”; and a bunch of those people went to work for smaller biotech and contract services companies in pharma. Some have done extremely well, including folks I know.

      The company lost a bunch of institutional knowledge, as the younger early retirees who went elsewhere have no remaining loyalty to help their former (mid-career) colleagues. The ones who actually retired probably do informally advise… but their culture changed, to the regrets of to those (mostly Boomers) who could have left but stayed.

      The stock price is higher without all that “overhead”, which in the end, is what matters to a wealth-driven CEO.

  8. Perhaps the biggest failure was the massive allocation of resources to those elective surgeries, to extend life or independence for a short time, while reducing funding for basic public health, possibly ending people’s lives years early.

    What I think needs to be asked any medical intervention is “what is the break even point,?” best case, worst case, average case.

    Surgery, drugs, etc. generally do damage make you less healthy and reduce your quality of life. At some point you hopefully recover, and then are better off than you had been. As you get older, however, the interventions cease to be worth it. There just isn’t enough time left to overcome the damage, even if it works, and risk.

    It sounds like the average break even point is getting worse. Older generations point to the i-phone and Starbucks and claim later born generations have it lucky. Life expectancy, the ultimate statistic, says otherwise.

    1. One of the dirtiest little secrets of the medical profession is that far too many treatments have never had their efficacy verified by rigorous double blind studies. Some treatments amount to little more than a hunch (lets see if this works). Several studies estimate up to a third of medical interventions are useless, if not counterproductive. And don’t forget the estimated 10% of patients harmed or killed by medical errors.

  9. Johnny, when you write about our medical ‘system’ I am reminded of another John Cleese bit. He was being interviewed about Fawlty Towers and where the ideas for the series came from. Cleese related a location trip the Pythons went on and the hotel they stayed in where the manager was worse than the fictional one on F T. The proprietor said at one point that the hotel “would run so much smoother without the guests.”
    It’s similar with hospitals. The management regards the patients as a, barely, necessary irritant and things would run much better without them. As long as they could figure out a way to get people to pay them every month without having to do anything.
    Oh NO, wait. Isn’t that called “medical insurance”?
    In addition to Dr. Cleese asking for the “machine that goes PING!” he also declared that they had the facilities “to perform a complete cash-ectomy, removing all funds from the patient!”
    Johnny I’m with you on the near-to-failure complexity of things. Seen this for my whole life. Things are held together with tape or perhaps a spring pulling on something and many machines have notes on them. “DO NOT TURN UP PAST ….” “IF SMOKE COMES OUT SHUT OFF IMMEDIATELY!”, “THIS IS THE CORRECT OPERATING SEQUENCE!” and my favorite, “DO NOT USE!” on a machine in use everyday.
    What was most, at first, astonishing to me was how most ‘drank the flavor-aid’ and found this to be normal. Everywhere I have been there were a few, with small halos, that kept things going. With lots of tweaks here and there.
    When the inevitable happened management was always aghast. “How could you let this happen!” when they had been told of the death of this or that system long ago.
    There was always a lot of hair pulling about the money that was needed for new equipment and as Johnny said, a search for who got the last out in the bottom of the ninth inning. Because they “lost the ball game!”

  10. Thanks for another fascinating post, Johnny.

    Funny, the situation in this post echoes my own wacky work-arounds in moving scads upon scads of otherwise unreadable files from an old PC to a MAC, and also in salvaging the content and URLs of a ginormous webpage made in a discontinued program on said PC. The headaches inspired me to buy a refurbished 1960s-era typewriter. Though I now primarily rely on my MacBook Pro laptop, Word and WordPress, I actually also use my typewriter just about every day. By Jove! No software to get updated or in any way outdated! EMR and distraction free! No wifi needed! Yes, I can get ribbons, and I have access to a local typewriter repair service, should I need it. So far so good. And so appealing less complex.

    Back to your post. It sounds to me like the timeline is about like my computer files and website issues. 20 years, and boom, mega problems. But with patience, Advil, and a Charlie Chaplin movie or three…

    Meanwhile, I’ll aim to stay out of the hospital.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.