Dev News Daily ENDE

Victoria Shinder: A vendor patch is a change nobody on your team reviewed, and it ships anyway

Teams spend enormous effort on the changes they author. Branch protection, review, tests, staged rollout, a rollback plan. Then a vendor ships an update on a Tuesday, it lands on every machine, and none of that machinery applies to it.

This month's Excel paste failure is a small, clean illustration: a security update fixed a real vulnerability and, as a side effect, made an everyday operation fail silently. Nothing in the affected organisations' release process had an opinion about it, because the change did not come through their release process.

What a guard would look like

You cannot review a vendor's patch. You can decide what happens when it arrives.

Know what updates itself. Most teams can list their dependencies and very few can list the things that update without being asked — the operating system, the office suite, the browser, the agent on the build machine, the base image that says :latest. The list is the first deliverable, and writing it is usually uncomfortable.

Have one machine that gets it first. Not a test environment for your software — a machine that takes vendor updates a day or a week before the rest, with somebody doing ordinary work on it. That is how a silent failure is found by a colleague instead of by a customer.

Write down what each tool is load-bearing for. "Excel is where finance reconciles the month" is a sentence that turns a broken paste from an annoyance into an incident with a deadline. Nobody writes it down until afterwards.

Victoria Shinder: A vendor patch is a change nobody on your team reviewed, and it ships anyway
Victoria Shinder: A vendor patch is a change nobody on your team reviewed, and it ships anyway — Dev News Daily

The part that is uncomfortable

The reason this is rarely done is that it has no visible output when it works. A patch arrives, nothing breaks, and the week in which nothing broke looks identical to the week in which nobody was watching.

That is the same argument people use against tests, and it loses for the same reason: the cost is paid once and the failure is paid every time. A release process that only governs your own commits is governing the smaller half of what changes in production.

A one-page version you can actually adopt

Nobody adopts a policy with nine steps. This one has three lines, and every team I have seen make it work wrote them down somewhere visible:

  1. A list of what updates itself. Not dependencies — the things nobody declared: the OS, the browser, the office suite, the agent on the build machine, the base image pinned to a moving tag.
  2. One machine that takes updates first, used for ordinary work by somebody who will notice and say something.
  3. One sentence per tool about what it is load-bearing for, written before the incident rather than during it.

The measurement that makes the case

If you want the argument to survive a planning meeting, bring the number: count the production incidents of the last year and sort them into changes we made and changes that arrived. Most teams have never done that split, and are surprised by it. The second column is the one nobody owns, and it is rarely small.