Why Publish a Protocol and Not a Result
The normal thing to publish is an answer. Here is what I simulated, here is what came out, here is why it matters. I have done a fair amount of that on this blog, and there will be more of it.
This post is about the opposite thing: the machinery for finding out that an answer was wrong.
I am publishing it for two reasons. The first is that it is genuinely useful and almost nobody writes it down. Every computational group I know of has some version of this discipline, held informally in the heads of whoever has been burned most recently, and it dies when that person moves on. The second reason is more selfish. A protocol you have published is a protocol you have to keep following. Writing it down in public is a way of making it expensive to get lazy.
There are no numbers in this post. That is deliberate — most of the values involved sit behind the disclosure line, and in any case the numbers are not the lesson. The lesson is the shape of the mistakes.
The Failure Mode Is Inheritance, Not Arithmetic
The naive picture of a numerical error is that somebody did the sum wrong. In practice that almost never happens. Arithmetic errors are caught immediately, because the result is usually absurd by orders of magnitude and/or something downstream explodes.
The errors that survive are different. They are values that were computed correctly, in a context where they were correct, and then travelled.
A parameter gets derived once, carefully, with full attention, for a particular geometry or a particular operating point. It lands in a file. Six weeks later that file gets used as the starting point for a related simulation with a slightly different geometry. The parameter comes along. It is still a plausible number — it is the right order of magnitude, it has the right units, it does not make anything explode. It is simply no longer the right number for the thing it is now describing.
This is inheritance, and it is the dominant failure mode in any simulation stack that has been alive for more than a few months. The number is not wrong. The number has been orphaned — it has lost it's reason to exist, it's bearings, separated from the derivation that justified it, and adopted by a context that never checked whether it applied.
You cannot catch that by checking the arithmetic, because the arithmetic is fine. You catch it by refusing to accept any derived value on the authority of the file it is sitting in.
The Five Checks
The rule, stated as briefly as it can be: never trust a derived parameter. Every number that was not measured or defined must be re-established from scratch before it is used in anything that matters.
In practice that means five things.
1. Re-derive from primitives and first principles.
Take the value back to quantities that cannot themselves be wrong — fundamental constants, defined geometry, quantities that come straight from a measurement or from the definition of the structure. Then recompute forward. Not "check the formulae." Recompute, independently, without looking at the value you are checking, and compare at the end.
The discipline of not looking first matters more than it sounds. If you know the answer you are trying to reach, you will find a way to reach it. Derive blind, then compare.
2. Check an invariant.
Find something about the system that must be true regardless of the parameter, and verify that it still is. A symmetry that has to hold. A conservation law. Entropy. The Speed of Light in vacuum. A limiting case where the answer is known analytically. A dimensionless group that should sit in a particular range.
This is the most powerful of the five, because an invariant does not care how your number was produced. It fails on errors you did not anticipate — which is the only kind that matter, since the errors you anticipated were already handled.
3. State the convention.
An enormous fraction of "wrong" numbers are correct numbers under a different convention. Peak or RMS. Radius or diameter. Static or high-frequency. Angular frequency or ordinary frequency. Per-unit-length or total. Zero-to-peak or peak-to-peak. Energy or wavelength.
Every one of these is a factor lurking silently in a value that looks perfectly reasonable. So the convention gets written down next to the number, in the file, every time. Not in a comment at the top of the file — immediately adjacent to the value, because comments at the top of files are not read by the person who copies line 340 into another script.
4. Grep every file.
This is the step people skip, and it is the one that actually finds things.
Once a value has been corrected in the place you found it, search the entire tree for every other occurrence of it. Not just the exact string — the value at different precisions, the value in different units, the variable name, the comment that mentions it in prose. A number that was wrong in one file is almost never wrong in only one file, because the reason it was wrong is that it travelled, and travelling is what it does.
More on this below, because it deserves it.
5. Make it machine-checked.
A protocol that depends on remembering to run it is not a protocol, it is an intention. Every constant that has been through steps 1 to 4 gets an entry in a machine-checked audit: a script that re-derives the value from primitives, asserts agreement within a stated tolerance, and fails loudly if it drifts. The audit runs as a matter of course, not as a matter of virtue.
The point is not that the script is clever. The script is trivial. The point is that the knowledge from the first four steps stops living in somebody's memory and starts living somewhere that complains.
What It Has Caught
Four real errors, described by class rather than by value. Every one of these was in my own work, had survived review, and was found by the protocol above rather than by anything going visibly wrong.
A phase velocity carried across a geometry change. A propagation velocity in the RAVAN compact model had been derived correctly for one structure and then inherited by a variant it did not describe. Nothing broke. Everything continued to run and produce plausible output. It was caught by step 1 — a blind re-derivation from primitives that did not land where the file said it should.
A shell radius that could not physically exist. A geometric parameter that, when you stopped treating it as a number and actually drew the structure it described, placed one surface inside another. This is the class of error that an invariant catches instantly and that no amount of staring at the arithmetic will: the value was internally consistent and physically impossible. Step 2.
A noise floor quoted at the wrong impedance. The figure itself was right. It had been characterised properly, in the right way, for a particular load — and was then quoted in a context with a different one. A textbook orphan: correct number, correct derivation, wrong home. Caught by step 3, because the convention had not been written down next to the value, and once it was, the mismatch was visible immediately.
A permittivity mislabelled. A high-frequency value carrying the label of the static one. Two quantities, related, different by a substantial factor, and distinguished in the file only by a subscript that someone had typed from memory. Step 4 found the other three places it had spread to.
The pattern across all four is worth stating plainly: not one of them was a calculation error. Every one was a correct value that had lost its provenance.
Why the Grep Is the Most Important Step
I want to dwell on step 4, because it is the one that separates a protocol that works from a protocol that feels good.
When you find an error, there is an enormous psychological pull toward closure. You found it. You understand it. You fixed it. The file is correct now and the feeling of having caught something is genuinely pleasant, and that feeling is precisely the moment of maximum danger — because the fix is the least important part of what just happened.
What just happened is that you learned a specific number is untrustworthy. That knowledge applies everywhere that number appears, and you have only applied it in one place.
In every one of the four cases above, the grep found additional occurrences. Not sometimes — every time. That is not bad luck. It is structural: values propagate by copying, so a value found wrong in file A got there by the same mechanism that put it in files B and C. Finding it once and stopping is finding the symptom and leaving the infection.
The grep also has to be more imaginative than a single exact-match search. Round the value up and down. Search it at three significant figures and at six. Convert it to the other unit anyone might have used and search that. Search the variable name. Search the prose in the documentation, because the number is in the prose too, and prose does not throw errors.
The Cost
This is not free, and I do not want to present it as free.
Re-deriving a parameter blind, from primitives, takes between twenty minutes and half a day depending on how deep the chain goes. Doing that for every derived value in a stack of any size is weeks of work that produces, in the typical case, nothing — the value checks out and you have spent a morning confirming what you already believed.
I think it is worth it anyway, for a reason that has nothing to do with the hit rate. Work that is going into a patent specification, or into anything that another person will build on, has an asymmetric cost structure. Confirming a correct value costs a morning. Discovering a wrong one after it has been published, or filed, or built upon, costs something that cannot be paid in mornings at all.
Three of the four errors above were caught before they reached anything irreversible. The fourth was not — a geometric parameter went into a filed specification and is being carried corrected into the applications that follow it. That is the honest version, and I would rather write it down than round it up to four. The protocol is not a guarantee that nothing gets through. It is the reason I know which one did.
What This Does Not Protect Against
Honesty requires the limits.
This protocol catches wrong values. It does not catch a wrong model. If the physics is misconceived, every constant in it can be immaculately derived and the answer will still be wrong, confidently and reproducibly. Nothing in the five steps above would notice.
It also does not catch errors in quantities that were measured rather than derived, or in anything that entered the stack as a definition. Those need a different discipline.
And it does not make the results right. It makes them auditable — which is a weaker claim than people usually want, and the only one I am prepared to make.