A lot of micros have JTAG capability that can be used for both boundary scan and for talking to the CPU's internal debugging capability. I'm most familiar with ARM Cortex-M, and I've used the capability for ST Cortex-M3 SoCs and Cypress PSoC 5LP. They use the standard debug capabilities from ARM, which are well documented. By talking to the CPU's debug port, you can (by default) read and write any memory location in the SoC, including all memory-mapped I/O. (I say "by default" because someone using the SoC in a product can turn off JTAG debug to prevent reverse-engineering of the product, but leave it on in development prototypes.)
ARM also offers Single-Wire Debug, which allows you to access the debug port over just two wires, a bidirectional data line and a clock. (It's called SWD because there's a version that uses asynchronous coding over just the data line, but I've only seen the synchronous version with separate clock in actual implementations.)
"An then you have to decipher the hieroglyphics. And if there was software involved, trying to remember what you did and how it explains the effect that you are seeing."
And hope the software language still exists and that you have hardware to connect to it. Since you're planning on 20 + years of support, I expect you have those contingencies covered, but I can imagine scenerios where that might not be the case.
For example, the equipment might use old UV-erased EPROMS. If the test equipment was purchsed, rather than developed in house, a bad EPROM or a need for a simple code change might mean the end of that piece of equipment.
I have software laying around my house on 5 1/4 floppy discks, 3 1/2 inch disks, old Mac formatted disks, iOmega Zip drives, and some old EPROMs. Fortunately, I won't be needing any of that, but with a support life like you deal with, you certainly might.
I hear a lot about JTAG, but rearely hear from anyone that uses it a lot for test. The promise is quite good, but it seems a lot more limited in actual use
I share your impression. The devices I see with JTAG interfaces seem to me to be 16 bit logic devices, but I am ignorant of any micro that has aJATG interface that you can control the I/O directly or any interfac device like a DAC/ADC so that comprehensive testing on a small embedded system seems limited.
Having said that I now pause, take a breath and wait for someone to set me right.
I once did a course on FPGAs and one of the exercises was to implement a JTAG interface around the finctions on the FPGA to aid with testing. Seems to be a good idea if you are using this kind of design.
Tell me about it. I still have to support test jigs that I designed over 20 years ago. Back then the jigs were scratched out in a notebook, so the first thing is finding the notebook. An then you have to decipher the hieroglyphics. And if there was software involved, trying to remeber what you did and how it expalins the effect that you are seeing.
We now try to document each jig as if it were a product on its own. The only problem is you only get to make one or two and so the natural debugging that comes from repeated building and testing is missing.