You realize just how much your OJT (on-the-job training) in software engineering was lacking when a planeload of FBI agents comes to get stuff fixed.
I had wanted to get away from the city, but when I arrived in a small mountain town to undertake software development for a police, life-flight, and special mission communication system, I had no idea what I was getting myself into.
My OJT (on-the-job training) had consisted pretty much of being told to buy parts to build a development PC, and also of being given a quick run-down on how the compiler and assembler were used to create code. Then I was given a list of bugs, along with a list of routines to be converted into assembly language to make room for the fixes. (Our first generation C compiler was about 5 to 7X less efficient than the assembler, and the project had run out of room without completing the firmware.)
I would frantically fix bugs during the day, take routines home to convert into assembly into the wee hours of the morning, and return to work at 7:00 a.m. to start all over again. The idea was to keep on converting routines until we could obtain a better version of the compiler. In the meantime, we started re-designing the processor to add more memory.
Eventually, we got enough features working (for the most part) and shipped a system to the FBI. They used the system with some success, but -- like all of our other customers -- bug reports kept coming in. Suspecting there had to be a better way, I visited the local aeronautical university library and began reading every book I could find on the subject of developing avionics software. There were things like creating the requirements first, such as code size and schedule estimates, and also using LINT applications and runtime memory checking. I started to apply some of these concepts to my own coding, resulting in much greater success on some of the later features.
One day the phone rang and we were informed that a planeload of FBI agents were on their way to the plant. They needed their system to be working, but it had one bug they could not work around. The next day there was a special agent in the plant manager's office, another agent in my boss's office, and yet another agent assigned to be with me while I went to the aircraft and tried to duplicate their issues.
First, we set up one of our lab systems to match the configuration they were using. Everything seemed to work fine until we took the system into the air and activated things in a different way. Pressing just one button introduced a totally different aspect of system operation.
One-by-one we located bugs and fixed them. Finally, we identified the major culprit -- a pointer in one area of the code that was impacting another section of the code. We got that bug fixed. Walter the mechanic got the plane put back together and working, at which point the agents set off on their way to track down America's most wanted.
Later, the company I was working for was bought out by another company, at which time I finally managed to get formal training in DO-178B and software project management. If only I had been given this training earlier, my life would have been much easier and my code would have been much better.