Incoming signals are behaving funny on a hardwired control system, much to the chagrin of the controls engineers.
Back in 1976, I was working as a Controls Engineer at the nuclear reactor fuelling machine test facility in the Atomic Energy Dept of India. We had a hardwired control system comprising of a number of racks (similar to those found in old telephone exchanges). All the incoming signals from the machine sensors and the outgoing commands from the operator panel were hard wired to build the LOGIC Control for the machine operation. The machine operating sequence was also completely hardwired. This facility was used to test the fuelling machines before the machines were sent to the actual reactor site for commissioning.
In one of testing sessions we observed that one or two of the incoming signals were behaving funny. They were randomly changing their level (from on to off). We thought that the true incoming signal wire may be getting mixed with another signal because of wire shorting. So we bypassed those signals (by forcing them to be true all the time at the junction point) and continued with our testing.
Next day we found some more signals also behaving the same way. We isolated these signals and continued with our work.
As the days went by the problem accelerated until almost the whole group of signals was behaving randomly. The testing could not be continued as vital interlock and safety signals were behaving funny.
Suspecting some major problem (maybe rats?), and not wanting to take further risk, we switched over to the backup control system and decided to have a thorough check on the complete wiring system under suspect.
As our technicians were tracing each and every wire on those racks, one of them accidentally hit the earthing screws at the bottom of one of the racks. Suddenly we could observe that almost all the lights on the control panel blinked for a moment. The cause was found. Almost all the screws providing the ground connection to the signals on each rack had loosened over time by the continuous vibration of the high-pressure, high- temperature pressure loop outside the control room . This had created a floating ground condition by which the logic level of the signals was behaving randomly!
The problem which persisted for almost three weeks was solved in just a matter of a few minutes. After all those screws were tightened, everybody had a hearty laugh over the silliness of the whole thing.
started his career as Controls Engineer in the Fuel Handling test facility of Dept of Atomic Energy of India and then moved to software development domain. He has rich experience in embedded systems software development in the areas of robotics, automation, telecom, data acquisition, process control, digital TV, automotive engine control, test automation, appliance control , electric bikes etc. He is currently a freelance consultant. Fields of interest are solar energy conversion and electric vehicles.