An EMI consultant investigates the source of a new conveyor's erratic behavior.
Remember the I Love Lucy episode in which the conveyor belt transporting chocolate candy speeds up faster and faster?
That was pretty much the scene at a major newspaper printing facility when I was called in to investigate whether electromagnetic interference (EMI) was the source of a problem with a new handling system. It would occasionally and randomly stop, while the printing press kept going. Talk about chaos.
The handling system was like a Ferris wheel turned out its side. It had a dozen bins to carry bundles of newspapers to waiting trucks at a loading dock. Individual embedded controllers had the task of ejecting labeled bundles at the appropriate location. A minicomputer located in a control room had supervisory responsibilities and the entire system was networked by Ethernet.
The system was a prototype, and it was still being debugged. As such, nobody was even sure it was EMI-related. My task was either to find an EMI cause or to eliminate EMI from further consideration. I focused on the big three EMI threats: electrostatic discharge (ESD), power disturbances, and RF interference.
My initial concern was ESD -- both human generated and machine generated. After all, this was a printing press, and this type of equipment can generate a lot of static discharge. But my client insisted ESD could not be the problem, as the press operated in a humid environment. Furthermore, nobody had ever complained of getting an ESD shock.
My client suspected RF, as the trucks had VHF radios to communicate with the control room. In addition, floor personnel had handheld VHF radios. Although RFI was not my highest priority, I started there mainly to appease the customer.
Day one was spent probing the system with a handheld radio, trying to force a failure. My investigation included getting up into the ceiling to be sure the Ethernet cable was not located next to a radio antenna. No problems occurred, so we ruled out RFI.
Day two was spent checking out the power. Before my visit, I had the client install a power disturbance analyzer at one of the stations. I then examined the power wiring all the way back to the service panel. No anomalies were noted, so I ruled out any power disturbances.
On day three, I hauled out my ESD gun. When I hit the first truck loading station's display panel at about 4 kV, the controller hung.
Bingo. Upon lowering the voltage, subsequent stations would hang at 1,500 V -- well below the threshold of what a human can sense and very easy to generate. Further examination revealed the display panels were not bonded to the main cabinets. They were mounted on small rubber bushings to minimize vibration. This meant any ESD current was forced into the embedded controller via an interconnection cable. Bad ground connections were the culprit.
As a quick experiment, I cleaned off paint and connected a piece of copper tape between the display panel and station cabinet. Now we could withstand about 15 kV. As a redesign, a commercial copper strap was installed between the panel and cabinet.
A simple fix for a multimillion-dollar machine.
But wait, there's more. When the first system hung, someone yelled from the control room that the main control computer had crashed. I decided to check it out. The display screen was frozen, but it had multiple messages, such as: "System 1 (which I had hit first) down, system 9 down, system 11 down, attempting to reconstruct, unable to do so, etc." and "System 2 (which I hit next) down, system 7 down, system 5 down, etc."
Apparently, the ESD hits caused other dominoes to fall.
So I dutifully recorded the messages from the frozen screen and noted in my final report that the system software appeared to be unstable in the presence of ESD.
About three weeks later, I got a rather heated call from the VP of the software company. He wanted to know who I thought I was to point fingers at his software. After letting him blow off some steam, I shared my recorded screen data. Suddenly contrite, he admitted the software had some problems.
I explained that our ESD repair was just a patch, and that the software still deserved attention before something else toppled the dominoes again. He agreed, and we parted friends.
There are several lessons to be shared here:
- The client's main concern may not be the problem.
- ESD problems can occur at very low levels, even below human feeling.
- Software can be a factor (and also a fix).
- Record everything. You may need it later to cover your behind.
Daryl Gerke (PE) is a partner at Kimmel Gerke Associates Ltd., an EMI/EMC consulting and training firm. He and his business partner Bill Kimmel (PE) have solved or prevented hundreds of EMI/EMC/ESD problems across a wide range of industries. They have also trained more than 10,000 students in their public and in-house EMC classes. Gerke has a BSEE from the University of Nebraska and resides in Mesa, Ariz. You can contact him at
www.emiguru.com or via his consulting blog at www.jumptoconsulting.com.