Good advice! Studies have shown that grounding is a major contributor to systems related EMI problems. Always a good place to start.
The reason for checking RF first was at the customer's insistence. Based on the symptoms, I felt that was the least likely cause. But as a consultant, it is important to keep the client happy.
Although we checked the power interface next, the power disturbance analyzer had been in place for a week and showed no perturbations. One parameter monitored was the voltage between the power neutral and safety ground. So in a sense, we did start with the power grounding.
A common power grounding problem is connecting the power neutral to the safety ground at more than one point. This often inadvertantly happens in sub-panels, which was part of my investigation. We chased the power wiring all the way back to the service panel, the point where the single return/safety ground bond should be made.
The reason for checking ESD last was again at the customer's insistence that ESD was not an issue (I suspected otherwise.) The poor ground connection between the panel and cabinet was not readily apparent. At first glance, it looked OK. Only after removing a display for inspection did it become apparent that the display was not properly grounded to the cabinet. Oh well...
Chasing problems in the field is often a process of elimination. Grounding is a good place to start as it is often relatively easy to check (or at least eliminate from further consideration.) Thanks for your comment!
When a client insists on some inefficient or unnecessary procedure, there's a simple mantra to repeat to oneself to make the situation tolerable; "The meter is running, the meter is running.....". (As long as it's not a fixed-price contract.)
I totally agree, Adequate Grounding is the First Test. Not by ohm meter, Use a Megometer. Flow some real power and measure the resistance under load.
As a retired USAF Aircraft Electronic Tech I can relate to the pressure of "It can't be that" But eventually, trust your instints & training, it usually pays off.
Worst case for me: The F-111 had "Grounding Rings" where many cable grounds and part chassis grounds were gathered to be taken to Airframe ground (No Earth ground available in airborne items). The rings would build up corrosion internally until various systems would start getting flaky, and writeups (Pilot Complaints) were not very consistent. RADAR was a common early fail item due to the power flowing in that system.
Eventually it always seemed to come down to a grounding ring somewhere that had not been touched in over a decade. Crank them apart and capture the crud falling out for evidence. Clean with a wire wheel and then re-assemble. Conductive grease not allowed ! Problems in multiple systems would evaporate.
I once lost a PC motherboard to ESD while installing a new hard drive--never felt the zap. Ever since then, I use an ESD wrist strap when working on a computer and I try to keep one hand on the chassis at all times (not always possible).
I opend the 9-year-old desktop computer just the other day for its semiannual cleaning. It still runs.
Good plan! It doesn't take much ESD, particularly when you are inside a system.
In this case, my client kept insisting it couldn't be ESD, since nobody ever felt a shock. But like you, we proved that ESD does, in fact, exist below the threshold of human feeling. Fortunately no damage - just upsets.
Glad to see you're still using the old (proven?) technology. I guess we've both reached the age where old is OK - just like fine wine :-)
As we unveil EE Times’ 2015 Silicon 60 list, journalist & Silicon 60 researcher Peter Clarke hosts a conversation on startups in the electronics industry. Panelists Dan Armbrust (investment firm Silicon Catalyst), Andrew Kau (venture capital firm Walden International), and Stan Boland (successful serial entrepreneur, former CEO of Neul, Icera) join in the live debate.