This is a story of a new factory. I was at a company whose product was a specialized computer used for telephone cost management, it collected the data for call routing and applied the cost tables so that connections could be billed correctly. As the senior design engineer in the test engineering department, I was in charge of getting all the test systems running and debugged for the new factory, with its state of the art conveyor system to deliver computer systems via static free trays to each test station. The pride and joy of our design were the Burn Tunnels, all 200 feet of them, where we weeded out the early failures before final test.
We had a pretty tight team, so meeting the launch date was easy, and senior management were happy as they watched the production pipeline fill station by station. Until we hit the 'problem'. During pretest, just before the units hit the Burn Tunnel bake, we were getting low battery indication failures on the CMOS backup battery, a freshly installed alkaline cell. This failure galvanized Test Engineering, Quality Control, and Design Engineering, since we couldn't ship any product! QC checked the battery date codes, they were all fresh, and instituted a cold storage ECN to lengthen inventory life. Design Engineering reviewed their design, started failure analysis on the CMOS memory device and didn't understand why the design, which previously worked fine for years, was now causing trouble. The design spec was two years of backup for the CMOS configuration, with current draw in the micro amp range. Obviously, blame was placed on the factory, and they plopped the problem squarely in Test Engineering's lap.
Of course, as it usually happens, this problem occurred on a Friday morning. The Senior Vice President of Sales marshaled us together to let us know that we would be working around the clock to solve this problem and get production moving by Monday morning.
I consider Test Engineering as a noble endeavor, a great training ground for developing critical thinking, which sometimes is sorely lacking with our Design brethren.
First up, just how much current is the CMOS memory using? Lucky for us we are talking about a DIP package, so we could pull the power pin and wire in a meter. Fifteen micro amps, right on spec! Next let's try stressing the device; temperature, voltage excursions, power cycling, static discharge, everything we could think of, some of which shouldn't be in print. Then, finally we were starring in disbelief as the chip started pulling fifteen milliamps; we had pushed the chip into 'latch up' mode, where the CMOS structure looks like an SCR. A quick calculation of battery capacity versus travel time from assembly to test confirmed that we could drain the battery in a few hours. Eureka, we now knew 'what' was going on, now all we had to figure out was the 'how'. Easier said than done.
We checked the assembly process from board to system. We checked the stations for static protection and how the assemblers were plugged in and static free. We checked everything, and found total adherence to the process, yet the cause eluded us. It was now 8 PM, we were hungry, plumb out of ideas, and I was sitting with one of my engineers at the pretest station with one of the computers sitting on its conductive tray, which was lassoed since the conveyor system was running. The computer was all metered up, we had a scope next to us and we were trying to plan our next move, as the engineer was inadvertently bouncing the scope probe in the free air under the conveyor. That's when I noticed the giant spikes on the scope screen. Yikes! Where were the 50 V negative spikes coming from?
First critical thought, measure the chassis! Yes sir, the chassis was seeing 50 V negative spikes! The board ground was tied to the chassis, so our little CMOS chip was seeing this negative voltage on its ground and of course it was getting thrown into latch up mode. Second critical thought, scope the roller on the conveyor, and wow, we are looking at a couple hundred volts. Now the path was obvious, the conductive antistatic tray picked up the charge from the roller, and capacitively coupled it to the chassis. Hey, we are close to solving the problem. How are the metal conveyor rollers getting voltage? We checked the wiring to the drive motors and everything was properly done, no shorts. Next we measured the steel roller and the mounting rail which was specified to be grounded, it wasn't. Each roller was not connected to the frame. Getting in touch with our inner scientist, we noticed that the drive mechanism was a nylon pulley with a rubber drive belt connected to the steel roller. A perfect Van de Graf generator! And we had 15,000 of them! It was time to go home, since we had to show up on Saturday to report our progress to the Senior VP.
Early Saturday morning, we had the conveyor contractor install grounding springs on all but 6 of the rollers, metered up a system, and were ready for our demo. The Senior VP wanted the full explanation, which we were happy to provide, with the demo as the topping on the cake. The metered system traveled by tray down the conveyor as we watched the current, and just as we had sleuthed out, as soon as it hit the six rollers we had not yet modified, bang, the current went to fifteen milliamps.
Truth will out, as we found out from our conveyor vendor, the Senior VP had complained to them that the rollers were noisy, so they changed the original grounding steel ball bearings with nylon ones, isolating the rollers and turning them into fifteen thousand Van de Grafs.
We had met the enemy, and it's management.
David Braverman (Director of Sales & Marketing) 38 years of experience in engineering design , production and test management, regional management of application engineers and salesman, manager of corporate applications, strategic marketing manager for microcontroller business unit, independent consultant, director of sales and marketing. Dave has managed production and test of laser instruments, managed test engineering design for a major computer manufacturer, and program managed design consultants, design programs with manufacturing in Asia, and trade shows. He has given training on Statistical Process Control. He is experienced with technology, manufacturing, program management, managing worldwide teams and a Senior Member of IEEE.
When I started, we were building microcontrollers in P-well cmos. It was very prone to latch up. I had problems on the tester with bondwires melting due to latch up. The ceramic parts would be dead after that, but the plastic parts would recover. The bond wire would melt and reform when it cooled. It took some work to figure out what was going on. Once I decapped a ceramic part, I knew.
I remember substrate SCR latchup, all right. I tend to be a little more paranoid about designing the circuit to be robust against that kind of problem in the first place. Carefully placed TVSs, clamp diodes, minimizing inductance from the chip to the ground plane, sometimes Faraday cages and eddy current shields to stop electric and magnetic fields from getting in, that sort of thing. I've salvaged a good many product designs with EMI hardening modifications.
Perhaps if the VP had talked to the engineering guys instead of going past them, the problem would have been seen before it happenned.
I can't help thinking that in an antistatic environment, the conveyor belts themselves should be antistatic and there should be one or more dedicated discharge roller(s) that are verified as working. But then the former may already be true and if the rollers were all supposed to be conductive, maybe my thought is unreasonable.
Guys, you're trying to defend the indefensible. Yes, the conveyor manufacturere did what he was asked for. He probably didn't know what the conveyor was to be used for and even if he did, he's a conveyor guy, not an electronics guy.
And yes, the Veep has every right to ASK if the bearings can be made quieter. Unless he was an engineer (sounds like he wasn't) he should have ASKED about it and deferred to the opinion of someone who WAS an engineer and knew what the project was about.
I agree we don't have ALL the facts here, but with what we do have, it does sound like the Veep was interfering in the project without consulting anyone who was running it.
And that is a total no-no.
C'mon be fair guys...
From the story related here, the VP did not say something like "change to insulated bearings" or specify any particular bearing. He said "make the noise go away" and the belt manufacturer replaced the bearings with insulating ones.
It would be perfectly valid for the VP to assume that the conveyor engineers would change the noise without changing other operational parameters. After all, anybody that has been building conveyors should understand the issues of static (even in non electronic industries) and if they change the bearings should then design in new grounding paths.
I'll put the blame squarely on the conveyor engineers.
In any industry, the only people that never screw up are the people doing NOTHING!
DM has a good point. I too would not be so quick to ID the VEEP as "the enemy...", to paraphrase the Author. If it came down to nixing the grounding option due to cost, assuming the conveyor vendor offered such an option with their nylon bearings, then yes, a rotten fruit barrage of the VEEP is justified. On the other hand, the request for a quieter conveyor may have been out of genuine concern, or even acting on suggestions from the line staff, to reduce noise stress on line personnel. Whether EE or not, the conveyor grounding issue may have been missed if it is not a common knowledge kind of point in the EM industry. Perhaps the VEEP is guilty of not consulting the line manager before requesting the nylon bearings, but I hate to say, EE's do not have a monopoly on collaborative and consesus thinking either.
And if the senior VP knew -anything- about electronics, he would have thought twice before changing to insulating bearings on a conveyor for sensitive electronic assemblies.
As above, this is why I call them PSMs....
While I have a dim view of "senior" management, the moral of this story seems to be complexity and compartmentalization.
The Sr. VP only knew the original bearings were noisy, and asked for quieter ones. He had no idea why the original design was spec'd as it was. (And in fairness, why should he? Not his problem.)
The conveyer manufacturer had the specs, but had they been told *why* those particular specs were given? They just filled the customer's order. So when the Sr. VP complained about the noise, they changed the bearings, blissfully unaware it would cause problems. You can argue that someone on their end should have raised a query about it, but hey, a Sr. VP at the customer site changes the specs...
Problems occurred because each group knew only it's particular piece of the puzzle, and no one with an overview was in a position to say "Wait a minute!" when the Sr. VP meddled.
Good detective work, indeed! And isn't it interesting how those who should stay away instead cause problems by making changes that are certainly not backed up by engineering. It is even more fun when they keep it a secret. It sometimes demonstrates, at least to those who can learn, that engineering design is far more than just drawing straight lines and doing neat lettering.
Join our online Radio Show on Friday 11th July starting at 2:00pm Eastern, when EETimes editor of all things fun and interesting, Max Maxfield, and embedded systems expert, Jack Ganssle, will debate as to just what is, and is not, and embedded system.