We have met the enemy and it is management
4/29/2011 11:07 AM EDT
This is a story of a new factory. I was at a company whose product was a specialized computer used for telephone cost management, it collected the data for call routing and applied the cost tables so that connections could be billed correctly. As the senior design engineer in the test engineering department, I was in charge of getting all the test systems running and debugged for the new factory, with its state of the art conveyor system to deliver computer systems via static free trays to each test station. The pride and joy of our design were the Burn Tunnels, all 200 feet of them, where we weeded out the early failures before final test.
We had a pretty tight team, so meeting the launch date was easy, and senior management were happy as they watched the production pipeline fill station by station. Until we hit the 'problem'. During pretest, just before the units hit the Burn Tunnel bake, we were getting low battery indication failures on the CMOS backup battery, a freshly installed alkaline cell. This failure galvanized Test Engineering, Quality Control, and Design Engineering, since we couldn't ship any product! QC checked the battery date codes, they were all fresh, and instituted a cold storage ECN to lengthen inventory life. Design Engineering reviewed their design, started failure analysis on the CMOS memory device and didn't understand why the design, which previously worked fine for years, was now causing trouble. The design spec was two years of backup for the CMOS configuration, with current draw in the micro amp range. Obviously, blame was placed on the factory, and they plopped the problem squarely in Test Engineering's lap.
Of course, as it usually happens, this problem occurred on a Friday morning. The Senior Vice President of Sales marshaled us together to let us know that we would be working around the clock to solve this problem and get production moving by Monday morning.
I consider Test Engineering as a noble endeavor, a great training ground for developing critical thinking, which sometimes is sorely lacking with our Design brethren.
First up, just how much current is the CMOS memory using? Lucky for us we are talking about a DIP package, so we could pull the power pin and wire in a meter. Fifteen micro amps, right on spec! Next let's try stressing the device; temperature, voltage excursions, power cycling, static discharge, everything we could think of, some of which shouldn't be in print. Then, finally we were starring in disbelief as the chip started pulling fifteen milliamps; we had pushed the chip into 'latch up' mode, where the CMOS structure looks like an SCR. A quick calculation of battery capacity versus travel time from assembly to test confirmed that we could drain the battery in a few hours. Eureka, we now knew 'what' was going on, now all we had to figure out was the 'how'. Easier said than done.
We checked the assembly process from board to system. We checked the stations for static protection and how the assemblers were plugged in and static free. We checked everything, and found total adherence to the process, yet the cause eluded us. It was now 8 PM, we were hungry, plumb out of ideas, and I was sitting with one of my engineers at the pretest station with one of the computers sitting on its conductive tray, which was lassoed since the conveyor system was running. The computer was all metered up, we had a scope next to us and we were trying to plan our next move, as the engineer was inadvertently bouncing the scope probe in the free air under the conveyor. That's when I noticed the giant spikes on the scope screen. Yikes! Where were the 50 V negative spikes coming from?
First critical thought, measure the chassis! Yes sir, the chassis was seeing 50 V negative spikes! The board ground was tied to the chassis, so our little CMOS chip was seeing this negative voltage on its ground and of course it was getting thrown into latch up mode. Second critical thought, scope the roller on the conveyor, and wow, we are looking at a couple hundred volts. Now the path was obvious, the conductive antistatic tray picked up the charge from the roller, and capacitively coupled it to the chassis. Hey, we are close to solving the problem. How are the metal conveyor rollers getting voltage? We checked the wiring to the drive motors and everything was properly done, no shorts. Next we measured the steel roller and the mounting rail which was specified to be grounded, it wasn't. Each roller was not connected to the frame. Getting in touch with our inner scientist, we noticed that the drive mechanism was a nylon pulley with a rubber drive belt connected to the steel roller. A perfect Van de Graf generator! And we had 15,000 of them! It was time to go home, since we had to show up on Saturday to report our progress to the Senior VP.
Early Saturday morning, we had the conveyor contractor install grounding springs on all but 6 of the rollers, metered up a system, and were ready for our demo. The Senior VP wanted the full explanation, which we were happy to provide, with the demo as the topping on the cake. The metered system traveled by tray down the conveyor as we watched the current, and just as we had sleuthed out, as soon as it hit the six rollers we had not yet modified, bang, the current went to fifteen milliamps.
Truth will out, as we found out from our conveyor vendor, the Senior VP had complained to them that the rollers were noisy, so they changed the original grounding steel ball bearings with nylon ones, isolating the rollers and turning them into fifteen thousand Van de Grafs.
We had met the enemy, and it's management.
David Braverman (Director of Sales & Marketing) 38 years of experience in engineering design , production and test management, regional management of application engineers and salesman, manager of corporate applications, strategic marketing manager for microcontroller business unit, independent consultant, director of sales and marketing. Dave has managed production and test of laser instruments, managed test engineering design for a major computer manufacturer, and program managed design consultants, design programs with manufacturing in Asia, and trade shows. He has given training on Statistical Process Control. He is experienced with technology, manufacturing, program management, managing worldwide teams and a Senior Member of IEEE.