Engineering Investigations
Comment
daleste
When I started, we were building microcontrollers in P-well cmos. It was very ...
W1PK
I remember substrate SCR latchup, all right. I tend to be a little more ...
We have met the enemy and it is management
Dave Braverman
4/29/2011 11:07 AM EDT
This is a story of a new factory. I was at a company whose product was a specialized computer used for telephone cost management, it collected the data for call routing and applied the cost tables so that connections could be billed correctly. As the senior design engineer in the test engineering department, I was in charge of getting all the test systems running and debugged for the new factory, with its state of the art conveyor system to deliver computer systems via static free trays to each test station. The pride and joy of our design were the Burn Tunnels, all 200 feet of them, where we weeded out the early failures before final test.
We had a pretty tight team, so meeting the launch date was easy, and senior management were happy as they watched the production pipeline fill station by station. Until we hit the 'problem'. During pretest, just before the units hit the Burn Tunnel bake, we were getting low battery indication failures on the CMOS backup battery, a freshly installed alkaline cell. This failure galvanized Test Engineering, Quality Control, and Design Engineering, since we couldn't ship any product! QC checked the battery date codes, they were all fresh, and instituted a cold storage ECN to lengthen inventory life. Design Engineering reviewed their design, started failure analysis on the CMOS memory device and didn't understand why the design, which previously worked fine for years, was now causing trouble. The design spec was two years of backup for the CMOS configuration, with current draw in the micro amp range. Obviously, blame was placed on the factory, and they plopped the problem squarely in Test Engineering's lap.
Of course, as it usually happens, this problem occurred on a Friday morning. The Senior Vice President of Sales marshaled us together to let us know that we would be working around the clock to solve this problem and get production moving by Monday morning.
I consider Test Engineering as a noble endeavor, a great training ground for developing critical thinking, which sometimes is sorely lacking with our Design brethren.
First up, just how much current is the CMOS memory using? Lucky for us we are talking about a DIP package, so we could pull the power pin and wire in a meter. Fifteen micro amps, right on spec! Next let's try stressing the device; temperature, voltage excursions, power cycling, static discharge, everything we could think of, some of which shouldn't be in print. Then, finally we were starring in disbelief as the chip started pulling fifteen milliamps; we had pushed the chip into 'latch up' mode, where the CMOS structure looks like an SCR. A quick calculation of battery capacity versus travel time from assembly to test confirmed that we could drain the battery in a few hours. Eureka, we now knew 'what' was going on, now all we had to figure out was the 'how'. Easier said than done.
We checked the assembly process from board to system. We checked the stations for static protection and how the assemblers were plugged in and static free. We checked everything, and found total adherence to the process, yet the cause eluded us. It was now 8 PM, we were hungry, plumb out of ideas, and I was sitting with one of my engineers at the pretest station with one of the computers sitting on its conductive tray, which was lassoed since the conveyor system was running. The computer was all metered up, we had a scope next to us and we were trying to plan our next move, as the engineer was inadvertently bouncing the scope probe in the free air under the conveyor. That's when I noticed the giant spikes on the scope screen. Yikes! Where were the 50 V negative spikes coming from?
First critical thought, measure the chassis! Yes sir, the chassis was seeing 50 V negative spikes! The board ground was tied to the chassis, so our little CMOS chip was seeing this negative voltage on its ground and of course it was getting thrown into latch up mode. Second critical thought, scope the roller on the conveyor, and wow, we are looking at a couple hundred volts. Now the path was obvious, the conductive antistatic tray picked up the charge from the roller, and capacitively coupled it to the chassis. Hey, we are close to solving the problem. How are the metal conveyor rollers getting voltage? We checked the wiring to the drive motors and everything was properly done, no shorts. Next we measured the steel roller and the mounting rail which was specified to be grounded, it wasn't. Each roller was not connected to the frame. Getting in touch with our inner scientist, we noticed that the drive mechanism was a nylon pulley with a rubber drive belt connected to the steel roller. A perfect Van de Graf generator! And we had 15,000 of them! It was time to go home, since we had to show up on Saturday to report our progress to the Senior VP.
Early Saturday morning, we had the conveyor contractor install grounding springs on all but 6 of the rollers, metered up a system, and were ready for our demo. The Senior VP wanted the full explanation, which we were happy to provide, with the demo as the topping on the cake. The metered system traveled by tray down the conveyor as we watched the current, and just as we had sleuthed out, as soon as it hit the six rollers we had not yet modified, bang, the current went to fifteen milliamps.
Truth will out, as we found out from our conveyor vendor, the Senior VP had complained to them that the rollers were noisy, so they changed the original grounding steel ball bearings with nylon ones, isolating the rollers and turning them into fifteen thousand Van de Grafs.
We had met the enemy, and it's management.
David Braverman (Director of Sales & Marketing) 38 years of experience in engineering design , production and test management, regional management of application engineers and salesman, manager of corporate applications, strategic marketing manager for microcontroller business unit, independent consultant, director of sales and marketing. Dave has managed production and test of laser instruments, managed test engineering design for a major computer manufacturer, and program managed design consultants, design programs with manufacturing in Asia, and trade shows. He has given training on Statistical Process Control. He is experienced with technology, manufacturing, program management, managing worldwide teams and a Senior Member of IEEE.
We had a pretty tight team, so meeting the launch date was easy, and senior management were happy as they watched the production pipeline fill station by station. Until we hit the 'problem'. During pretest, just before the units hit the Burn Tunnel bake, we were getting low battery indication failures on the CMOS backup battery, a freshly installed alkaline cell. This failure galvanized Test Engineering, Quality Control, and Design Engineering, since we couldn't ship any product! QC checked the battery date codes, they were all fresh, and instituted a cold storage ECN to lengthen inventory life. Design Engineering reviewed their design, started failure analysis on the CMOS memory device and didn't understand why the design, which previously worked fine for years, was now causing trouble. The design spec was two years of backup for the CMOS configuration, with current draw in the micro amp range. Obviously, blame was placed on the factory, and they plopped the problem squarely in Test Engineering's lap.
Of course, as it usually happens, this problem occurred on a Friday morning. The Senior Vice President of Sales marshaled us together to let us know that we would be working around the clock to solve this problem and get production moving by Monday morning.
I consider Test Engineering as a noble endeavor, a great training ground for developing critical thinking, which sometimes is sorely lacking with our Design brethren.
First up, just how much current is the CMOS memory using? Lucky for us we are talking about a DIP package, so we could pull the power pin and wire in a meter. Fifteen micro amps, right on spec! Next let's try stressing the device; temperature, voltage excursions, power cycling, static discharge, everything we could think of, some of which shouldn't be in print. Then, finally we were starring in disbelief as the chip started pulling fifteen milliamps; we had pushed the chip into 'latch up' mode, where the CMOS structure looks like an SCR. A quick calculation of battery capacity versus travel time from assembly to test confirmed that we could drain the battery in a few hours. Eureka, we now knew 'what' was going on, now all we had to figure out was the 'how'. Easier said than done.
We checked the assembly process from board to system. We checked the stations for static protection and how the assemblers were plugged in and static free. We checked everything, and found total adherence to the process, yet the cause eluded us. It was now 8 PM, we were hungry, plumb out of ideas, and I was sitting with one of my engineers at the pretest station with one of the computers sitting on its conductive tray, which was lassoed since the conveyor system was running. The computer was all metered up, we had a scope next to us and we were trying to plan our next move, as the engineer was inadvertently bouncing the scope probe in the free air under the conveyor. That's when I noticed the giant spikes on the scope screen. Yikes! Where were the 50 V negative spikes coming from?
First critical thought, measure the chassis! Yes sir, the chassis was seeing 50 V negative spikes! The board ground was tied to the chassis, so our little CMOS chip was seeing this negative voltage on its ground and of course it was getting thrown into latch up mode. Second critical thought, scope the roller on the conveyor, and wow, we are looking at a couple hundred volts. Now the path was obvious, the conductive antistatic tray picked up the charge from the roller, and capacitively coupled it to the chassis. Hey, we are close to solving the problem. How are the metal conveyor rollers getting voltage? We checked the wiring to the drive motors and everything was properly done, no shorts. Next we measured the steel roller and the mounting rail which was specified to be grounded, it wasn't. Each roller was not connected to the frame. Getting in touch with our inner scientist, we noticed that the drive mechanism was a nylon pulley with a rubber drive belt connected to the steel roller. A perfect Van de Graf generator! And we had 15,000 of them! It was time to go home, since we had to show up on Saturday to report our progress to the Senior VP.
Early Saturday morning, we had the conveyor contractor install grounding springs on all but 6 of the rollers, metered up a system, and were ready for our demo. The Senior VP wanted the full explanation, which we were happy to provide, with the demo as the topping on the cake. The metered system traveled by tray down the conveyor as we watched the current, and just as we had sleuthed out, as soon as it hit the six rollers we had not yet modified, bang, the current went to fifteen milliamps.
Truth will out, as we found out from our conveyor vendor, the Senior VP had complained to them that the rollers were noisy, so they changed the original grounding steel ball bearings with nylon ones, isolating the rollers and turning them into fifteen thousand Van de Grafs.
We had met the enemy, and it's management.
David Braverman (Director of Sales & Marketing) 38 years of experience in engineering design , production and test management, regional management of application engineers and salesman, manager of corporate applications, strategic marketing manager for microcontroller business unit, independent consultant, director of sales and marketing. Dave has managed production and test of laser instruments, managed test engineering design for a major computer manufacturer, and program managed design consultants, design programs with manufacturing in Asia, and trade shows. He has given training on Statistical Process Control. He is experienced with technology, manufacturing, program management, managing worldwide teams and a Senior Member of IEEE.
|
|
|
|
|
Navigate to related information
Rick_Hille
4/29/2011 2:17 PM EDT
A masterpiece of keen observation and deductive reasoning, leveraging a good background in physics. That your Sr-VEEP didn't give you a "career wedgie" for making him/her look foolish is a good example of political puckhandling, assuming it was the very same Sr-VEEP that complained about the noise.
On the technology side, although its gotten better over the years through awareness by chip vendors, I swear there were some CMOS chips out there that latched up if you gave them a stern look! I have an old CASIO Data Bank watch I still wear that occasionally latches its beeper ON when I slip off my coat on a dry, cold winter day. Luckily a single button press clears it, otherwise it would be a battery shark.
Sign in to Reply
no clever name
4/29/2011 4:24 PM EDT
many years and jobs ago we were temperature testing a load cell application, the in final testing, the strain gages were practially blasted off the beam. Turns out a 200 volt spike on the liner of the temperature chamber when the power cycled. Careful shielding and insulation fixed the problem and it was a very late night.
Rod
Sign in to Reply
David Ashton
4/29/2011 7:50 PM EDT
I have an acronym for these guys - PSMs - Pig Stupid Managers. Especially good when you can prove it! Great story.
Sign in to Reply
http://www.lulu.com/spotlight/poconoarmchairreview
4/30/2011 3:56 AM EDT
It would have been fun to put the conveyor belt on high speed before the grounding springs were put on, to see if it put out any lightning bolts.
Sign in to Reply
daleste
4/30/2011 10:36 AM EDT
Great story. I love to hear these because it helps me think of ideas when I have issues I need to solve.
Sign in to Reply
Sheetal.Pandey
4/30/2011 12:22 PM EDT
What a story...all i can say is it so true..
Sign in to Reply
zeeglen
4/30/2011 1:39 PM EDT
Great story!
Cost of line shutdown: tens of thousands?
Cost of replacing 15,000 roller bearings: hundreds of thousands?
The look on the Senior VP's face when told he was the cause of the problem that he is forcing others to resolve by Monday morning: Priceless!
Just two questions - how did the conveyor contractor manage to get grounding springs installed on 14,994 rollers in one Saturday morning, or were just enough rollers grounded to accomplish the demo? And was that the permanent solution (seems that springs would eventually wear through), or did the bearings get changed back to steel?
Nice to see that the author eventually got into management and became a Director of Sales and Marketing. These positions SHOULD be filled by those who have come up through the technical ranks with competence, then problems like that described would not happen.
Sign in to Reply
David Ashton
4/30/2011 8:31 PM EDT
Right on Glen! Especially re those who came up through the ranks. I just posted on the survey thread saying almost exactly that.
Sign in to Reply
WKetel
4/30/2011 9:54 PM EDT
Good detective work, indeed! And isn't it interesting how those who should stay away instead cause problems by making changes that are certainly not backed up by engineering. It is even more fun when they keep it a secret. It sometimes demonstrates, at least to those who can learn, that engineering design is far more than just drawing straight lines and doing neat lettering.
Sign in to Reply
DMcCunney
5/3/2011 12:12 AM EDT
While I have a dim view of "senior" management, the moral of this story seems to be complexity and compartmentalization.
The Sr. VP only knew the original bearings were noisy, and asked for quieter ones. He had no idea why the original design was spec'd as it was. (And in fairness, why should he? Not his problem.)
The conveyer manufacturer had the specs, but had they been told *why* those particular specs were given? They just filled the customer's order. So when the Sr. VP complained about the noise, they changed the bearings, blissfully unaware it would cause problems. You can argue that someone on their end should have raised a query about it, but hey, a Sr. VP at the customer site changes the specs...
Problems occurred because each group knew only it's particular piece of the puzzle, and no one with an overview was in a position to say "Wait a minute!" when the Sr. VP meddled.
Sign in to Reply
David Ashton
5/3/2011 4:45 AM EDT
And if the senior VP knew -anything- about electronics, he would have thought twice before changing to insulating bearings on a conveyor for sensitive electronic assemblies.
As above, this is why I call them PSMs....
Sign in to Reply
cdhmanning
5/4/2011 11:36 PM EDT
C'mon be fair guys...
From the story related here, the VP did not say something like "change to insulated bearings" or specify any particular bearing. He said "make the noise go away" and the belt manufacturer replaced the bearings with insulating ones.
It would be perfectly valid for the VP to assume that the conveyor engineers would change the noise without changing other operational parameters. After all, anybody that has been building conveyors should understand the issues of static (even in non electronic industries) and if they change the bearings should then design in new grounding paths.
I'll put the blame squarely on the conveyor engineers.
In any industry, the only people that never screw up are the people doing NOTHING!
Sign in to Reply
Rick_Hille
5/4/2011 1:56 PM EDT
DM has a good point. I too would not be so quick to ID the VEEP as "the enemy...", to paraphrase the Author. If it came down to nixing the grounding option due to cost, assuming the conveyor vendor offered such an option with their nylon bearings, then yes, a rotten fruit barrage of the VEEP is justified. On the other hand, the request for a quieter conveyor may have been out of genuine concern, or even acting on suggestions from the line staff, to reduce noise stress on line personnel. Whether EE or not, the conveyor grounding issue may have been missed if it is not a common knowledge kind of point in the EM industry. Perhaps the VEEP is guilty of not consulting the line manager before requesting the nylon bearings, but I hate to say, EE's do not have a monopoly on collaborative and consesus thinking either.
Sign in to Reply
charly5139
5/8/2011 2:30 AM EDT
Oh Lord, forgive them, they don't know what they are doing!
A controlled process (in this case installing/modifying of product) that is controlled by "calls from outside" is not a controlled process.
Sign in to Reply
David Ashton
5/8/2011 4:45 AM EDT
Guys, you're trying to defend the indefensible. Yes, the conveyor manufacturere did what he was asked for. He probably didn't know what the conveyor was to be used for and even if he did, he's a conveyor guy, not an electronics guy.
And yes, the Veep has every right to ASK if the bearings can be made quieter. Unless he was an engineer (sounds like he wasn't) he should have ASKED about it and deferred to the opinion of someone who WAS an engineer and knew what the project was about.
I agree we don't have ALL the facts here, but with what we do have, it does sound like the Veep was interfering in the project without consulting anyone who was running it.
And that is a total no-no.
Sign in to Reply
GordonScott
5/9/2011 4:47 AM EDT
Perhaps if the VP had talked to the engineering guys instead of going past them, the problem would have been seen before it happenned.
I can't help thinking that in an antistatic environment, the conveyor belts themselves should be antistatic and there should be one or more dedicated discharge roller(s) that are verified as working. But then the former may already be true and if the rollers were all supposed to be conductive, maybe my thought is unreasonable.
Sign in to Reply
W1PK
5/23/2011 5:19 PM EDT
I remember substrate SCR latchup, all right. I tend to be a little more paranoid about designing the circuit to be robust against that kind of problem in the first place. Carefully placed TVSs, clamp diodes, minimizing inductance from the chip to the ground plane, sometimes Faraday cages and eddy current shields to stop electric and magnetic fields from getting in, that sort of thing. I've salvaged a good many product designs with EMI hardening modifications.
Sign in to Reply
daleste
5/23/2011 9:41 PM EDT
When I started, we were building microcontrollers in P-well cmos. It was very prone to latch up. I had problems on the tester with bondwires melting due to latch up. The ceramic parts would be dead after that, but the plastic parts would recover. The bond wire would melt and reform when it cooled. It took some work to figure out what was going on. Once I decapped a ceramic part, I knew.
Sign in to Reply