Design Con 2015
Breaking News
Comments
Dave33
User Rank
Rookie
re: It always worked before, so you broke it
Dave33   5/18/2011 8:30:24 PM
NO RATINGS
I've written and debugged firmware for many years and I can really relate to this story. Solving problems like those described in the story take time and a very good engineer. Unfortunately there's no way to measure the difficulty of a problem so there's no way to measure the value of the solution. Managers usually just look at how long you took and draw their own conclusions.

jnissen
User Rank
Manager
re: It always worked before, so you broke it
jnissen   5/18/2011 10:20:04 PM
NO RATINGS
Been there done that! Don't care to repeat it!

old account Frank Eory
User Rank
Rookie
re: It always worked before, so you broke it
old account Frank Eory   5/18/2011 10:35:10 PM
NO RATINGS
Interesting that in a new system -- new hardware and new software -- what ultimately turned out to be a signal integrity problem was blamed on software. "Mechanical and hardware assemblers said it was obviously software." And management simply took their word for it, without any data to back up that claim?

cdhmanning
User Rank
Rookie
re: It always worked before, so you broke it
cdhmanning   5/23/2011 3:14:40 AM
NO RATINGS
There are two reasons for that: 1) It is relatively easy to see how hardware works and much harder to see how software works. That means people will always tend to believe the problem is where they can't see it (ie. in the software). 2) It is also a matter of hope. Software errors can be fixed on short deadlines, but not hardware errors. Thus management/sales people always hope it is a software problem. When a product is on stop ship they will readily believe anything that gives them hope that a solution is at hand. Even if it is not really a software problem, there is often a software solution. Whenever this happens it becomes a software problem. I once had to fix a lubrication problem in software. Without that, many idetms would have had to be recalled and scrapped costing the company a few hundred thousand dollars. On another occasion a mechanical stop was repositioned. This could have cost approx $100k and 4 months to fix mechanically meaning the company would have lost a few million in sales. I fiddled for a day and found a software solution. At the end of the day, all that matters is that a solution is found. The customer doesn't care what is broken or who's fault it is. Just work together to find a solution. Everyone is trying to earn money for the same company.

DutchUncle
User Rank
Rookie
re: It always worked before, so you broke it
DutchUncle   5/23/2011 12:41:41 PM
NO RATINGS
The trouble is, when you find a software WORKAROUND (emphasis important) to a hardware PROBLEM, and all that people remember is "we had to rev the software to fix it", the software side loses respect - including at review time.

cdhmanning
User Rank
Rookie
re: It always worked before, so you broke it
cdhmanning   5/23/2011 6:48:51 PM
NO RATINGS
Why should the software guys lose respect? Make sure that it does not get framed that way. Make sure that the software guys get the credit for turning around a multi-million dollar loss. If you want recognition in a company then the best way to do that is to make an association between what you do and the performance of the company. Demonstrate **value**. In your performance review, point out the value of what you did.

J Kosin
User Rank
Rookie
re: It always worked before, so you broke it
J Kosin   5/27/2011 2:35:47 PM
NO RATINGS
The software guy looses respect, because he caused the product to be late in the first place. It doesn't matter if it was a HARDWARE problem, we get the short end of the stick; because we fixed the problem after everyone said the product was ready to ship.

Tombo0
User Rank
Rookie
re: It always worked before, so you broke it
Tombo0   5/19/2011 1:27:39 PM
NO RATINGS
I deal with this everyday

t.alex
User Rank
Rookie
re: It always worked before, so you broke it
t.alex   5/29/2011 1:06:15 PM
NO RATINGS
It is important the sw guy pick up some hardware debugging to prove themselves 'innocent' :)

zeeglen
User Rank
Blogger
re: It always worked before, so you broke it
zeeglen   5/20/2011 6:41:20 PM
NO RATINGS
This reminds me of a time when a watchdog timer would sporadically time out when a processor took too long to execute a command/response sequence on a control bus. This was a new product in both HW and SW, and the late night sessions in the labs were not burdened by "it's a HW fault / no it's a SW fault". We just admitted that nobody knew yet where the fault was and we had to work together to find it. Finally, using an analog scope (digital scopes had not been invented yet) the SW guy and myself saw an event whiz by that the timeout occurred within the 1 second allocated time of the hardware. I took another look at the hardware, a long-chain ripple counter and realized that the guy who designed this had done the stage count based on a complete cycle at the final stage. He forgot that the timeout actually occurred on the rising edge HALFWAY through the cycle. Solution: knife and green wire. Lesson learned: Never assume HW or SW. Test, test, test....

t.alex
User Rank
Rookie
re: It always worked before, so you broke it
t.alex   5/29/2011 1:07:22 PM
NO RATINGS
Isn't it true most of the time the sw guy do the test and debugging instead of the hardware?

zeeglen
User Rank
Blogger
re: It always worked before, so you broke it
zeeglen   5/29/2011 4:53:51 PM
NO RATINGS
Depends on what is being tested and debugged.

jimfordbroadcom
User Rank
CEO
re: It always worked before, so you broke it
jimfordbroadcom   5/20/2011 7:02:34 PM
NO RATINGS
Well, I've worked with enough boneheaded software types who didn't know anything about hardware to say that you are the exception, Tim. Embedded engineers who know both are worth their weight in gold, even if boneheaded managers don't realize it. I totally agree with zeeglen; forget the finger-pointing and just figure out what's not working. Sometimes it's both H/W and S/W!

J Kosin
User Rank
Rookie
re: It always worked before, so you broke it
J Kosin   5/27/2011 2:38:59 PM
NO RATINGS
Yes, sometimes it is both. But we mustnot start pointing fingers before exploring the problem and what the cause is. I recently spent two days troubleshooting a problem and have worked with the HW engineer to try and determine the cause. So far, we have elimiated SW as being a possible cause to the issue.

DutchUncle
User Rank
Rookie
re: It always worked before, so you broke it
DutchUncle   5/20/2011 8:10:24 PM
NO RATINGS
Sometimes this is all in the software domain. Reviewing DOS driver code while writing new interface code on an ISA plug-in board, I found some code that seemed to have a race condition and asked about it. Since this had been written by one of the original product group, now a technical leader, and had been working for years, my question was disdained and dismissed out of hand. A mere few months later, intractable customer problems were blamed on my interface code. I finally traced it back the the same DOS driver code. Turned out this was the first customer PC system we had seen at whatever speed it was (500 MHz?) and it was the first time that the PC was fast enough to catch the race condition. Now it was still my fault for not having pushed harder when I found the problem before.

J Kosin
User Rank
Rookie
re: It always worked before, so you broke it
J Kosin   5/27/2011 2:40:46 PM
NO RATINGS
Unfortunately, it is a loose-loose situation. Sorry to bring you bad news.

Alan60
User Rank
Rookie
re: It always worked before, so you broke it
Alan60   5/20/2011 9:29:31 PM
NO RATINGS
Been there, done it, got the company golf shirt. Years ago we had a hydraulic problem, excess pressure when braking. The redesign fix was messy, even for hydraulics. The $1M machine was down and people were not happy. I suggested a SW workaround to control deceleration, which we did and were soon back up running. However, word got around that I had to fix the SW to get the machine running. Unfortunately now I have to think twice before helping others.

WKetel
User Rank
Rookie
re: It always worked before, so you broke it
WKetel   5/21/2011 12:40:48 AM
NO RATINGS
Nobody has an exclusive right to make the mistakes, that is for certain. The problems arise when either the product definition is fuzzy, or the designer makes a mistake, or the builders don't follow the drawings. The funniest one that I experienced was a fairly simple circuit that I had designed and then built. I carefully selected the resistors to all be in the middle of where they needed to be. But when the production model was built, it was miles out of specifications. The problem eventually was traced to the arrogant detailers who did the circuit drawing. They corrected the "Mistake" made by "That dumb engineer", and added the "K" that I had neglected to put on several low value resistors. So instead of 22 ohms and 470 ohms there was 22,000 ohms and 470,000 ohms. The shortcoming of my designs are that they don't work as intended unless they are built as designed. This was reported to my manager at my next performance review, when the problem with that product was brought up. The response was "OH".

Bob Lacovara
User Rank
Rookie
re: It always worked before, so you broke it
Bob Lacovara   5/23/2011 5:27:14 PM
NO RATINGS
Mr. Fyock, welcome to the monkey house. Your best strategy would have been to quit, although for sure that is not always a feasible solution. Since idiotic management and moronic sales types have a tendency to butt into engineering (software and hardware) you might want to consider this advice that I was once offered: "the time to start looking for a job is the day you land your new one." Sounds cynical? Wait until you've been in the field for, let's see, 11 minus 74, ah, 37 years, and consider it again. Well, in any event, we've all been there...

GarySXT
User Rank
Freelancer
re: It always worked before, so you broke it
GarySXT   5/26/2011 5:51:42 PM
NO RATINGS
In the groups I manage, when there is a bug both hardware and software are considered guilty until proven innocent. Hardware and software engineers are expected to work as a team. As mentioned above software often fixes hardware problems. Also hardware debugging techniques are sometimes helpful in tracking down a software bug. Management should not allow the different disciplines to blame each other.

J Kosin
User Rank
Rookie
re: It always worked before, so you broke it
J Kosin   5/27/2011 2:58:19 PM
NO RATINGS
I look at it a bit differently. Without hardware, software is useless and without software, hardware is pretty lame. It is the cooperation of both the hardware and software that makes a well designed system; which is our goal ultimately for software and hardware.

LiketoBike
User Rank
CEO
re: It always worked before, so you broke it
LiketoBike   5/27/2011 6:26:01 PM
NO RATINGS
Software and hardware are often trained by management to blame each other :-) (As in: when the "real" mission to create and ship and SUCCEED [as a team] is derailed into the "fake" mission to grab credit and avoid fault, this blame-game usually results...)

jsandjs
User Rank
Rookie
re: It always worked before, so you broke it
jsandjs   6/9/2011 11:03:27 PM
NO RATINGS
This is the second story about the hardware induced noise I read here. Since noise mixed in a signal is so common and most of the time it is random, problems happen here and there somehow give a hint, even not strong and clear. A specific software in general will hardly produce random troubles here and there.

DutchUncle
User Rank
Rookie
re: It always worked before, so you broke it
DutchUncle   6/10/2011 12:29:50 PM
NO RATINGS
Fresh real-life example: A few weeks ago I released code that worked exactly as expected on prototype hardware. When used on the next board-spin of prototypes, it didn't work, which of course meant it was a software fault. Yesterday it was proven that not only was the circuit changed, but a wrong part had been used on the dozen proto builds, so it wasn't even the proper changed circuit. Put in the right part, and the software works again. Gee, what a concept.

oster
User Rank
Rookie
re: It always worked before, so you broke it
oster   11/3/2012 12:35:08 AM
NO RATINGS
The solution seems to be that SW guys need to learn more HW skills and vice versa. It's the ones that are at ease straddling disciplines that bridge the divide and I've certainly seen this in action.

The Gen
User Rank
Rookie
re: It always worked before, so you broke it
The Gen   6/16/2013 8:28:11 AM
NO RATINGS
I worked for a company for 12 years designing a range of fibre optic switches. (check out the spelling of fibre!) My bit was the opto-mechanics. I can sympathise deeply with the sentiments in these posts. Guys, I feel your pain. One of the posts earlier mentioned that it is obvious when mechanics isn't working, but not so software and electronics. In the fibre switch, the mechanics was small and delicate, and small mechanical disturbances (thermal, vibration, electrical noise) could make a difference but it wasn't obvious by observation. When there was a problem the blame was assigned thus:- 1. it's a mechanical problem, the mirror is in the wrong place (it was once many years previously but not on the five plus subsequent problems) 2. Right, the mechanics is OK, must be an electrical problem 3. Right the electrics is OK, must be a software problem Usually the 'head in charge would decide on what the root cause was and waste time and money trying to fix something that wasn't broken. The three disciplines had enough respect for each other to get on and sort the problem (eventually when the management "help" had evaporated when the solution was looking intractable) Oh, and I relate to the story above about the saw project that was estimated to take 2 months scheduled to be done in one month and quoted to the customer as 2 weeks. I don't work for that company any more :-)



Top Comments of the Week
Flash Poll
Like Us on Facebook

Datasheets.com Parts Search

185 million searchable parts
(please enter a part number or hit search to begin)
EE Life
Frankenstein's Fix, Teardowns, Sideshows, Design Contests, Reader Content & More
<b><a href=Betajet">

The Circle – The Future's Imperfect in the Present Tense
Betajet
5 comments
The Circle, a satirical, dystopian novel published in 2013 by San Francisco-based writer Dave Eggers, is about a large, very powerful technology company that combines aspects of Google, ...

Max Maxfield

Recommended Reads From the Engineer's Bookshelf
Max Maxfield
27 comments
I'm not sure if I read more than most folks or not, but I do I know that I spend quite a lot of time reading. I hate to be idle, so I always have a book or two somewhere about my person -- ...

Martin Rowe

Make This Engineering Museum a Reality
Martin Rowe
Post a comment
Vincent Valentine is a man on a mission. He wants to make the first house to ever have a telephone into a telephone museum. Without help, it may not happen.

Rich Quinnell

Making the Grade in Industrial Design
Rich Quinnell
16 comments
As every developer knows, there are the paper specifications for a product design, and then there are the real requirements. The paper specs are dry, bland, and rigidly numeric, making ...

Special Video Section
The LT8640 is a 42V, 5A synchronous step-down regulator ...
The LTC2000 high-speed DAC has low noise and excellent ...
How do you protect the load and ensure output continues to ...
General-purpose DACs have applications in instrumentation, ...
Linear Technology demonstrates its latest measurement ...
10:29
Demos from Maxim Integrated at Electronica 2014 show ...
Bosch CEO Stefan Finkbeiner shows off latest combo and ...
STMicroelectronics demoed this simple gesture control ...
Keysight shows you what signals lurk in real-time at 510MHz ...
TE Connectivity's clear-plastic, full-size model car shows ...
Why culture makes Linear Tech a winner.
Recently formed Architects of Modern Power consortium ...
Specially modified Corvette C7 Stingray responds to ex Indy ...
Avago’s ACPL-K30T is the first solid-state driver qualified ...
NXP launches its line of multi-gate, multifunction, ...
Doug Bailey, VP of marketing at Power Integrations, gives a ...
See how to ease software bring-up with DesignWare IP ...
DesignWare IP Prototyping Kits enable fast software ...
This video explores the LT3086, a new member of our LDO+ ...
In today’s modern electronic systems, the need for power ...