Engineering Pop Culture!
Comment
Bob Lacovara
Agreed, Mr.Acharya, the com style depends on the organization. The bridge ...
Sanjib.Acharya
Another lesson learned (or addendum to #2 in the document) could be a well ...
Software is from Mars, hardware is from Pluto
Glen Chenier
8/25/2010 6:49 PM EDT
It was the big day for the Software Team – they were scheduled to demonstrate their latest and greatest rendition of their SNMP network management software package to the Marketing Team. Months of intensive sweat-and-blood software design was about to be judged. The new hardware was… well, just hardware.
Our new prototype hardware was set up in the lab and was fully debugged, transmitting all the test traffic packets between multiport network hubs without error. This was a bit of a problem because the Software Team wanted to demonstrate how their Management Information Base (MIB) could tally and report all sorts of transmission anomalies – runt packets, CRC errored packets, collided packets, discombobulated packets – whatever. Unfortunately the Hardware Team had done its job superbly so there were no damaged packets to tally during the demonstration.
As a member of the Hardware Team I watched from the sidelines as the Software Team went through their dog-and-pony show. They demonstrated the various packet counters and potential alarm scenarios that would be flagged upon receipt of damaged packets in excess of preset thresholds. The Marketing Team was very impressed with how our product’s software was about to kick assorted competitors’ products into oblivion.
At the end of the Software Team’s presentation I realized that a significant feature had not been demonstrated, but rather than bring it up without sufficient data I waited until I had the equipment all to myself. With nobody watching, I simulated the backhoe-cutting-the-cable by removing the fiber optic cable that interconnected two hubs and waited for the alarm. And waited…and waited… What the heck? No alarm… After nothing happened I quietly conferred with a member of the Software Team. I asked him why the software did not raise an alarm when presented with my “loss of optical signal” hardware alarm bits. He replied “We don’t monitor those bits. It was never in the requirements for the MIB.”
Now that got me a bit annoyed since I had put a lot of effort into the fiber optic link design and had specifically included the optical carrier sense hardware circuit for alarm reporting purposes. After all, if any of the inter-hub fiber links were to go down hundreds of users would be cut off.
But when I approached the Director of Software about the omission, he stated that to design software specific to the optical link functions would have taken extra development time, so to save time he chose to keep all port software the same and as simple as possible. After all, zero packets did not necessarily constitute a network outage, it just meant that nobody was using the network. He could not comprehend the difference between an individual twisted pair port for a single user and an optical inter-hub port that served hundreds of users. So while the magnificent MIB was great at reporting damaged packets, it had no means of alarming due to a complete network failure that resulted in zero packets. BIG oversight.
The Marketing Team was quite upset to hear that the MIB was not capable of reporting something as simple as a broken optical link and insisted that the code be re-written to include monitoring and reporting of my optical carrier detect bits. It was not too long afterward that the Director of Software suddenly stopped coming into work and was never seen again.
Lessons learned:
1. * When a hardware function is designed into a product, be sure the Software Team understands why. Does not do much good to just write it into the Functional Specification, they might never read that page.
Then re-check partway through the project to be sure they got it right. 2. Always keep in contact with other design teams during a project even if their work does not directly impact your work. Sometimes design errors are more easily recognized by those who do not have to report to that team’s manager.
3. Recognizing another team’s design oversight early in a project saves a lot of trouble at the end of the project when all becomes integrated together.
* Addendum to Lesson 1. I had previously had an issue with this same Director of Software on an earlier project. I had designed a power supply into in a copper ring network interface to provide regeneration power on the data cables to neighboring ring interfaces to cover for remote AC power outages. I also included a power supply monitor bit for alarm reporting to the software. Then I discovered that the Software Director’s plan was to shut down the entire line interface if its power supply failed and take the hub offline. It took a lot of explaining to get him to comprehend that the power supply was for the neighboring hubs in the event of their AC power failure, and that a failure of the local power supply did NOT mean that the interface data transmission functions had gone down. The only thing that his software had to do was report a minor alarm since AC power failure fault coverage was no longer functional.
Glen Chenier is a design consultant based in Allen, TX.



WSOCT
8/26/2010 11:27 AM EDT
I think it’s fair to say that “Software is from Mars, and hardware is from Pluto”. I truly understand and appreciate the concerns highlighted by you.
Leaving out critical features in the name of “extra development time”, “not in requirements” and “out of scope” is a perfect recipe for disaster. Unfortunate as it is, scenarios such as these are quiet common in the embedded industry. The software and hardware teams are at loggerheads over what needs to be done and what can be left out.
As for the Marketing Team, I’m not sure if they truly appreciate the concerns of either developers or the customers. Either way, healthy collaboration with software and hardware teams is crucial as the product can’t run on software or hardware alone!
- Keith Schaub
Sign in to Reply
ttt3
8/26/2010 12:06 PM EDT
This is where a well-rounded lead engineer is very useful. Someone who can straddle the line between hardware and software; having experience with both sides. Too often I see projects go awry because the engineer in charge only has experience with one side (hardware or software), or even worse, when a non-technical "project manager" is put in charge of system requirements.
Sign in to Reply
ReneCardenas
8/26/2010 5:05 PM EDT
Well said, I hear the term bi-lingual as the skill to understand both sides of a project.
Not easy to acquire unless an individual has had the opportunity to wear the shoes and done some grunt work in both roles.
Becoming self-aware of the numerous issues is not trivial matter.
I exalt all engineers and programmers never to blame the other side unless ready to receive in kind the same retorts.
Expect “mud on your face” if you enter the gray area and your first knee-jerk reaction is to blame the opposite team.
Sign in to Reply
Neo1
8/27/2010 2:00 AM EDT
I agree that this kind of corner cutting is disastrous. But this chasm between the two roles is also due to lack of design considerations at the planning stage itself. It is not rare to find hardware always having some bits here and there which don't represent anything or also the way extra bits gets added later into the implementation phase which gets left behind by software team because it is not in their common requirements!
Sign in to Reply
DBG2
8/27/2010 2:19 PM EDT
Personally, I have no patience for this commonplace but IMHO artificial hardware/software divide. If you are going to work in embedded systems, then you really should have a good understanding of both sides of the system.
Sign in to Reply
BigTech
8/27/2010 2:46 PM EDT
Note that this cuts both ways: when hardware designers don't understand critical software requirements, or decide to do the software people a "favor" and add a feature they believe will be "useful", trouble can't be far behind. As others have said, the teams need to work together to ensure that all features are covered.
Sign in to Reply
bkshellboy
8/27/2010 2:57 PM EDT
That's where the "Systems Engineer" comes into play. (S)He's the person that is supposed to flow the requirements to each department and make sure all requirements are tested.
Sign in to Reply
CodeRed
8/27/2010 3:21 PM EDT
Err.... this is why requirements documents exist. It would have been more useful to point out this omission while the project was still at a planning stage rather than during the release tail. That's not the time to add new features. If you find a team that has dependencies on your deliverables or they have dependencies on yours - do everyone a huge favor and REVIEW THEIR PROJECT PLANS.
Sign in to Reply
CodeRed
8/27/2010 3:22 PM EDT
I mean, "or you on theirs".
Sign in to Reply
WKetel
8/27/2010 6:13 PM EDT
This shows the fallacy of assuming that systems do not interact. Of course they do, and not just electronics and code. For a project to be right the first time, without several tons of luck, it is mandatory that all the different areas be aware of what the others are doing. By "aware", I mean having an adequate understanding. This does take extra effort, and it does require a few with a lot more understanding, but it winds up being cheaper, when things all work the first time. How much does a simple chip re-spin cost? Or a new board layout? and what about a case molding dies rework?
Sign in to Reply
jimcondon
8/28/2010 10:24 PM EDT
I find it a little funny the hardware engineer calling out that the software engineer may not read a page in the functional specification, but never calls himself out for not reviewing the software requirements specification. By doing this, he could have discovered the oversight early and pushed for his feature to be covered.
I also find the us vs. them tone of the whole article disconcerting. It's not hardware vs. software, it's a team putting together a product. By watching each others backs in a team effort, issues will be found earlier and fixed faster.
Sign in to Reply
Skyhigh
8/29/2010 7:05 AM EDT
Strange this still happens today.
It depends how the engineering team is composed and managed.
Usually there is an embedded team engineer/manager who can manage software and hardware, or bridges both domains.
A department without an embedded expert usually runs into problems.
In fact, more companies today start to cut on pure software guys, and shift more effort on embedded (cos these are the best guys who can do RF, analog, digital, MCU and practically like a multi-purpose commandos)
Sign in to Reply
Robotics Developer
8/30/2010 1:42 PM EDT
I have witnessed the hardware / software divide my self over many years as a EE. I can't help but think part of the separation arises from there being both a hardware development group and a software development group as separate structures in most companies. This just reinforces the lines of separation. Perhaps, it would be best to develop products with an integrated team from the beginning as this would reinforce the team approach. I am also wondering why there was a “loss of optical signal” hardware alarm bit designed in if it was not in the requirements document for the product?
Sign in to Reply
lifewingmate
8/31/2010 2:41 AM EDT
As daunting as this scenario may seem, I believe it is all too common in the hardware and IC design sphere. I am interested to hear what method was used during development? Instead of head down mode, I want to add another tip...which is to follow Agile development methods and for a program manager and project manager to require daily scrum meetings so that hardware and software work together to make decisions about what and how requirements can be met within the project scope. In addition, real-time communication should happen when both hardware and software teams analyze the requirements document. The marketing team should also be alerted about the outcomes of these team analyses to ensure clear communication on all fronts. Prototyping and modeling should also be evaluated. There should be multiple safeguards to this process (not Waterfall) to ensure that the team's excellent efforts are not wasted or held back because of either poor decisions or miscommunication.
Sign in to Reply
Bob Lacovara
9/1/2010 10:23 AM EDT
Even if you have decent requirements, there are other pitfalls. I worked for MITRE once upon a time, and was hired by NASA to review a contract for a very large software configuration management system. The aeorspace contractor wanted to build it themselves for $50K. We thought this was just the usual loss leader... I found out that one of the commercial CM systems had 250K lines of code: no one was going to duplicate that for $50K. Needless to say, with that little bit of research my popularity was rising on one side of the fence, falling on the other. Next, the contractor had a meeting with NASA with me present to explain just how wonderful their extras were going to be. After about the third time the contractor explained "we don't have that in the requirements" in response to a NASA question, the woman in charge for NASA leaned over to the contractor's requirements manual. She flipped about 4 or 5 pages, and announced: you only have the odd pages of the requirements document. It was hilarious... although I managed to keep a straight face. Make sure that you have decent requirements, and while you are copying them, do it both sides...
Sign in to Reply
rpcy
9/1/2010 2:03 PM EDT
There's another lesson here. If your design has a means for detecting errors, then your design must also provide a means for generating those errors, so that you can tell if your detection scheme is working. It's amazing how few designers get this right, until after they've shipped a system that did not. (I'm not one of the few; I had to learn this the hard way.)
Sign in to Reply
WKetel
11/24/2010 9:33 PM EST
The reason for the divide between software and hardware is that the two are so very different. Those who are able to write good code based on a set of complete requirements have a quite different skill set from a design engineer who knows how much power each section of the circuit must deliver, and how much it will consume, and what signals must be protected from other signals. IN short, there are two completely different skill sets, and the few folks who are in possession of both are quite valuable, and therefore probably quite expensive, if they are any good. The higher price assures that many managers will seek to avoid having many with both skill sets around.
Sign in to Reply
Robert.Reavis
11/26/2010 1:49 PM EST
My career centered around automatic test equipment for twenty years, one foot in hardware and one foot in software. I once asked a question in a meeting and heard a yes and no at the same time. I told them that it did not matter what the answer was, but that both of them had to say the same thing. He said, she said, and then I said.
Regarding an unplugged device not giving any errors, even worse is when unplugging something causes a crash of the whold system and you are unable to report an unplugged device condition.
Sign in to Reply
Sanjib.Acharya
12/12/2010 7:12 AM EST
Another lesson learned (or addendum to #2 in the document) could be a well set-up communication process between the hardware and the software team. The communication style mostly depends on the organizational structure and in this case since the hardware & software teams do not share the same reporting authority, setting-up an inter-team communication process is essential. A hardware-software integration requirement document or something like that generated by the software team and then reviewed and approved by hardware team would have been helpful in catching the miss-outs.
Sign in to Reply
Bob Lacovara
12/13/2010 9:48 AM EST
Agreed, Mr.Acharya, the com style depends on the organization. The bridge document between hardware and software for the Command and Control Processor on the "International" Space Station was written in PowerPoint. In a series of about 20 such documents, the chief engineer described how he expected the hardware and software to interact, subsystem by subsystem. We followed the document, and when questions arose, we could just ask for clarification. It worked. And it didn't involve mediation by the system engineering clergy, or $60K/seat "requirements" CAD tools.
Sign in to Reply