Design Con 2015
Breaking News
News & Analysis

Did Upside-Down Sensors Cause Rocket Crash?

7/10/2013 01:05 PM EDT
36 comments
NO RATINGS
More Related Links
View Comments: Threaded | Newest First | Oldest First
DrQuine
User Rank
CEO
Preventing Upside-Down Sensors
DrQuine   7/10/2013 4:43:31 PM
NO RATINGS
The use of keyed connectors that can only be assembled one way is a good precaution to prevent reversed connections.  Of course, if the connector is wired backwards that won't help.  The second essential step is to gather baseline data and confirm that the device is operating properly. If indeed the sensor was installed upside-down, it should have been reported inverted data when standing on the launchpad. Exhaustive testing certainly proves its worth if it can catch a tiny error before it causes a huge failure.

CMathas
User Rank
Manager
Re: Preventing Upside-Down Sensors
CMathas   7/10/2013 6:11:11 PM
NO RATINGS
Huge failure is right. One would think that there would be several tests and confirmations in place before actual launch. That's a huge loss of resources based on something that should have been easily caught.

AlNav
User Rank
Rookie
Re: Preventing Upside-Down Sensors
AlNav   7/11/2013 9:48:49 AM
NO RATINGS
@DrQuine: I agree that a simple thing like keying the package should help ensure proper mounting.  In this case, the source stated the "angular velocity sensors" may have been the wrong way around.  The only way to test those after they're installed may be to rotate the rocket, as they only respond to dynamic rotational motion - sitting on the launch pad, they will report zeros.  (although the Proton system seems to include rotating the rocket in place, I'm not sure of the feasibility of having the navigation systems in a test mode during that operation).

I'm sure there will be an analysis to prevent this from happening in the future.

_hm
User Rank
CEO
Quality Control and testing in Space Electronics
_hm   7/10/2013 8:37:23 PM
NO RATINGS
Quality control in Space electronics is very essential. And this type of mistakes are often discovered. One common exapmple is polarized capacitor. Many times, they are mounted with polarity reversed.

Another aspect is testing. Testing is very improtant in this type of critical missions. It should try to encompass all this possibilities. And yes, poke yoke problem must also be avoided.

LarryM99
User Rank
CEO
Re: Quality Control and testing in Space Electronics
LarryM99   7/10/2013 11:55:08 PM
NO RATINGS
We can't necessarily be too smug about this. I seem to remember an American Mars probe that sailed right past that planet because someone put a period instead of a comma in a line of FORTRAN code. I have had friends that worked on the Saturn 5 that left because of the tension of building systems that could never be completely tested prior to a launch. We may be better off today with a combination of more experience and better test equipment, but we are also building more complex systems.

Duane Benson
User Rank
Blogger
Re: Quality Control and testing in Space Electronics
Duane Benson   7/11/2013 11:34:12 AM
NO RATINGS
Larry - re: "We can't necessarily be too smug about this. I seem to remember an American Mars probe that sailed right past that planet..." There was also the Mars probe that crashed because of a Imperial vs. Metric mix up. And there was also the Genesis solar wind sample & return mission that crashed because the G sensor used to trigger parachute deployment was installed backwards.

Etmax
User Rank
Rookie
Re: Quality Control and testing in Space Electronics
Etmax   7/13/2013 5:59:43 AM
NO RATINGS
There was also an Ariane rocket that died due to Metric/Imperial issues. I can't understand why anybody is using imperial any more in science. The whole world's science community is supposed to be metric now. Australia went metric in 1970 (enacted into law) and made it illegal to use metric offically. All packaging had to have metric measure and good were sold by metric measure. Even building products which were often imperial sized had their metric equivalent printed on them. The end result was it all happened and very few look back on it as a negative thing. The Russians are of course metric so that one is ruled out but the argument for testing is a difficult one in an assembled rocket. It probably wasn't electrical orientation but rather mechanical, but how do you rotate a fully assembled rocket fast enough to test gyros? I guess the more complex these things become the more difficult an ideal testing regime becomes. BTW, I replied to your post only to mention the metric issue, sorry to add so much more :-)

kfield
User Rank
Blogger
Re: Quality Control and testing in Space Electronics
kfield   7/27/2013 6:05:17 PM
NO RATINGS
@etmax Good question about English versus metric units. I remember back in grade school the U.S. preparing to "go metric," and that was more years ago than I care to admit. I think the cost of conversion would be so great at this point, that it's simply easier to not deal with it. Not that it's the right thing to do.

Etmax
User Rank
Rookie
Re: Quality Control and testing in Space Electronics
Etmax   7/28/2013 11:47:55 PM
NO RATINGS
I'm a take the bull by the horns sort of guy, so I would suggest that the cost of changing will get ever larger while the cost of doing nothing will grow larger still, sort of like global warming (if you subscribe to the idea). I think the cost of change as a percentage relative to GDP is probably near a contant, where as the cost of not changing will just continue to go up. When I started in design all part footprints were imperial, now virtually all new footprints are metric, even new ones from US design houses. Even though the US is the worlds largest economy, if you do a "size of imperial economy" vs. "size of metric economy" and factor in how much of the US output goes into export goods to metric economies, it soon becomes clear that there must be a significant financial burden in running dual systems. Add to that I was watching an air crash investigation episode where a airport attendant was refueling a metric plane and did his fuel calculations as if the tank level reading he was getting was gallons and as a result the plane was forced to land without fuel at a desert airport. If it would have been an ocean landing all on board would have died. These sort of things can only happen in a dual measurement world and are extremely costly my guess is $350million per plane and maybe $10 million per passenger (how much is a life really worth). The ESA crashed a Mars lander because the landing computer was metric and someone did the calculations in imperial, that was $3billion. I don't mean to harp on this issue, but there are just so many examples of this, some we just don't hear about. What would the additional cost of insurance be if this was deemed an additional cost of doing business with the US?

kfield
User Rank
Blogger
Re: Quality Control and testing in Space Electronics
kfield   7/29/2013 11:44:00 AM
NO RATINGS
@ETmax I don't disgree with you that it's costing a bundle of money to run a dual system, but unless those costs appear as a line item on somebody's balance sheet, there isn't much impetus for government or industry or somebody to pony up and do something about it. Since nobody can measure these hidden costs, how can the ROI of a massive, one-time investment be calculated? Sorry, but it's just the way our financial systems are wired--do nothing until our back is up against the wall, and even then, only maybe do something.

Tom Murphy
User Rank
Blogger
Re: Quality Control and testing in Space Electronics
Tom Murphy   7/29/2013 12:58:13 PM
NO RATINGS
ETmax, Karen:  Do you really think that Americans (half of whom apparently think public healthcare is a socialistic concept) would be willing to switch to metrics -- throwing out their beloved tape measures and rulers in the process and recasting every property measurement in the country?   Consider American football alone: can you imagine the Super Bowl played on a 91.44-meter field? 

Cost aside, I think it is a noble but unobtainable goal. (Besides, what are you going to do about that pesky Imperial Gallon in Canada -- 20% larger than the US gallon)   There is need for  change in America. I'm not sure this one tops my list.  What do others think? 

CMathas
User Rank
Manager
Re: Quality Control and testing in Space Electronics
CMathas   7/11/2013 8:13:56 AM
NO RATINGS
Interesting comment--this type of thing happens often? It's a bit amazing that the systems that we look at to be in the forefront of innovation can be brought down by such errors. I really appreciate all of these comments. At first I thought this couldn't be true, but the more I read from those of you in the know, it seems pretty possible.

 

Tom Murphy
User Rank
Blogger
Dubious Source
Tom Murphy   7/10/2013 9:45:20 PM
NO RATINGS
I'm not sure I'd put a lot of stock in an unidentified source and the Interfax news agency. I mean -- these are rocket scientists we're talking about, and they know how to install sensors.  In any case, how would they know at this point?

Sanjib.A
User Rank
CEO
Re: Dubious Source
Sanjib.A   7/11/2013 1:18:21 AM
NO RATINGS
@Tom: I agree with you. I am a bit skeptic about the source, which has pointed to an important sensor getting installed "Up-side Down" as the cause of the failure. Why I think so?...several reasons. One is already mentioned by Tom. Also:

1. There should have been well-established processes for design, development and manufacturing space equipments as this kind of space programs deal with huge money, reputation and safety. The process should have been rugged enough to prevent this kind of human error. If not prevented, atleast, it should have been caught in inspection or some kind of quality checking/reviews.

2. Even if it was a mistake, I think design should have had redundant sensors to take care of this kind errors. Hope not all of those were mounted "upside-down"...otherwise there must had been a serious process gap somewhere.

I don't think this kind of error getting un-detected is something which could be expected from a well settled space organization isn't it?

Tom Murphy
User Rank
Blogger
Re: Dubious Source
Tom Murphy   7/11/2013 1:28:08 AM
NO RATINGS
Good point, Sanjib.  You would think that there would be redundant systems. In the US, it's standard practice to have triple redundancy on space probes. And the monitoring stations on the ground watch every indicator.  There is certainly a possibility that this report is true, but I think there is a larger chance that the report is based on poor information.

Kevin Neilson
User Rank
Manager
Re: Dubious Source
Kevin Neilson   7/11/2013 3:34:46 AM
NO RATINGS
Genesis had redundant accelerometers.  They were all mounted backwards.

Sanjib.A
User Rank
CEO
Re: Dubious Source
Sanjib.A   7/11/2013 9:28:23 AM
NO RATINGS

@Kevin N: Okay if all redundant sensors were mounted backwards, this could be due to manual error that instructions for installation were not followed appropriately or instruction for installation was dubious; Also then, there was a process gap that the installation was not reviewed by somebody independently.

By the way...will there be any official report released some point of time?

CMathas
User Rank
Manager
Re: Dubious Source
CMathas   7/11/2013 4:29:00 PM
NO RATINGS
All mounted backwards? Well consistency is good I guess, but the comment made me laugh! Can you imagine explaining this sort of thing to the bosses?

Kevin Neilson
User Rank
Manager
Genesis
Kevin Neilson   7/11/2013 12:45:56 AM
NO RATINGS
Wait a second--I hope this isn't true, because this is exactly what happened with the Genesis probe.  It returned to earth and the drogue was supposed to come out after it had decelerated to a certain speed, but the accelerometers were mounted upside down so the deceleration was interpreted as acceleration and the drogue never came out and the probe cratered in the desert.

CMathas
User Rank
Manager
Re: Genesis
CMathas   7/11/2013 8:10:59 AM
NO RATINGS
So if it could happen with Genesis, it surely could have happened here.

As to information in this by a "source", I think it will be very difficult to ever put a name with a comment when something like this occurs.

prabhakar_deosthali
User Rank
CEO
Re:
prabhakar_deosthali   7/11/2013 7:23:32 AM
NO RATINGS
It is very difficult to believe that such a mistake could have been made !

It will be interesting to know the design of the sensors- where there was an indication of how to mount them . If the basic instructions has been overlooked then it shows a poor standard of inspection and testing of a billion dollar product

 

mcgrathdylan
User Rank
Blogger
Shocking
mcgrathdylan   7/11/2013 10:26:06 AM
NO RATINGS
Not sure about you guys, but I for one am pretty shocked that a Russian government organization is being less than forthcoming with information about something that makes them look bad.

Tom Murphy
User Rank
Blogger
Re: Shocking
Tom Murphy   7/12/2013 12:34:24 PM
NO RATINGS
Dylan: I'm shocked that you're shocked about Russia's reluctance to be forthcoming after the recent disclosures about the NSA's spying campaign by the US.  Forthcoming, indeed!

Warren3
User Rank
CEO
A Program in Distress
Warren3   7/11/2013 1:57:14 PM
NO RATINGS
Interesting FACTS: This crash is the fifth major LAUNCH failure of a Proton-M rocket since 2010. The program was grounded by the Russians just last December and restarted only as recently as March, since which they had successfully launched THREE Proton-M rockets before this launch failure. That's roughly one launch per month; seems like a program in a hurry. [Source= http://www.scientificamerican.com/article.cfm?id=russian-rocket-crash-deta] It's clearly a difficult task to push 20 tons of payload into orbit... and as of late it's proving especially difficult for the RFSA.

mcgrathdylan
User Rank
Blogger
Re: A Program in Distress
mcgrathdylan   7/11/2013 2:16:19 PM
NO RATINGS
Warren3- useful facts to add to the discussion. Thank you. It does seem as though they are perhaps a little too aggressive on the scheduling.

CMathas
User Rank
Manager
Re: A Program in Distress
CMathas   7/11/2013 4:25:41 PM
NO RATINGS
Thanks for your post. That would be aggressie under the best of circumstances. I wonder how much the whole bill has been for the wasted efforts.

Warren3
User Rank
CEO
Re: A Program in Distress
Warren3   7/11/2013 6:47:13 PM
NO RATINGS
Payload, as I recall, was estimated at over $200m (while the insurance on the payload is reported to be $182m)... all in US dollars.

CMathas
User Rank
Manager
Surprise
CMathas   7/11/2013 4:26:54 PM
NO RATINGS
I agree - it shocked me too. Especially would have figured that the sources would have wanted the limelight. Go figure.

microjunkie
User Rank
Rookie
Murphy's Law
microjunkie   7/12/2013 12:25:33 AM
NO RATINGS
This is Murphy's Law at it's finest. If there's a chance that something can be done wrong, then someone will do it.

CMathas
User Rank
Manager
Re: Murphy's Law
CMathas   7/12/2013 9:37:57 AM
NO RATINGS
From one who has a dog named Murphy--this is one expensive lesson. I truly am amazed that it doesn't seem to be a completely isolated problem. It makes you wonder how much better industry does...

Tom Murphy
User Rank
Blogger
Re: Microjunkie's Law
Tom Murphy   7/12/2013 12:36:10 PM
NO RATINGS
Microjunkie: As someone who happens to be graced with the name Murphy, I'd ask you please to avoid the cliche of "Murphy's Law" in the future.  Thank you!

microjunkie
User Rank
Rookie
Re: Microjunkie's Law
microjunkie   7/13/2013 12:15:45 AM
NO RATINGS
Actually it is not a cliché but an adage or epigram according to Wikipedia. Here is an interresting link. http://www.murphyslaws.net I have been a techie for 50 years and when designing or building things, always keep Murphy's Law in mind. No insult intended.

Patk0317
User Rank
CEO
Russian Rocket Crash
Patk0317   7/14/2013 8:21:13 PM
NO RATINGS
Were the sensors mounted backwards or connected backwards? Seems to me for something like this the connectors would be keyed so that they could not be put together incorrectly. Anyhow, it would explain why the rocekt made a beeline for the goournd when is should've been going in the other direction.

seaEE
User Rank
CEO
Redundancy is best
seaEE   7/14/2013 8:57:59 PM
NO RATINGS
Word has it that future rockets will be launched in redundant fashion, on opposite sides of the world, to ensure that one always makes it to its destination.

docdivakar
User Rank
CEO
Re: Did Upside-Down Sensors Cause Rocket Crash?
docdivakar   7/16/2013 9:21:49 PM
NO RATINGS
Well, suffice it to say that we should wait for a full investigation and the report from it. If the sensors were wired wrong, how in the world did the test of 'system' pass?

MP Divakar

elctrnx_lyf
User Rank
Manager
Re: Did Upside-Down Sensors Cause Rocket Crash?
elctrnx_lyf   7/17/2013 7:07:42 AM
NO RATINGS
It could much more serious problem than just placing the sensors wrongly. I hope the investigative authorities will be able to find the right root cause.

Flash Poll
Like Us on Facebook

Datasheets.com Parts Search

185 million searchable parts
(please enter a part number or hit search to begin)
EE Times on Twitter
EE Times Twitter Feed
Top Comments of the Week