Sensors are highly reliable, especially the high-performance variety used in aviation/aerospace apps. However, human error sometimes interferes. It seems that failure to install angular velocity sensors correctly in the Russian Proton-M rocket resulted in a crash on July 2.
What happened? A source involved with the commission investigating the crash told the Interfax news service that angular velocity sensors were "connected the wrong way round" and delivered incorrect data to the guidance system.
The Russian space agency Roscosmos has not confirmed the cause. A source with links to the commission told Interfax that it is investigating a number of possibilities, including a problem with the navigation system, engine malfunction, and an error in the equipment at launch control.
A Proton-M booster rocket with a DM-03 upper stage and three Russian GLONASS-M satellites sits on the Baikonur Cosmodrome launch pad. (Source: RIA Novosti/Roskosmos press service)
Whether the cause was inverted sensors or something else, three navigation satellites and $1.3 billion of equipment was destroyed in the crash, according to Interfax. Just 17 seconds after takeoff, the rocket rapidly veered off course and tried to correct, but it plummeted towards Earth and exploded on contact only 2.5km from the launch site.
The Voice of Russia radio service said the rocket carried 172 tons of highly toxic heptyl fuel and oxidizer. Interfax said 100,000 residents were evacuated, and efforts are under way to detoxify the soil.
The use of keyed connectors that can only be assembled one way is a good precaution to prevent reversed connections. Of course, if the connector is wired backwards that won't help. The second essential step is to gather baseline data and confirm that the device is operating properly. If indeed the sensor was installed upside-down, it should have been reported inverted data when standing on the launchpad. Exhaustive testing certainly proves its worth if it can catch a tiny error before it causes a huge failure.
Huge failure is right. One would think that there would be several tests and confirmations in place before actual launch. That's a huge loss of resources based on something that should have been easily caught.
@DrQuine: I agree that a simple thing like keying the package should help ensure proper mounting. In this case, the source stated the "angular velocity sensors" may have been the wrong way around. The only way to test those after they're installed may be to rotate the rocket, as they only respond to dynamic rotational motion - sitting on the launch pad, they will report zeros. (although the Proton system seems to include rotating the rocket in place, I'm not sure of the feasibility of having the navigation systems in a test mode during that operation).
I'm sure there will be an analysis to prevent this from happening in the future.
We can't necessarily be too smug about this. I seem to remember an American Mars probe that sailed right past that planet because someone put a period instead of a comma in a line of FORTRAN code. I have had friends that worked on the Saturn 5 that left because of the tension of building systems that could never be completely tested prior to a launch. We may be better off today with a combination of more experience and better test equipment, but we are also building more complex systems.
Larry - re: "We can't necessarily be too smug about this. I seem to remember an American Mars probe that sailed right past that planet..." There was also the Mars probe that crashed because of a Imperial vs. Metric mix up. And there was also the Genesis solar wind sample & return mission that crashed because the G sensor used to trigger parachute deployment was installed backwards.
There was also an Ariane rocket that died due to Metric/Imperial issues. I can't understand why anybody is using imperial any more in science. The whole world's science community is supposed to be metric now. Australia went metric in 1970 (enacted into law) and made it illegal to use metric offically. All packaging had to have metric measure and good were sold by metric measure. Even building products which were often imperial sized had their metric equivalent printed on them. The end result was it all happened and very few look back on it as a negative thing. The Russians are of course metric so that one is ruled out but the argument for testing is a difficult one in an assembled rocket. It probably wasn't electrical orientation but rather mechanical, but how do you rotate a fully assembled rocket fast enough to test gyros? I guess the more complex these things become the more difficult an ideal testing regime becomes. BTW, I replied to your post only to mention the metric issue, sorry to add so much more :-)
@etmax Good question about English versus metric units. I remember back in grade school the U.S. preparing to "go metric," and that was more years ago than I care to admit. I think the cost of conversion would be so great at this point, that it's simply easier to not deal with it. Not that it's the right thing to do.
I'm a take the bull by the horns sort of guy, so I would suggest that the cost of changing will get ever larger while the cost of doing nothing will grow larger still, sort of like global warming (if you subscribe to the idea). I think the cost of change as a percentage relative to GDP is probably near a contant, where as the cost of not changing will just continue to go up. When I started in design all part footprints were imperial, now virtually all new footprints are metric, even new ones from US design houses. Even though the US is the worlds largest economy, if you do a "size of imperial economy" vs. "size of metric economy" and factor in how much of the US output goes into export goods to metric economies, it soon becomes clear that there must be a significant financial burden in running dual systems. Add to that I was watching an air crash investigation episode where a airport attendant was refueling a metric plane and did his fuel calculations as if the tank level reading he was getting was gallons and as a result the plane was forced to land without fuel at a desert airport. If it would have been an ocean landing all on board would have died. These sort of things can only happen in a dual measurement world and are extremely costly my guess is $350million per plane and maybe $10 million per passenger (how much is a life really worth). The ESA crashed a Mars lander because the landing computer was metric and someone did the calculations in imperial, that was $3billion. I don't mean to harp on this issue, but there are just so many examples of this, some we just don't hear about. What would the additional cost of insurance be if this was deemed an additional cost of doing business with the US?
@ETmax I don't disgree with you that it's costing a bundle of money to run a dual system, but unless those costs appear as a line item on somebody's balance sheet, there isn't much impetus for government or industry or somebody to pony up and do something about it. Since nobody can measure these hidden costs, how can the ROI of a massive, one-time investment be calculated? Sorry, but it's just the way our financial systems are wired--do nothing until our back is up against the wall, and even then, only maybe do something.
ETmax, Karen: Do you really think that Americans (half of whom apparently think public healthcare is a socialistic concept) would be willing to switch to metrics -- throwing out their beloved tape measures and rulers in the process and recasting every property measurement in the country? Consider American football alone: can you imagine the Super Bowl played on a 91.44-meter field?
Cost aside, I think it is a noble but unobtainable goal. (Besides, what are you going to do about that pesky Imperial Gallon in Canada -- 20% larger than the US gallon) There is need for change in America. I'm not sure this one tops my list. What do others think?
Interesting comment--this type of thing happens often? It's a bit amazing that the systems that we look at to be in the forefront of innovation can be brought down by such errors. I really appreciate all of these comments. At first I thought this couldn't be true, but the more I read from those of you in the know, it seems pretty possible.
I'm not sure I'd put a lot of stock in an unidentified source and the Interfax news agency. I mean -- these are rocket scientists we're talking about, and they know how to install sensors. In any case, how would they know at this point?
@Tom: I agree with you. I am a bit skeptic about the source, which has pointed to an important sensor getting installed "Up-side Down" as the cause of the failure. Why I think so?...several reasons. One is already mentioned by Tom. Also:
1. There should have been well-established processes for design, development and manufacturing space equipments as this kind of space programs deal with huge money, reputation and safety. The process should have been rugged enough to prevent this kind of human error. If not prevented, atleast, it should have been caught in inspection or some kind of quality checking/reviews.
2. Even if it was a mistake, I think design should have had redundant sensors to take care of this kind errors. Hope not all of those were mounted "upside-down"...otherwise there must had been a serious process gap somewhere.
I don't think this kind of error getting un-detected is something which could be expected from a well settled space organization isn't it?
Good point, Sanjib. You would think that there would be redundant systems. In the US, it's standard practice to have triple redundancy on space probes. And the monitoring stations on the ground watch every indicator. There is certainly a possibility that this report is true, but I think there is a larger chance that the report is based on poor information.
@Kevin N: Okay if all redundant sensors were mounted backwards, this could be due to manual error that instructions for installation were not followed appropriately or instruction for installation was dubious; Also then, there was a process gap that the installation was not reviewed by somebody independently.
By the way...will there be any official report released some point of time?
Wait a second--I hope this isn't true, because this is exactly what happened with the Genesis probe. It returned to earth and the drogue was supposed to come out after it had decelerated to a certain speed, but the accelerometers were mounted upside down so the deceleration was interpreted as acceleration and the drogue never came out and the probe cratered in the desert.
It is very difficult to believe that such a mistake could have been made !
It will be interesting to know the design of the sensors- where there was an indication of how to mount them . If the basic instructions has been overlooked then it shows a poor standard of inspection and testing of a billion dollar product
Interesting FACTS: This crash is the fifth major LAUNCH failure of a Proton-M rocket since 2010. The program was grounded by the Russians just last December and restarted only as recently as March, since which they had successfully launched THREE Proton-M rockets before this launch failure. That's roughly one launch per month; seems like a program in a hurry. [Source= http://www.scientificamerican.com/article.cfm?id=russian-rocket-crash-deta] It's clearly a difficult task to push 20 tons of payload into orbit... and as of late it's proving especially difficult for the RFSA.
Actually it is not a cliché but an adage or epigram according to Wikipedia.
Here is an interresting link.
I have been a techie for 50 years and when designing or building things, always keep Murphy's Law in mind. No insult intended.
Were the sensors mounted backwards or connected backwards? Seems to me for something like this the connectors would be keyed so that they could not be put together incorrectly. Anyhow, it would explain why the rocekt made a beeline for the goournd when is should've been going in the other direction.
David Patterson, known for his pioneering research that led to RAID, clusters and more, is part of a team at UC Berkeley that recently made its RISC-V processor architecture an open source hardware offering. We talk with Patterson and one of his colleagues behind the effort about the opportunities they see, what new kinds of designs they hope to enable and what it means for today’s commercial processor giants such as Intel, ARM and Imagination Technologies.