Engineering Investigations
Comment
David Ashton
ylshih...I think that AC "Gimli Glider" was on that "Air Crash Investigations" ...
ylshih
We're sorry, the host system isn't responding
David Ashton
7/29/2010 7:42 PM EDT
An engineer battles network problems while a grumpy new bride is getting impatient
My story is set in the late 1980s when I was network manager for an airline telecomms multinational in Harare in Zimbabwe. My job involved ensuring that all our customers (airline offices) had their terminal systems connected to their mainframes on the other side of the world, via our network.
In those days the now-universal TCP-IP protocols were still in their infancy, and we used a number of proprietary airline protocols, notably IPARS, a (now) weird 6-bits per character polling protocol designed to make the best use of the 2400 BPS lines that were commonly used in those days. Anyway – to my story:
Qantas (the Australian airline) came to me one day and asked me to provide a working terminal for them at a travel show that was being held at a conference center on the outskirts of town. Not a big ask, except that this show was in two weeks time and getting lines put in within that timeframe in Zimbabwe at that time was well nigh impossible. But being a good service-oriented manager, I said I would do my best, not to expect too much, and could I have a box of giveaways to ease the passage of our line application through the ponderous bureaucracy of the Zimbabwe PTT.
They responded with a box full of Qantas pens, caps, bottles of wine, etc and I got to work, calling in favours, turning on the charm and dispensing giveaways to anyone who might be able to help get all the ducks in a row for us. I borrowed an EEPROM programmer and reprogrammed a spare terminal’s EEPROM to make it talk IPARS. I sweet-talked our configurations people at head office to get the network configuration done in half the usual time.
There was one rather large fly in the ointment: Just before the show started, I was getting married at a resort about 300 Km away from Harare. So with some trepidation, I exhorted my staff to follow up everything that needed following up, and went and got married. Fast forward a couple of days. I returned to Harare in a state of newly wedded bliss, with one spare day before we left on honeymoon, and said to my wife that morning (the day before the show kicked off): “Dearest, I’m just popping into work to check if my Qantas terminal is up and running. Okay?”
So at 9 am that day I was back in at work expecting maybe the odd minor problem. And my staff had done a good job. The line was in, the modems were connected, the terminal was being polled and all looked good. Except that the terminal was not getting any responses from the Qantas host. The travel show was in a large conference hall and there was only one working phone there, which happened to be on the other side of the hall from the Qantas stand. Cellphones were not available in Zimbabwe in those days.
So to get them to make test entries on their terminal, I had to phone that number, ask whoever answered to get someone from the Qantas stand to phone me back , ask them to make entries for a few minutes, and then monitor their line with our protocol monitor. Yes, we could see their entries, but no responses. So we then had to get the next upline communication center,this case Johannesburg, to monitor, and repeat the whole procedure to get entries made on the terminal. Again they saw entries but no host responses.
My boss was understandably not keen on maxing out his international telephone budget, so all this had to be done by messaging rather than just phoning, which made it rather difficult to get the monitoring done at the same time as the entries were being made. By this time it was getting near lunchtime. I phoned my new wife and said I was having problems, would she mind if I missed lunch? More groans. And so we managed over the next few hours to monitor the traffic from our little terminal all the way through our network through Johannesburg, London and Sydney and finally got confirmation from our Sydney center that they could see entries going to the Qantas host system, but no replies.
Our network operator in Sydney sent me a message, saying that he had sent a message to the Qantas network center but there was as yet no one there who could check further (there was a 10-hour time difference) and that he was going off-shift but had let the new operator know about the problem. He also helpfully gave me a phone number for the Qantas data center. By now it was nearly 5 PM and in full desperation mode I asked my boss if I could phone Qantas in Sydney. “Go for it,” he replied.
So I phoned Qantas in Sydney and got a friendly Aussie lady on the line. I explained the problem and asked if she had any ideas. “Well let me check my configuration”, she said. We had about 5 address parameters for the communications centers where the traffic entered and exited our network, and our configuration message used a horrible mishmash of Decimal, Hexadecimal and Octal values. We went through the configuration and confirmed all the values until we got to the Terminal Cluster Identification (TCID) value.“That’s 43 Hex,” she said. “No,” I replied, "That’s 043 OCTAL." “So it is.” she said. “Sorry for that. Hang on a sec.”
I heard tapping on a keyboard on the other side of the world. “OK”, she said. “Try that now.”
So I had to go through the whole phoning rigmarole again. Finally the Qantas rep at the travel show got back to me. “It’s working!” he said. “I can sign in!” I said. “I’m out of here!” 20 years later, I’m still married, but I sure know the difference between Octal and Hex!
David Ashton is an Australian engineer who says this experience gave him the satisfaction of a job done on time, a happy customer AND a happy boss--at the cost of a slightly grumpy wife.
My story is set in the late 1980s when I was network manager for an airline telecomms multinational in Harare in Zimbabwe. My job involved ensuring that all our customers (airline offices) had their terminal systems connected to their mainframes on the other side of the world, via our network.
In those days the now-universal TCP-IP protocols were still in their infancy, and we used a number of proprietary airline protocols, notably IPARS, a (now) weird 6-bits per character polling protocol designed to make the best use of the 2400 BPS lines that were commonly used in those days. Anyway – to my story:
Qantas (the Australian airline) came to me one day and asked me to provide a working terminal for them at a travel show that was being held at a conference center on the outskirts of town. Not a big ask, except that this show was in two weeks time and getting lines put in within that timeframe in Zimbabwe at that time was well nigh impossible. But being a good service-oriented manager, I said I would do my best, not to expect too much, and could I have a box of giveaways to ease the passage of our line application through the ponderous bureaucracy of the Zimbabwe PTT.
They responded with a box full of Qantas pens, caps, bottles of wine, etc and I got to work, calling in favours, turning on the charm and dispensing giveaways to anyone who might be able to help get all the ducks in a row for us. I borrowed an EEPROM programmer and reprogrammed a spare terminal’s EEPROM to make it talk IPARS. I sweet-talked our configurations people at head office to get the network configuration done in half the usual time.
There was one rather large fly in the ointment: Just before the show started, I was getting married at a resort about 300 Km away from Harare. So with some trepidation, I exhorted my staff to follow up everything that needed following up, and went and got married. Fast forward a couple of days. I returned to Harare in a state of newly wedded bliss, with one spare day before we left on honeymoon, and said to my wife that morning (the day before the show kicked off): “Dearest, I’m just popping into work to check if my Qantas terminal is up and running. Okay?”
So at 9 am that day I was back in at work expecting maybe the odd minor problem. And my staff had done a good job. The line was in, the modems were connected, the terminal was being polled and all looked good. Except that the terminal was not getting any responses from the Qantas host. The travel show was in a large conference hall and there was only one working phone there, which happened to be on the other side of the hall from the Qantas stand. Cellphones were not available in Zimbabwe in those days.
So to get them to make test entries on their terminal, I had to phone that number, ask whoever answered to get someone from the Qantas stand to phone me back , ask them to make entries for a few minutes, and then monitor their line with our protocol monitor. Yes, we could see their entries, but no responses. So we then had to get the next upline communication center,this case Johannesburg, to monitor, and repeat the whole procedure to get entries made on the terminal. Again they saw entries but no host responses.
My boss was understandably not keen on maxing out his international telephone budget, so all this had to be done by messaging rather than just phoning, which made it rather difficult to get the monitoring done at the same time as the entries were being made. By this time it was getting near lunchtime. I phoned my new wife and said I was having problems, would she mind if I missed lunch? More groans. And so we managed over the next few hours to monitor the traffic from our little terminal all the way through our network through Johannesburg, London and Sydney and finally got confirmation from our Sydney center that they could see entries going to the Qantas host system, but no replies.
Our network operator in Sydney sent me a message, saying that he had sent a message to the Qantas network center but there was as yet no one there who could check further (there was a 10-hour time difference) and that he was going off-shift but had let the new operator know about the problem. He also helpfully gave me a phone number for the Qantas data center. By now it was nearly 5 PM and in full desperation mode I asked my boss if I could phone Qantas in Sydney. “Go for it,” he replied.
So I phoned Qantas in Sydney and got a friendly Aussie lady on the line. I explained the problem and asked if she had any ideas. “Well let me check my configuration”, she said. We had about 5 address parameters for the communications centers where the traffic entered and exited our network, and our configuration message used a horrible mishmash of Decimal, Hexadecimal and Octal values. We went through the configuration and confirmed all the values until we got to the Terminal Cluster Identification (TCID) value.“That’s 43 Hex,” she said. “No,” I replied, "That’s 043 OCTAL." “So it is.” she said. “Sorry for that. Hang on a sec.”
I heard tapping on a keyboard on the other side of the world. “OK”, she said. “Try that now.”
So I had to go through the whole phoning rigmarole again. Finally the Qantas rep at the travel show got back to me. “It’s working!” he said. “I can sign in!” I said. “I’m out of here!” 20 years later, I’m still married, but I sure know the difference between Octal and Hex!
David Ashton is an Australian engineer who says this experience gave him the satisfaction of a job done on time, a happy customer AND a happy boss--at the cost of a slightly grumpy wife.
Navigate to related information





Rich Krajewski
7/30/2010 5:30 PM EDT
Sounds like miles versus kilometers at NASA. Except your version had a happy ending. Sort of.
Sign in to Reply
DrQuine
7/31/2010 8:10 PM EDT
It sounds like you really went the extra mile to ensure customer satisfaction! It reminds me of the advertisement that I once saw for a mail sorting machine that moved envelopes at 300,000 km / second. Pretty impressive that they were moving "snail mail" at the speed of light. If the truth were to be told, I believe that they really moved the mail at 300 cm / second. Getting units right can be so pesky.
Sign in to Reply
MikeLC
7/31/2010 9:11 PM EDT
Great story! Next time I design a hex codes for a protocol I'll try to have at least one of those digits greater than 7! :-)
Congrats to both of you on your 20 years!
Sign in to Reply
t.alex
7/31/2010 11:43 PM EDT
This happens to me sometimes. Decimal and hexadecimal can be easily mistaken if there is no clear indication on the GUI.
Sign in to Reply
ylshih
8/1/2010 12:25 AM EDT
Great story and an understanding wife!
Bases/units have caused many problems. The Air Canada "Gimli Glider" (767 airplane that had to glide into a rural airport in Canada) incident arose in part, after a chain of other events, due to a mixup between gallons and liters during the switchover to metric.
Sign in to Reply
David Ashton
8/15/2010 5:57 AM EDT
ylshih...I think that AC "Gimli Glider" was on that "Air Crash Investigations" programme...have you ever noticed that it's almost always a combination of 2 things going wrong?
Most of you would have seen that T-shirt / Bumper sticker:
There are 10 kinds of people in the world....those who speak binary and those who don't!
Thanks all for the nice comments.
Sign in to Reply