· 8 years ago · Feb 11, 2018, 08:14 AM
1
2
3https://en.wikipedia.org/wiki/Mars_Climate_Orbiter
4 The Mars Climate Orbiter (formerly the Mars Surveyor '98 Orbiter) was a 338-kilogram (745 lb) robotic space probe launched by NASA on December 11, 1998 to study the Martian climate, Martian atmosphere, and surface changes and to act as the communications relay in the Mars Surveyor '98 program for Mars Polar Lander. However, on September 23, 1999, communication with the spacecraft was lost as the spacecraft went into orbital insertion, due to ground-based computer software which produced output in non-SI units of pound (force)-seconds (lbf·s) instead of the SI units of newton-seconds (N·s) specified in the contract between NASA and Lockheed. The spacecraft encountered Mars on a trajectory that brought it too close to the planet, causing it to pass through the upper atmosphere and disintegrate.[1][2]
5
6
7------
8
9ftp://ftp.hq.nasa.gov/pub/pao/reports/1999/MCO_report.pdf
10 The MCO MIB has determined that the root cause for the loss of the MCO spacecraft was the failure to use metric units in the coding of a ground software file, “Small Forces,†used in trajectory models. Specifically, thruster performance data in English units instead of metric units was used in the software application code titled SM_FORCES (small forces). A file called Angular Momentum Desaturation (AMD) contained the output data from the SM_FORCES software. The data in the AMD file was required to be in metric units per existing software interface documentation, and the trajectory modelers assumed the data was provided in metric units per the requirements.
11
12
13
14They had a program called "SM_FORCES"
15
16I guess this ran on desktop?
17
18The software requirements and documentation said that this takes in a file "small forces," with metric units, and outputs a file "Angular Momentum Desaturation," with units also in metric, which would be used to guide the probe.
19
20But someone put the data "small forces" in English units instead of metric, so the data output by "small forces" and fed to their navigation software was junk.
21
22
23 During the 9-month journey from Earth to Mars, propulsion maneuvers were periodically performed to remove angular momentum buildup in the on-board reaction wheels (flywheels). These Angular Momentum Desaturation (AMD) events occurred 10-14 times more often than was expected by the operations navigation team. This was because the MCO solar array was asymmetrical relative to the spacecraft body as compared to Mars Global Surveyor (MGS) which had symmetrical solar arrays. This asymmetric effect significantly increased the Sun-induced (solar pressure-induced) momentum buildup on the spacecraft. The increased AMD events coupled with the fact that the angular momentum (impulse) data was in English, rather than metric, units, resulted in 7 small errors being introduced in the trajectory estimate over the course of the 9-month journey. At the time of Mars insertion, the spacecraft trajectory was approximately 170 kilometers lower than planned. As a result, MCO either was destroyed in the atmosphere or re-entered heliocentric space after leaving Mars’ atmosphere.
24
25
26this built-up small errors in its trajectory over time, and the probe ended up 170 KM off target.
27
28
29------
30
31ftp://ftp.hq.nasa.gov/pub/pao/reports/1999/MCO_report.pdf
32 ...The MSP ’98 Development Project used a prime contract vehicle to support project implementation. Lockheed Martin Astronautics (LMA) of Denver, Colorado was selected as the prime contractor. LMA’s contracted development responsibilities were to design and develop both spacecraft, lead flight system integration and test, and support launch operations. JPL retained responsibilities for overall project management, spacecraft and instrument development management, project system engineering, mission design, navigation design, mission operation system development, ground data system development, and mission assurance.
33
34
35JPL was managing the project and handling "navigation design" and "mission operation system development". Lockheed Martin was designing and developing the probe and the "flight system".
36
37goes on to say JPL does "mission management" and LMA does "spacecraft operations" after it's launched.
38
39
40------
41
42
43 ...Nine and a half months after launch, in September 1999, MCO was to fire its main engine to achieve an elliptical orbit around Mars. See figure 1. The spacecraft was to then skim through Mars’ upper atmosphere for several weeks in a technique called aerobraking to reduce velocity and move into a circular orbit. Friction against the spacecraft’s single, 5.5- meter solar array was to have slowed the spacecraft as it dipped into the atmosphere each orbit, reducing its orbit period from more than 14 hours to 2 hours. On September 23, 1999 the MCO mission was lost when it entered the Martian atmosphere on a lower than expected trajectory.
44
45
46The plan was for it to enter the atmosphere a bit so it could slow down with "aerobraking." But the error made it enter too close and (probably) crash.
47
48
49 ...The MCO had been on a trajectory toward Mars since its launch on December 11, 1998. All spacecraft systems had been performing nominally until an abrupt loss of mission shortly after the start of the Mars Orbit Insertion burn on September 23, 1999. Throughout spring and summer of 1999, concerns existed at the working level regarding discrepancies observed between navigation solutions. Residuals between the expected and observed Doppler signature of the more frequent AMD events was noted but only informally reported. As MCO approached Mars, three orbit determination schemes were employed. Doppler and range solutions were compared to those computed using only Doppler or range data. The Doppler-only solutions consistently indicated a flight path insertion closer to the planet. These discrepancies were not resolved.
50
51
52They knew it was off track and something was wrong during the months it was travelling to mars, but this was "informally reported." Their tracking said the probe was going to enter too close but they didn't fix it.
53
54
55 During the 24 hours preceding MOI, MCO began to feel the strong effects of Mar’s gravitational field and tracking data was collected to measure this and incorporate it into the orbit determination process. Approximately one hour prior to MOI, processing of this more accurate tracking data was completed. Based on this data, the first periapse altitude was calculated to be as low as 110km. The minimum periapse altitude considered survivable by MCO is 80 km.
56
57 The MOI engine start occurred at 09:00:46 (UTC) on September 23, 1999. All systems performed nominally until Mars’s occultation loss of signal at 09:04:52 (UTC), which occurred 49 seconds earlier than predicted. Signal was not reacquired following the 21 minute predicted occultation interval. Exhaustive attempts to reacquire signal continued through September 25, 1999, but were unsuccessful.
58
59
60an hour before it first entered mars orbit, their readings said that its orbit was going to be too low, but that it was still high enough to be survivable.
61
62all systems were nominal until its signal became blocked due to Mars being in the way. They never got its signal back after that.
63
64
65after the probe was lost, the "operations navigation team" (pretty sure the report says this is JPL) consulted the "spacecraft engineers" (in LMA) to talk about the navigation discrepencies (not sure if this is referring to the ones while crusing there or the ones while trying to enter orbit). Two days later they found the Pounds-Newtons problem.
66
67
68 ...Finally, after the fact navigation estimates, using all available data through loss of signal, with corrected values for the small forces ∆ V’s, indicated an initial periapsis (lowest point of orbit) of 57 km which was judged too low for spacecraft survival.
69
70
71after figuring out the pounds-newtons problem and substituting the right numbers, they found that the probe was actually 57 km high, not 110 km, which they judged was too low for the probe to make it.
72
73So the error made it so they both couldn't correctly control the probe and couldn't figure out where it was.
74
75
76------
77
78ftp://ftp.hq.nasa.gov/pub/pao/reports/1999/MCO_report.pdf
79 ...The output from the SM_FORCES application code as required by a MSOP Project Software Interface Specification (SIS) was to be in metric units of Newton- seconds (N-s). Instead, the data was reported in English units of pound-seconds (lbf-s). The Angular Momentum Desaturation (AMD) file contained the output data from the SM_FORCES software. The SIS, which was not followed, defines both the format and units of the AMD file generated by ground-based computers.
80
81
82Their official software interface spec said to use Newtons, but this wasn't followed.
83
84
85 ...Angular momentum management is required to keep the spacecraft’s reaction wheels (or flywheels) within their linear (unsaturated) range. This is accomplished through thruster firings using a procedure called Angular Momentum Desaturation (AMD). When an AMD event occurs, relevant spacecraft data is telemetered to the ground, processed by the SM_FORCES software, and placed into a file called the Angular Momentum Desaturation (AMD) file. The JPL operations navigation team used data derived from the Angular Momentum Desaturation (AMD) file to model the forces on the spacecraft resulting from these specific thruster firings. Modeling of these small forces is critical for accurately determining the spacecraft’s trajectory. Immediately after the thruster firing, the velocity change ( ∆ V) is computed using an impulse bit and thruster firing time for each of the thrusters. The impulse bit models the thruster performance provided by the thruster manufacturer. The calculation of the thruster performance is carried out both on-board the spacecraft and on ground support system computers. Mismodeling only occurred in the ground software.
86
87
88The probe has to periodically fire its thrusters for "AMD events." When this happens, JPL's navigation team on the ground needs to know its new position. They get data from the probe on how long the thruster has been firing, then put this into their navigation software to figure out how the probe's velocity changed after firing. LMA configured their navigation software with bogus data, so whenever an "AMD event" happened they would have a wrong idea of where the probe was.
89
90
91 ...The Software Interface Specification (SIS), used to define the format of the AMD file, specifies the units associated with the impulse bit to be Newton-seconds (N-s). Newton- seconds are the proper units for impulse (Force x Time) for metric units. The AMD software installed on the spacecraft used metric units for the computation and was correct. In the case of the ground software, the impulse bit reported to the AMD file was in English units of pounds (force)-seconds (lbf-s) rather than the metric units specified. Subsequent processing of the impulse bit values from the AMD file by the navigation software underestimated the effect of the thruster firings on the spacecraft trajectory by a factor of 4.45 (1 pound force=4.45 Newtons).
92
93
94The software they installed on the probe itself didn't have this problem (why?)
95
96
97 During the first four months of the MCO cruise flight, the ground software AMD files were not used in the orbit determination process because of multiple file format errors and incorrect quaternion (spacecraft attitude data) specifications. Instead, the operations navigation team used email from the contractor to notify them when an AMD desaturation event was occurring, and they attempted to model trajectory perturbations on their own, based on this timing information. Four months were used to fix the file problems and it was not until April 1999 that the operations team could begin using the correctly formatted files. Almost immediately (within a week) it became apparent that the files contained anomalous data that was indicating underestimation of the trajectory perturbations due to desaturation events. These file format and content errors early in the cruise mission contributed to the operations navigation team not being able to quickly detect and investigate what would become the root cause.
98
99
100Lockheed's navigation data ("AMD file") couldn't be used at all for about the first half of the flight there (4 months) due to format errors. So JPL relied on emails from LMA to let them know whenever an "AMD event" happened and tried to model things themselves. When the file was finally fixed, they realized almost immediately that the data was underestimating how the probe's velocity changed when its thrusters fired for AMD events. If LMA had fixed their file earlier, JPL might have had enough time to find and fix the pounds-newtons problem.
101
102
103I wonder if this contributed to the confusion about finding the cause of the navigation problem too. Their first instinct might be: what if we calculated things wrong? (did they ever calculate the trajectory wthout the software? the report only says they "attempted" to)
104
105They might also wonder: what if LMA forgot to email us? what if they emailed us something wrong?
106
107In general it would be more complicated to sort things out with half their calculations done by hand with data over email and half with the software.
108
109
110goes on to say it was hard for them to measure how off the thrusters were since they could only measure effects along line-of-sight to the probe and the thrusters fired sideways
111
112
113------
114
115ftp://ftp.hq.nasa.gov/pub/pao/reports/1999/MCO_report.pdf
116 ...The operations navigation team was not intimately familiar with the attitude operations of the spacecraft, especially with regard to the MCO attitude control system and related subsystem parameters. This unfamiliarity caused the operations navigation team to perform increased navigation analysis to quantify an orbit determination residual error. The error was masked by the lack of information regarding the actual velocity change ( ∆ V) imparted by the angular momentum desaturation (AMD) events. A line of sight error was detectable in the processing of the tracking measurement data, but its significance was not fully understood. Additionally, a separate navigation team was used for the MCO development and test phase. The operations navigation team came onboard shortly before launch and did not participate in any of the testing of the ground software. The operations navigation team also did not participate in the Preliminary Design review nor in the critical design review process. Critical information on the control and desaturation of the MCO momentum was not passed on to the operations navigation team.
117
118 MPL Recommendation:
119
120 The Board recommends that the MPL operations navigation team be provided with additional training and specific information regarding the attitude subsystems and any other subsystem which may have an impact on the accuracy of navigation solutions. To facilitate this, a series face-to-face meetings should be conducted with the spacecraft development, and operations teams to disseminate updated information and to discuss anomalies from this point forward. Long-term onsite support of an LMA articulation and attitude control system (AACS) person should be provided to the operations navigation team or a JPL resident AACS expert should be brought on the team to help facilitate better communication.
121
122
123
124The people who designed/tested the craft had problems communicating with the people who operated it, so they weren't completely sure how to control it. MPL reccomends "a series of face-to-face meetings" to "disseminate updated information and discuss anomolies."
125
126sounds like there were problems or quircks with the craft besides the pounds-newtons things that weren't communicated to the operating team.
127
128the operating team "didn't participate" in design review but apparently they should have.
129
130
131------
132
133ftp://ftp.hq.nasa.gov/pub/pao/reports/1999/MCO_report.pdf
134 ...During the MCO approach, a contingency maneuver plan was in place to execute an MCO Trajectory Correction Maneuver (TCM) -5 to raise the second periapsis passage of the MCO to a safe altitude. For a low initial periapsis, TCM-5 could also have been used shortly before the Mars Orbit Insertion (MOI) as an emergency maneuver to attain a safer altitude. A request to perform a TCM-5 was discussed verbally shortly before the MOI onboard procedure was initiated, but was never executed.
135
136 Several concerns prevented the operations team from implementing TCM-5. Analysis, tests, and procedures to commit to a TCM-5 in the event of a safety issue were not completed, nor attempted. Therefore, the operations team was not prepared for such a maneuver. Also, TCM-5 was not executed because the MOI maneuver timeline onboard the spacecraft took priority. This onboard procedure did not allow time for the upload, execution, and navigation verification of such a maneuver. Additionally, any change to the baselined orbit scenario could have exceeded the time for the MCO aerobraking phase when MCO was needed to support the communications of the MPL spacecraft. The criticality to perform TCM-5 was not fully understood by the spacecraft operations or operations navigation personnel.
137
138
139 ...MPL Recommendation:
140
141 The board recommends that the operations team adequately prepare for the possibility of executing TCM-5. Maneuver planning and scheduling should be baselined as well as specific criteria for deciding whether or not the maneuver should be executed. The full operations team should be briefed on the TCM-5 maneuver execution scenario and should be fully trained and prepared for its execution.
142
143
144shortly before the probe hit Mars atmosphere, JPL discussed doing a correction maneuver to raise its orbit, but never did it. The operations team wasn't breifed or trained on how to do this, and generally weren't prepared for this. They were also worried it would take too long when the climate orbiter was needed for the mars polar lander. They didn't understand the urgency of needing to raise the orbit.
145
146
147
148------
149
150
151ftp://ftp.hq.nasa.gov/pub/pao/reports/1999/MCO_report.pdf
152 ...One of the problems observed by the Board on MCO was that the systems engineering process did not adequately transition from development to operations. There were a number of opportunities for the systems engineering organization to identify the units problem leading to mission loss of MCO.. The lack of an adequate systems engineering function contributed to the lack of understanding on the part of the navigation team of essential spacecraft design characteristics and the spacecraft team understanding of the navigation challenge.
153
154
155the "systems engineering organization" was inadequate and missed multiple opportunities to catch the pounds-newtons problem (what were they?). They also contributed to the operations team confusion problem (how?).
156
157I think this is on JPL? the report says they were in charge of "project system engineering"
158
159
160 ...For example, the Angular Momentum Desaturation (AMD) events on MCO occurred 10- 14 times more often than was expected by the operations navigation team. This was because the MCO solar array was asymmetrical relative to the spacecraft body as compared to Mars Global Surveyor which had symmetrical solar arrays. This asymmetric effect significantly increased the Sun-induced (solar pressure-induced) momentum buildup on the spacecraft. To minimize this effect, a daily 180 o flip was baselined to cancel the angular momentum build up. Systems engineering trade studies performed later determined that this so-called “barbecue†mode was not needed and it was deleted from the spacecraft operations plan. Unfortunately, these systems engineering decisions and their impact to the spacecraft and the spacecraft trajectory were not communicated to the operations navigation team. The increased AMD events resulting from this decision coupled with the fact that the angular momentum (impulse) data was in English, rather than metric, units contributed to the MCO mission failure.
161
162
163due to a design change decision by "systems engineering", "AMD events" occurred 10-14 times more than the operations team expected and they didn't know why. The "systems engineering" team didn't inform them of the design change.
164
165probably made finding the problem harder? I guess one of the first things they'd ask is "why so many AMD events? can we even trust this data?"
166
167If they hadn't made the decision, the report says the error with the AMD units might not have mattered. So I think you can say this was just as big of a reason the probe didn't survive, it's just not as weird as the pounds-newtons problem.
168
169
170the asymetrical solar arrays probably aren't an unusual decision: other probes have it: http://www.planetary.org/explore/space-topics/space-missions/missions-to-mars.html (MOM, Mars Oddysey)
171
172what's unusual is that they didn't tell the nav team they removed their fix for the change in angular moment caused by this
173
174
175------
176
177
178ftp://ftp.hq.nasa.gov/pub/pao/reports/1999/MCO_report.pdf
179
180
181goes on to say there were communications problems between the teams. The operations navigation team didn't communicate concerns effectively and spacecraft operations team didn't understand them.
182
183 ...For example, the operations navigation team did not know until long after launch that the spacecraft routinely calculated, and transmitted to Earth, velocity change data for the angular momentum desaturation events. An early comparison of these spacecraft- generated data with the tracking data might have uncovered the units problem that ultimately led to the loss of the spacecraft.
184
185
186the operations navigation team could have compared velocity change data transmitted from the craft to their navigation data and found the problem right away, but they didn't know the craft transmitted this data "until long after launch."
187
188
189 When conflicts in the data were uncovered, the team relied on e-mail to solve problems, instead of formal problem resolution processes such as the Incident, Surprise, Anomaly (ISA) reporting procedure. Failing to adequately employ the problem tracking system contributed to this problem “slipping through the cracks.â€
190
191
192when they finally compared the probe and navigation data, they didn't put the problem in their official system (used email instead) and it "slipped through the cracks."
193
194This seems weird. They knew the probe was slightly of course, and at this point they also knew the ground model and thruster data from the probe mismatched. They store telemetry--at this point they would be able to compare all the delta-v data from the probe and from the ground model and figure out everything's off by a factor of 4.45.
195
196When they went to LMA after the probe crashed, it took them two days to find the problem, and they were able to use "all available data" recorded to figure out the probe was actually 57 km high. Why couldn't they have done that before the probe crashed?
197
198The report says it "slipped through the cracks," but how?
199
200
201------
202
203ftp://ftp.hq.nasa.gov/pub/pao/reports/1999/MCO_report.pdf
204 ...End-to-end testing to validate the small forces ground software performance and its applicability to the specification did not appear to be accomplished. It was not clear that the ground software independent verification and validation was accomplished for MCO. The interface control process and the verification of specific ground system interfaces was not completed or was completed with insufficient rigor.
205
206
207they didn't validate the software affected by the pounds-newtons problem (why?)
208
209
210------
211
212ftp://ftp.hq.nasa.gov/pub/pao/reports/1999/MCO_report.pdf
213 ...MCO Observation No. 3: Contingency Planning Process
214
215 Inadequate contingency planning for TCM-5 was observed to play a part in the MCO failure. The MCO operational contingency plans for TCM-5 were not well defined and or completely understood by all team members on the MCO operational team. The MCO project did not have a defined set of Go–No Go criteria for using TCM-5. There was no process in place to review the evaluation and decision criteria by the project and subsystem engineers before commitment to TCM-5. Polling of the team by the MCO Flight Operations Manager should establish a clear commitment from each subsystem lead that he or she has reviewed the appropriate data and believes that the spacecraft is in the proper configuration for the event.
216
217
218reading between the lines: apparently some people were more worried about the navigation errors than others. But they didn't have a way to resolve this to make a decision. The report reccomends they should have criteria to make this decision.
219
220
221 ... In addition, science personnel with relevant expertise should be included in all decisions where expert knowledge of Mars is required. While this was generally the case for the Mars ’98 program, such experts were not fully involved in the decisions not to perform TCM-5 prior to Mars orbit insertion.
222
223
224report says science experts in Mars weren't "fully involved" in the decision not to raise the probe's orbit.
225
226unfortunate that the report doesn't go into more detail on this decision
227
228
229 ...JPL’s navigation of interplanetary spacecraft has worked well for 30 years. In the case of MCO there was a widespread perception that “Orbiting Mars is routine.†This perception resulted in inadequate attention to navigation risk mitigation.
230
231 MPL Recommendation:
232
233 MPL project personnel should question and challenge everything—even those things that have always worked. JPL top management should provide the necessary emphasis to bring about a cultural change.
234
235
236The report says JPL should expect the unexpected.
237
238
239
240
241------
242
243
244https://edition.cnn.com/TECH/space/9909/30/mars.metric.02/
245 (CNN) -- NASA lost a $125 million Mars orbiter because a Lockheed Martin engineering team used English units of measurement while the agency's team used the more conventional metric system for a key spacecraft operation, according to a review finding released Thursday.
246
247
248the spec they all agreed on said to use metric
249
250
251 ...No one is pointing fingers at Lockheed Martin, said Tom Gavin, the JPL administrator to whom all project managers report.
252
253 "This is an end-to-end process problem," he said. "A single error like this should not have caused the loss of Climate Orbiter. Something went wrong in our system processes in checks and balances that we have that should have caught this and fixed it."
254
255
256 ...Metric system used by NASA for many years
257
258 A NASA document came out several years ago, when the Cassini mission to Saturn was under development, establishing the metric system for all units of measurement, Gavin said.
259
260 The metric system is used for the Polar Lander mission, as well as upcoming missions to Mars, he said.
261
262 That review panel's findings now are being studied by a second group -- a special review board headed up by John Casani, which will search for the processes that failed to find the metric to English mismatch. Casani retired from JPL two months ago from the position of chief engineer for the Lab.
263
264
265Casani did a second report
266
267
268 "We're going to look at how was the data transferred," Gavin said. "How did it originally get into system in English units? How was it transferred? When we were doing navigation and Doppler (distance and speed) checks, how come we didn't find it?"
269
270 ...Error points to nation's conversion lag
271
272 Lorelle Young, president of the U.S. Metric Association, said the loss of Climate Orbiter brings up the "untenable" position of the United States in relation to most other countries, which rely on the metric system for measurement. She was not surprised at the error that arose.
273
274 "In this day and age when the metric system is the measurement language of all sophisticated science, two measurements systems should not be used," Young said.
275
276 "Only the metric system should be used because that is the system science uses," she said.
277
278 She put blame at the feet of Congress that she said has squeezed NASA's budget to the point that it has no funds to completely convert its operations to metric.
279
280 "This should be a loud wake-up call to Congress that being first in technology requires funding," she said, "and it's a very important area for the country."
281
282
283funny
284
285
286-----
287
288second report March 2000:
289
290https://science.ksc.nasa.gov/mars/msp98/misc/MCO_MIB_Report.pdf
291 This second report, prepared by the Mars Climate Orbiter Mishap Investigation Board, presents a vision and recommendations to maximize the probability of success for future space missions. The Mars Climate Orbiter Phase I Report, released Nov. 10, 1999, identified the root cause and factors contributing to the Mars Climate Orbiter failure. The charter for this second report is to derive lessons learned from that failure and from other failed missions — as well as some successful ones — and from them create a formula for future mission success.
292
293
294why did they decide they needed a second report?
295
296did notice some problems with the first one--too muddled and not enough examples in some places. Seems like for some things it doesn't go into as much detail as it should.
297
298
299 ...The success of “Faster, Better, Cheaper†is tempered by the fact that some projects and programs have put too much emphasis on cost and schedule reduction (the “Faster†and “Cheaper†elements of the paradigm). At the same time, they have failed to instill sufficient rigor in risk management throughout the mission lifecycle. These actions have increased risk to an unacceptable level on these projects.
300
301
302this report blames FBC, not seen in 1st one
303
304goes on to blab about FBC more, then talk about a new "paradigm": "Mission Success First"
305
306didn't the mars polar lander report say this thinking lead to the decision to not have telemetry and to them not being able to learn much from the mission?
307
308
309 ...The Changing Environment
310
311 Significant change has taken place in the environment for NASA projects over the past five to seven years. The “Faster, Better, Cheaper†paradigm has been extremely successful in producing a greater number of smaller missions, with significantly shortened development cycles. Many of these missions are selected on the basis of proposals from principal investigators, who become responsible for managing all aspects of the mission through a NASA center. With freedom to operate outside traditional, NASA-specified management approaches, managers may use smaller teams and a strict “design-to-cost†philosophy in implementing projects.
312
313 One of the consequences of this approach has been increased partnering between NASA, industry, academia and other government agencies, necessitating increased and improved communications.
314
315
316written like an MBA freshman trying to hit the min page count for his essay
317
318says FBC led to using contractors more, and this introduces new "communications challenges."
319
320
321 ...A critical deficiency in Mars Climate Orbiter project management was the lack of discipline in reporting problems and insufficient follow-up. The primary, structured problem-reporting procedure used by the Jet Propulsion Laboratory — the Incident, Surprise, Anomaly process — was not embraced by the whole team. Project leadership did not instill the necessary sense of authority and responsibility in workers that would have spurred them to broadcast problems they detected so those problems might be articulated, interpreted and elevated to the highest appropriate level, until resolved.
322
323
324this report is worse than 1st one
325
326
327 This error was at the heart of the mission’s navigation mishap. If discipline in the problem reporting and follow-up process had been in place, the operations navigation team or the spacecraft team may have identified the navigation discrepancies, using the Incident, Surprise, Anomaly process, and the team would have made sure those discrepancies were resolved.
328
329
330still some useful info though
331
332sounds like it's saying the person who found the mismatch between the probe and ground nav data who didn't "embrace" reporting things formally more or less discovered the problem.
333
334
335 ...The Mars Climate Orbiter program did not incorporate a project-level mission assurance function during the operations phase. The Board observed lapses in the mission assurance function, such as the absence of an Incident, Surprise, Anomaly submittal documenting anomalies impacting the Angular Momentum Desaturation module.
336
337
338I guess this report sees this incident as the "heart" of the problem. Shifts more blame on NASA than Lockheed.
339
340also differs from first report in a key way, but vague on it:
341
342 The root cause of the mission failure may have been eliminated had there been a rigorous approach to the definition of mission-critical software — thereby allowing the aforementioned module to receive the appropriate level of review.
343
344
345first report says:
346
347
348ftp://ftp.hq.nasa.gov/pub/pao/reports/1999/MCO_report.pdf
349 ...The output from the SM_FORCES application code as required by a MSOP Project Software Interface Specification (SIS) was to be in metric units of Newton- seconds (N-s). Instead, the data was reported in English units of pound-seconds (lbf-s). The Angular Momentum Desaturation (AMD) file contained the output data from the SM_FORCES software. The SIS, which was not followed, defines both the format and units of the AMD file generated by ground-based computers.
350
351
352the second report seems to shift more blame to NASA, chalk up the failure to "faster better cheaper," and make the whole thing sound less weird than it is. A lot of media reports on the mars polar lander and the climate orbiter did these same three things.
353
354
355------
356
357
358summary of the 1st report:
359
360
361
362NASA's contract, software specification, and standard practice said to configure thruster-related software in metric units, and the rest of the project was done in metric, but LMA configured a particular component of the ground navigation software in English units.
363
364This misconfiguration led the probe to veer slightly off course as it made a series of small adjustments to its orientation on the trip to mars.
365
366A change to the spacecraft's system design made the need for such orientation corrections 10-14 times more frequent. This enabled the incorrect adjustments to build up enough to set the probe off course. The navigation team were not told of the design change or that the frequent corrections were expected behavior.
367
368At launch, the problematic component of the ground navigation software didn't work at all, and LMA took 4 months to get it working. Shortly after it started working, NASA realized this component was giving incorrect navigation data and began investigating the problem for the remaining half of the probe's journey. The investigators had trouble trying to find the root cause, as LMA did not pass on critical information on the probe's orientation control operations to the navigation team.
369
370Over the months of its journey, NASA found discrepancies in the probe's trajectory but could not resolve these discrepancies. As the navigation team learned more about the probe's operation, some members found discrepancies between probe telemetry and ground navigation data that might have lead NASA to discover the units problem, but these discrepencies were not correctly reported, contributing to the units problem "slipping through the cracks."
371
372As the probe approached Mars, data from Mars' gravity and other sources improved NASA's estimates of the probe's location, and one hour before entry they calculated the probe would enter the atmosphere up to half as low as initially predicted, but still 30% higher than minimum survivable altitude.
373
374As the probe approached mars, a request was made to perform contingency maneuvers to boost the probe's orbit, but this request was never executed. The decision not to boost the orbit was made without commitment from all subsystem leads and engineers, and did not fully involve scientists with expert knowledge on mars.
375
376At the time the probe entered Mars' atmosphere, communications with earth were occluded by Mars and its signal was lost. Its signal could not be found after that.
377
378After the probe was lost, the navigation team consulted LMA to figure out what went wrong. Two days later, they found the units problem. With corrected values, NASA found the probe's calculated trajectery entered Mars' atmosphere too low for the probe to survive.
379
380
381weird things about this:
382
383All the things that came together to make the problem work seem a little too convenient--software dev just happens to make more unusual choice for units without reading spec, systems engineers happen to make late change that makes problem accumulate 10-14 times faster, the software happens to not work at all for first half of mission and cut NASA's time to fix it, the person(s) who come closest to discovering the root cause happens to not "embrace" NASA's official reporting system.
384
385
386At the time better data estimated the probe was much lower, NASA already knew the orbital thruster data was wrong and that their nav tracking was off by an unknown amount. Why take the risk then to not put the probe higher? Reading between the lines in the report, it sounds like the decision to ignore the boost request didn't involve people it should have, and these people would have gone ahead with it.
387
388
389NASA knew the orientation thruster software was giving wrong values a week after LMA got it running. They were in contact with LMA about the problem for the remaining 5 months, but they never found the bug. After the probe crashed, it took them 2 days to find it. Once LMA knew the software was giving the wrong values, why couldn't they find the problem? they had 5 months to do a review that took them 2 days.
390
391
392The probe was lost while telemetry was cut off--can they really know for sure what caused it to be lost?
393
394
395------
396
397
398Kathy Sawyer before report released:
399
400http://www.washingtonpost.com/wp-srv/national/longterm/space/stories/orbiter100199.htm
401 Mystery of Orbiter Crash Solved
402
403 By Kathy Sawyer
404 Washington Post Staff Writer
405 Friday, October 1, 1999; Page A1
406
407 NASA's Mars Climate Orbiter was lost in space last week because engineers failed to make a simple conversion from English units to metric, an embarrassing lapse that sent the $125 million craft fatally close to the Martian surface, investigators said yesterday.
408
409 ...It now appears the error had affected the orbiter mission from its launching almost 10 months and 416 million miles before its Sept. 23 failure. And yet the problem was never caught and corrected by the system of checks and balances at the Jet Propulsion Laboratory (JPL) in California, which manages this and numerous other interplanetary missions for NASA.
410
411
412misleading in that WaPo makes NASA sound oblivious to the problem, but they found that there was an error almost as soon as they were able to.
413
414NASA was waiting on LMA to get the software working until mid-mission
415
416
417 As a result, flight controllers believe the spacecraft plowed into the Martian atmosphere, where the stresses crippled it, aborted its insertion into Martian orbit and most likely left it hurtling on through space in an orbit around the sun.
418
419 Baffled NASA officials said they were struggling to figure out how this happened, and bracing themselves for an onslaught of derision.
420
421 "Our inability to recognize and correct this simple error has had major implications," said JPL director Edward Stone.
422
423
424WaPo coming down hard on NASA
425
426
427 The initial error was made by contractor Lockheed Martin Astronautics in Colorado, which, like the rest of the U.S. launch industry, traditionally uses English measurements. The JPL navigation team, on the other hand, uses metric measurements in the complex business of figuring out a spacecraft's position relative to moving planets and keeping it on course. The contractor, by agreement, is supposed to convert its measurements to metrics.
428
429
430All of the software was configured in the correct units except that one component. According to this (https://www.space.com/3332-nasa-finally-metric.html
431), NASA had officially been using metric units since 1990.
432
433WaPo makes it sound like that one team at NASA was using weird units. Their report gives the impression that LMA forgot to convert units in general, not for a specific component, making it sound like an obvious problem and even more embarrassing that NASA didn't catch it than it actually was.
434
435
436 ...The initial error occurred in computer files forwarded regularly by Lockheed Martin to JPL navigators, according to Elachi's deputy, Tom Gavin. The wrong numbers referred to tiny thruster firings performed routinely twice a day. The navigators, in turn, performed their analysis of the spacecraft's position in space based on the assumption that the descriptions of these firings were in metric units of force per second (newtons). In fact, the numbers instead represented pounds (of force per second). This led to tiny miscalculations of the spacecraft's course that compounded over time. There were subtle clues in the data that something was off, but no one recognized the cause until yesterday.
437
438
439WaPo doesn't make clear NASA knew something was wrong and was investigating ("sublte clues ... no one recognized")
440
441probe being half as low as it should be not exactly a "subtle clue"
442
443there was an argument over whether or not to boost the probe before entry, which would make for interesting news (why didn't this make it to press while other info did?)
444
445
446 ...In this case, they have the surveyor in Martian orbit and the lander on its way.
447
448 This eases the sting of the talk show jokes that are bound to fly concerning the foolishness of the mistake, said Carl Pilcher, NASA's chief of solar system exploration. "Oh, God, there's nothing we can do about that. We'll live through it. . . . Then we'll wow them again."
449
450
451
452
453Kathy Sawyer after report released:
454
455
456http://www.washingtonpost.com/wp-srv/WPcap/1999-11/11/069r-111199-idx.html
457 Staff, Training Faulted in Loss of Mars Probe
458 By Kathy Sawyer
459 November 11, 1999
460
461 Poor training and communication inside the Mars team, and its failure to follow procedures, allowed a tiny metric conversion mistake to destroy the $125 million Mars Climate Orbiter, NASA investigators said yesterday.
462
463 The devastating report comes just three weeks before a sister ship, the Mars Polar Lander, is scheduled to land on the frozen wastes near the Red Planet's south pole. The team responsible for both missions, based at the Jet Propulsion Laboratory in Pasadena, Calif., is conducting its own urgent internal reviews while undergoing scrutiny by outside experts--all to ensure that similar problems have not doomed this mission as well.
464
465 "We clearly made a serious error," laboratory Director Edward Stone said at a tense NASA headquarters briefing.
466
467
468still coming down hard on NASA
469
470"devestating report"
471
472NASA employees burying the telementry-nav data discrepency and deciding not to boost orbit might be devestating, but those aren't mentioned in this article.
473
474
475 ...The immediate cause of the spacecraft's Sept. 23 disappearance as it entered Mars orbit was a failure by a young and inexperienced engineer at contractor Lockheed-Martin Astronautics to make a simple conversion from English units to metric in the coding of ground software used by the spacecraft navigators.
476
477 "To be very blunt about it, it was overlooked," said Lockheed-Martin official Noel W. Hinners. The NASA contract specified that the conversion should be made.
478
479 Far more serious was the failure by an established system of checks and balances to catch the error, officials said.
480
481 Investigation leader Arthur G. Stephenson, director of NASA's Marshall Space Flight Center in Huntsville, Ala., called the conversion error "a little thing" that should have been corrected.
482
483 The investigation "has identified other significant factors that allowed this error to be born, and then let it linger and propagate to the point where it resulted in a major error in our understanding of the spacecraft's path as it approached Mars," according to the 45-page report.
484
485
486quote not in report
487
488WaPo says the units error was a "little thing," faults NASA for not catching it
489
490Sawyer still doesn't make clear that NASA knew something was wrong for half the mission, apparently panicking toward end
491
492paints it like NASA just clumsy and "overlooked" the whole thing
493
494story from report is of an investigation, troubleshooting trying to find the cause, LMA being unhelpful, story from WaPo is NASA not paying attention
495
496a shame, since reader would miss learning some interesting puzzles about the failure
497 (like: why on earth couldn't LMA find the problem once they knew it was narrowed down to that component?)
498
499
500 ...The failure is a blemish on laboratory's record as world leader in the difficult business of interplanetary exploration. But NASA officials rejected suggestions that the problems stemmed from the agency's "smaller, faster, cheaper," philosophy, which has replaced complex billion-dollar projects with lower-budget missions flying much more frequently.
501
502
503chalks up to "smaller, faster, cheaper" (it's actually "faster, cheaper, better")
504
505kindof weird for WaPo to get that wrong?
506
507I guess if confronted about this article, Sawyer could have easy alibi, just say she was tired, rushed. Happens to everyone.
508
509But real weird thing is why does it seem like every article on this mission is that way...
510
511Sawyer still missed interesting story on decision not to boost orbit, even though this time it was right in the report. Is that excusable?
512
513
514 ... The investigators, noting that lab's deep space navigation techniques have worked well for 30 years, found at the laboratory "a widespread perception that 'orbiting Mars is routine.' This perception resulted in inadequate attention to navigation risk mitigation."
515
516 Said Weiler, "We need more people asking what can go wrong."
517
518
519doubling-down on "NASA not paying attention" narrative in last quote
520
521although, last quote is technically true...
522
523
524------
525
526
527https://www.wired.com/1999/09/mars-missions-metric-mixup/
528 09.30.99
529
530 Mars Mission's Metric Mixup
531
532 ...NASA officials said a simple miscommunication over different measurement standards – metric versus US – by teams controlling its Mars Climate Orbiter likely caused last week's loss of the spacecraft.
533
534 "The peer review preliminary findings indicate that one team used English units [inches, feet and pounds] while the other used metric units for a key spacecraft operation," said a statement released Thursday by NASA's Jet Propulsion Laboratory at the California Institute of Technology.
535
536
537why does Wired (and NASA officials?) say this was a miscommunication? NASA communicated to Lockheed that the units should be in metric.
538
539this article was written before the 1st report was released
540
541
542------
543
544
545press release from JPL shortly after crash (sep 23). This is before they found the units bug on Sep 29:
546
547https://nssdc.gsfc.nasa.gov/planetary/news/mco_19990923.html
548 "We had planned to approach the planet at an altitude of about 150 kilometers (93 miles). We thought we were doing that, but upon review of the last six to eight hours of data leading up to arrival, we saw indications that the actual approach altitude had been much lower. It appears that the actual altitude was about 60 kilometers (37 miles). We are still trying to figure out why that happened," said Richard Cook, project manager for the Mars Surveyor Operations Project at NASA's Jet Propulsion Laboratory. "We believe that the minimum survivable altitude for the spacecraft would have been 85 kilometers (53 miles)."
549
550makes it sound like they were clueless and didn't find out something wrong until after the fact
551
552
553report says:
554
555ftp://ftp.hq.nasa.gov/pub/pao/reports/1999/MCO_report.pdf
556 The MCO had been on a trajectory toward Mars since its launch on December 11, 1998. All spacecraft systems had been performing nominally until an abrupt loss of mission shortly after the start of the Mars Orbit Insertion burn on September 23, 1999. Throughout spring and summer of 1999, concerns existed at the working level regarding discrepancies observed between navigation solutions. Residuals between the expected and observed Doppler signature of the more frequent AMD events was noted but only informally reported. As MCO approached Mars, three orbit determination schemes were employed. Doppler and range solutions were compared to those computed using only Doppler or range data. The Doppler-only solutions consistently indicated a flight path insertion closer to the planet. These discrepancies were not resolved.
557
558 On September 8,1999, the final planned interplanetary Trajectory Correction Maneuver-4 (TCM-4) was computed. This maneuver was expected to adjust the trajectory such that soon after the Mars orbital insertion (MOI) burn, the first periapse altitude (point of closest approach to the planet) would be at a distance of 226km. See figure 4. This would have also resulted in the second periapse altitude becoming 210km, which was desired for the subsequent MCO aerobraking phase. TCM-4 was executed as planned on September 15, 1999.
559
560 Mars orbit insertion was planned on September 23, 1999. During the weeklong timeframe between TCM-4 and MOI, orbit determination processing by the operations navigation team indicated that the first periapse distance had decreased to the range of 150-170km During the 24 hours preceding MOI, MCO began to feel the strong effects of Mar’s gravitational field and tracking data was collected to measure this and incorporate it into the orbit determination process. Approximately one hour prior to MOI, processing of this more accurate tracking data was completed. Based on this data, the first periapse altitude was calculated to be as low as 110km. The minimum periapse altitude considered survivable by MCO is 80 km.
561
562 The MOI engine start occurred at 09:00:46 (UTC) on September 23, 1999. All systems performed nominally until Mars’s occultation loss of signal at 09:04:52 (UTC), which occurred 49 seconds earlier than predicted. Signal was not reacquired following the 21 minute predicted occultation interval. Exhaustive attempts to reacquire signal continued through September 25, 1999, but were unsuccessful.
563
564
565then talks about request to boost orbit--makes it pretty clear they weren't blind-sided
566
567
568why put out a press-release to set themselves up like this?
569
570
571...
572
573
574this is Richard Cook:
575
576https://www.jpl.nasa.gov/about/bio_cook.php
577
578responsible for Curiosity rover
579
580
581another press release sep 30:
582
583https://mars.nasa.gov/msp98/news/mco990930.html
584 "People sometimes make errors," said Dr. Edward Weiler, NASA's Associate Administrator for Space Science. "The problem here was not the error, it was the failure of NASA's systems engineering, and the checks and balances in our processes to detect the error. That's why we lost the spacecraft."
585
586 The peer review preliminary findings indicate that one team used English units (e.g., inches, feet and pounds) while the other used metric units for a key spacecraft operation. This information was critical to the maneuvers required to place the spacecraft in the proper Mars orbit.
587
588 "Our inability to recognize and correct this simple error has had major implications," said Dr. Edward Stone, director of the Jet Propulsion Laboratory. "We have underway a thorough investigation to understand this issue."
589
590------
591
592http://chronicle.augusta.com/stories/1999/10/01/tec_271066.shtml
593 10/01/99
594
595 Metric mixup doomed Mars spacecraft
596
597
598 By Associated Press
599
600
601 LOS ANGELES -- The $125 million spacecraft that was destroyed on a mission to Mars last week was probably doomed by NASA scientists' embarrassing failure to convert English units of measurement to metric ones, the space agency said Thursday.
602
603
604But that was LMA, not NASA scientists... why say that?
605
606
607 ...JPL said that its preliminary findings showed that Lockheed Martin Astronautics in Colorado submitted acceleration data in English units of pounds of force instead of the metric unit called newtons. At JPL, the numbers were entered into a computer that assumed metric measurements.
608
609
610I guess they're trying to say "NASA scientists" should have converted the file into metric themsevles before they were "entered into a computer"?
611
612from report:
613
614 ...On September 29, 1999, it was discovered that the small forces ∆ V’s reported by the spacecraft engineers for use in orbit determination solutions was low by a factor of 4.45 (1 pound force=4.45 Newtons) because the impulse bit data contained in the AMD file was delivered in lb-sec instead of the specified and expected units of Newton-sec.
615
616 "In our previous Mars missions, we have always used metric," Gavin said.
617
618
619file was machine-readable only
620
621computer read it from its filesystem, it wasn't "entered into a computer"
622
623
624a pattern with these--if you don't know details of what's going on you get completely different impression
625
626like before I looked at this I thought Lockheed did everything in pounds and didn't tell NASA, or handed NASA some calculations on a piece of paper in pounds without writing units (like this article suggests)
627
628this was written before the report, so author could blame their source
629
630
631
632------
633
634
635https://www.theguardian.com/science/1999/oct/02/spaceexploration.internationalnews
636 Mars scientists fail to measure up
637
638 Julia Hartley-Brewer
639
640 Fri 1 Oct 1999 21.28 EDT
641 First published on Fri 1 Oct 1999 21.28 EDT
642
643 Converting imperial measurements into metric units may not be rocket science, but it was clearly beyond the wits of Nasa scientists building the Mars space probe, which disappeared moments after reaching its destination.
644
645 Nasa yesterday had the embarrassing task of admitting that the disastrous loss of its £78m Mars Climate Orbiter last week was due to faulty flight calculations based on the simple failure to convert imperial measurements into their metric equivalents.
646
647 ...Although the faulty numbers had been used since the spacecraft's launch last December, the effect was so small that it went unnoticed over the months as the spacecraft journeyed toward Mars.
648
649
650------
651
652
653https://www.nature.com/articles/43974
654 NASA reworks its sums after Mars fiasco
655
656 07 October 1999
657
658 Tony Reichhardt
659
660 ...A preliminary analysis has revealed that the builder of the spacecraft, Lockheed Martin Astronautics of Denver, had been supplying information about routine propulsion manoeuvres in imperial units. But NASA space navigators assumed throughout the nine-month voyage from Earth to Mars that the acceleration data were in metric units, resulting in a large targeting error when the orbiter reached its destination.
661
662
663true, but article doesn't mention NASA told them to give data in metric
664
665doesn't mention that "supplying data" was a software config file
666
667
668 ...The confusion between imperial and metric that doomed the spacecraft originated with a Lockheed Martin subcontractor, says Hinners. The supplier of the spacecraft's propulsion system provided accompanying data for its system in imperial units, and Lockheed Martin simply incorporated those data and passed them through thesystem.
669
670 â€We should have caught [the mistake] then and there,†he says. The propulsion system for Lockheed Martin's Mars Global Surveyor, which is now operating successfully in Martian orbit, was built by another subcontractor who used metric units.
671
672
673this article slightly better
674
675Hinners' story doesn't explain everything though--if it's that simple, why did they get the right units for the same software on the probe? and what about other thrusters?
676
677 The embarrassing mistake also underlines the lack of uniformity in the US space sector. While virtually all scientists use metric units, many US engineers, both inside and outside the space programme, use imperial units, converting them when necessary. If the United States had converted to the metric system when the subject was debated more than 20 years ago, muses Hinners, “it might have saved this missionâ€.
678
679
680------
681
682a pattern in some articles to ignore that the problem was with software, in a single computer file among hundreds (thousands?), unreadable to humans, with everything else in right units
683
684so it gives the impression that it was some scientists forgetting to convert their units, like people could relate it to mistake in high-school physics homework
685
686
687------
688
689
690http://articles.latimes.com/1999/oct/01/news/mn-17288
691 Mars Probe Lost Due to Simple Math Error
692 October 01, 1999|ROBERT LEE HOTZ | TIMES SCIENCE WRITER
693
694 NASA lost its $125-million Mars Climate Orbiter because spacecraft engineers failed to convert from English to metric measurements when exchanging vital data before the craft was launched, space agency officials said Thursday.
695
696 A navigation team at the Jet Propulsion Laboratory used the metric system of millimeters and meters in its calculations, while Lockheed Martin Astronautics in Denver, which designed and built the spacecraft, provided crucial acceleration data in the English system of inches, feet and pounds.
697
698 As a result, JPL engineers mistook acceleration readings measured in English units of pound-seconds for a metric measure of force called newton-seconds.
699
700 In a sense, the spacecraft was lost in translation.
701
702 "That is so dumb," said John Logsdon, director of George Washington University's space policy institute. "There seems to have emerged over the past couple of years a systematic problem in the space community of insufficient attention to detail."
703
704
705
706------
707
708
709http://www.nytimes.com/1999/10/01/us/missing-what-didn-t-add-up-nasa-subtracted-an-orbiter.html
710 Missing What Didn't Add Up, NASA Subtracted an Orbiter
711
712 By ANDREW POLLACK OCT. 1, 1999
713
714 Simple confusion over whether measurements were metric or not led to the loss of a $125 million spacecraft last week as it approached Mars, the National Aeronautics and Space Administration said today.
715
716 An internal review team at NASA's Jet Propulsion Laboratory said in a preliminary conclusion that engineers at the Lockheed Martin Corporation, which had built the spacecraft, specified certain measurements about the spacecraft's thrust in pounds, an English unit, but that NASA scientists thought the information was in the metric measurement of newtons.
717
718 The resulting miscalculation, undetected for months as the craft, the Mars Climate Orbiter, was designed, built and launched, meant it was off course by about 60 miles as it approached Mars.
719
720
721
722it's not really what it says so much as how it says it and what it doesn't say
723
724if a reader were learning about the failure for the first time reading this, without reading the JPL report, what impression would they get?
725
726
727''This is going to be the cautionary tale that is going to be embedded into introductions to the metric system in elementary school and high school and college physics till the end of time,'' said John Pike, director of space policy at the Federation of American Scientists in Washington.
728
729
730 Lockheed's reaction was equally blunt.
731
732 ''The reaction is disbelief,'' said Noel Hinners, vice president for flight systems at Lockheed Martin Astronautics in Denver. ''It can't be something that simple that could cause this to happen.''
733
734
735 The finding was a major embarrassment for NASA, which said it was investigating how such a basic error could have gone through a mission's checks and balances.
736
737 ''The real issue is not that the data was wrong,'' said Edward C. Stone, the director of the Jet Propulsion Laboratory, in Pasadena, Calif., which was in charge of the mission. ''The real issue is that our process did not realize there was this discrepancy and correct for it.''
738
739 Dr. Stone added, ''The issue of English versus metric is a known issue because propulsion systems typically are designed in English units.''
740
741 Some experts also wondered how something so basic could have gone undetected so long.
742
743 ''Last time I checked I could sort of visually detect the difference between a foot and a meter,'' Mr. Pike said. ''This is kind of the very first thing in Physics 101 or Engineering 101. This is the only significant program failure that anyone's ever heard of that's due to this.''
744
745
746NY times spends half page smearing NASA for not paying attention
747
748goes on to chalk up to "cutting corners"
749
750
751 ...From the start NASA suspected human or software error, rather than mechanical failure. NASA and Lockheed officials said the full details of how the mistake occurred were not known. But basically, Lockheed was providing the Jet Propulsion Laboratory with data on the amount of force imparted to the spacecraft by its thrusters, which are fired periodically.
752
753
754are they trying to be confusing on purpose?
755
756
757 This was measured in pounds, Dr. Hinners said. But scientists at the Jet Propulsion Laboratory assumed the figure was in newtons and incorporated it into computer models that are used to calculate the spacecraft's position and direction.
758
759
760- be professor
761
762- go to teach class
763
764- TA sends you link to porno instead of class slides
765
766- "A local professor incorporated lurid pornographic material in his lecture today..."
767
768
769------
770
771http://www.slate.com/articles/news_and_politics/explainer/1999/10/why_hasnt_the_us_gone_metric.html
772 Oct. 6 1999 3:48 PM
773 Why Hasn't the U.S. Gone Metric?
774
775 On Sept. 23, the $123 million Mars Climate Orbiter was lost when it veered off course and burned up in the Martian atmosphere. Last week, NASA revealed the cause: One part of the orbiter's navigation system was speaking in English units and another in metric. Why hasn't the U.S. adopted the metric system?
776
777
778funny, Slate more accurate than NY times
779
780
781
782------
783
784
785http://news.bbc.co.uk/2/hi/science/nature/462264.stm
786 Thursday, September 30, 1999 Published at 18:53 GMT 19:53 UK
787
788 Sci/Tech
789
790 Confusion leads to Mars failure
791
792 The Mars Climate Orbiter Spacecraft was lost because one Nasa team used imperial units while another used metric units for a key spacecraft operation.
793
794 This information was critical to the manoeuvres required to place the spacecraft in the proper Mars orbit.
795
796 "Our inability to recognise and correct this simple error has had major implications," said Dr Edward Stone, director of the Nasa's Jet Propulsion Laboratory (JPL). "We have underway a thorough investigation to understand this issue."
797
798 The confusion over units was identified in preliminary findings by JPL's internal peer review. The transfer of information took place between the Mars Climate Orbiter spacecraft team in Colorado and the mission navigation team in California.
799
800 "People sometimes make errors," said Dr Edward Weiler, Nasa's Associate Administrator for Space Science. "The problem here was not the error, it was the failure of Nasa's systems engineering, and the checks and balances in our processes to detect the error. That's why we lost the spacecraft."
801
802
803...
804
805
806https://translate.google.com/translate?hl=en&sl=de&u=http://www.spiegel.de/wissenschaft/mensch/mars-climate-orbiter-absturz-wegen-leichtsinnsfehler-beim-rechnen-a-44777.html&prev=search
807 01.1.10.1999
808
809 Mars Climate Orbiter crash due to reckless calculation
810
811 Not because of a technical breakdown, but because the scientists involved counted in different units of measurement
812
813
814...
815
816
817what could explain all this?
818
819maybe:
820
821- handful of NASA officials deeply ashamed that they couldn't catch Lockheed's stupid easter egg error
822
823- rush to confess to media, atone for their sins
824
825- forget to tell media major details of what actually happen except that they're
826"really embarrassed"
827
828??
829
830
831------
832
833
834NASA press release on initial finding of cause for crash, Sep 30 1999 (first press release said "don't know yet"):
835
836https://mars.jpl.nasa.gov/msp98/news/mco990930.html
837 MARS CLIMATE ORBITER TEAM FINDS LIKELY CAUSE OF LOSS
838
839 A failure to recognize and correct an error in a transfer of information between the Mars Climate Orbiter spacecraft team in Colorado and the mission navigation team in California led to the loss of the spacecraft last week, preliminary findings by NASA's Jet Propulsion Laboratory internal peer review indicate.
840
841 "People sometimes make errors," said Dr. Edward Weiler, NASA's Associate Administrator for Space Science. "The problem here was not the error, it was the failure of NASA's systems engineering, and the checks and balances in our processes to detect the error. That's why we lost the spacecraft."
842
843 The peer review preliminary findings indicate that one team used English units (e.g., inches, feet and pounds) while the other used metric units for a key spacecraft operation. This information was critical to the maneuvers required to place the spacecraft in the proper Mars orbit.
844
845 "Our inability to recognize and correct this simple error has had major implications," said Dr. Edward Stone, director of the Jet Propulsion Laboratory. "We have underway a thorough investigation to understand this issue."
846
847
848NASA PR shoots self in foot once again
849
850press release had exact same problem as all the articles following it (and pulling quotes from it)--explains almost nothing, makes it sound like one NASA team forgot to write units talking to another NASA team.
851
852Any journalist writing above articles can say were just following what NASA said. Quotes from NASA officials made it sound like PR on mars climate orbiter was a persistent problem for them.
853
854Even still Slate, got it right for that one paragraph, NY times at least mentioned the word "software," and nature had some interesting if misleading info on a Lockheed subcontractor. So there must have been some actual journalism going on.
855
856
857...
858
859
860how to explain media reporting after JPL report?
861
862------
863
864after report:
865
866
867
868
869http://www.nytimes.com/1999/12/08/us/experts-warn-of-mars-on-the-cheap.html
870 Experts Warn of Mars on the Cheap
871
872 By WILLIAM J. BROAD
873
874 Dec 8. 1999
875
876 ...The panel, painting a picture of widespread design and management woes, cited not only a single main culprit but eight contributing factors ''that led directly or indirectly to the loss of the spacecraft.''
877
878 A mixup between English and metric units in navigation data was cited as the main reason the spacecraft went off course, and probably to its destruction, as it was trying to orbit Mars in September. All radio contact with the craft was lost.
879
880 Contributing flaws cited by the panel included insufficient team training, undetected errors in computer models of thruster firings, failures of the navigation team to know the spacecraft's orientation in space, inadequate effort by system engineers to track and double-check all mission aspects, a tendency for communications to be too informal, failure to weigh the need for an additional engine firing and the overworking of the navigation team.
881
882
883Why is NY times reporting it this way?
884
885"undetected errors in computer models of thruster firings" and "failures of the navigation team to know the spacecraft's orientation in space" were caused directly by the units "mixup," but NY times lists them as additional problems. The rest were all related to the units problem, but NY times doesn't say this.
886
887
888most of article is about managment and "cost cutting" problems at NASA
889
890NY times still doesn't explain what actually happened to the craft
891
892like WaPo, misses the aborted boost story
893
894so they basically let the "NASA wasn't paying attention" narrative from before stand, even though the report upends that.
895
896this appears to be NY times only article about the report
897
898
899------
900
901http://www.nytimes.com/1999/12/06/business/compressed-data-software-on-trial-in-mars-mission.html
902 Compressed Data; Software on Trial In Mars Mission
903
904 By LAURIE J. FLYNN
905 DEC. 6, 1999
906
907 It was the ultimate field test: the temperature is minus 75 and the terrain is entirely unfamiliar. And with a $165 million NASA mission at stake, there was no room for errors -- or even error messages.
908
909 ''We don't have the luxury of rebooting,'' said Curt Shacker, vice president of marketing at Wind River Systems, a software company based in Alameda, Calif., that supplied the operating system for the Mars Polar Lander.
910
911 After a trip that took 11 months, the spacecraft was to have landed on Mars last Friday and sent an electronic message back to NASA to confirm its arrival.
912
913 Instead, there was a silence that continued through late yesterday afternoon, with NASA mission control and Wind River officials playing a tense waiting game.
914
915 ''We got signals that the operating system was performing as it should have,'' said John Fogelin, vice president of engineering. ''The frustrating thing is, if we don't establish contact, we may never know what happened.''
916
917 Wind River's Vx Works, a commercially available computer operating system, is acting as the brains of the lander, controlling its various arms and collection devices and transmitting data back to NASA.
918
919 The program is embedded on an I.B.M. microprocessor, just as it was in the Mars Pathfinder mission in 1997.
920
921 Wind River said NASA selected Vx Works because of its reliability and compactness (its code fills only a fourth of a floppy disc), and because modifying commercial products is cheaper than creating one's own.
922
923 The stakes are high for all the participants. Exploration of the planet has been studded with technical failures.
924
925 Ten weeks ago, for example, the $125 million Mars Climate Orbiter, a sibling of the Polar Lander, burned up in the planet's atmosphere because of a failure to convert navigation data from feet and inches into metrics units. (The orbiter was also using the Wind River operating system, but the disaster was caused by human error, company officials want to make clear.)
926
927
928In an article about Mars probe software reliability writing after the MOC report released, NY times makes sure to point out the MCO units bug was "human error"
929
930not really helping to clear things up
931
932
933
934------
935
936
937
938
939https://spaceflightnow.com/news/0003/14mcoreport/
940 Sloppy management blamed for Mars Climate Orbiter loss
941 BY WILLIAM HARWOOD
942 Posted: March 14, 2000
943
944 An independent review board blames the loss of NASA's Mars Climate Orbiter last year on sloppy project management, a lack of agency oversight, poor communications and shortsighted engineering.
945
946 ...In an initial failure report released Oct. 15, the review board blamed the navigation error on a communications foulup between NASA's Jet Propulsion Laboratory and prime contractor Lockheed Martin.
947
948 The contractor provided propulsion data for periodic orientation maneuvers in English units instead of metric units as contracted. While the orientation maneuvers were minor, they affected the spacecraft's trajectory over time.
949
950 In a strongly worded report, the review team Monday criticized NASA for exercising poor project oversight and for cutting corners in a variety of areas to save money and/or to keep the mission on schedule. Overall, the review team concluded the Climate Orbiter project team took on the job without the proper resources to ensure mission success.
951
952 "The Mars Surveyor Program agreed to significant cuts in the monetary and personnel resources available to support the Mars Climate Orbiter mission, as compared to previous projects," the panel concluded.
953
954 "More importantly, the program failed to introduce sufficient discipline in the processes used to develop, validate and operate the spacecraft, and did not adequately install a mission-success culture that would shore up the risk introduced by these cuts. These process and project leadership deficiencies introduced sufficient risk to compromise mission success to me point of mission failure."
955
956
957Bill Harwood of spaceflightnow suggests the crash was not caused by the probe veering off course due to bogus thruster data, but by sloppiness and cutting corners at NASA.
958
959Relies on the second report
960
961rest of the article about management and funding problems at NASA
962
963
964------
965
966
967http://news.bbc.co.uk/2/hi/science/nature/686674.stm
968 Wednesday, 22 March, 2000, 15:40 GMT
969 Nasa's disastrous year
970
971 The disappearance of the Mars Polar Lander in December concluded a year of major failures for Nasa.
972
973 The lander vanished less than three months after Nasa lost its sister spacecraft, the Mars Climate Orbiter, in highly-embarrassing circumstances.
974
975 The $125m craft, which was to study the Red Planet's climate, went missing on 23 September after a mix-up between imperial and metric measurements
976
977 It transpired that one team had been working in centimetres, metres and kilogrammes, while another was using inches, feet and pounds.
978
979 The error meant that instead of the craft passing 150km above the planet's surface it flew at 60km, and probably burnt up.
980
981
982all I could find on BBC on MCO in months after report released
983
984with exception of:
985
986http://news.bbc.co.uk/2/hi/talking_point/552655.stm
987 Thursday, 9 December, 1999
988
989 Is space exploration a waste of money?
990
991 Nasa scientists have lost contact with the latest unmanned spacecraft sent to Mars, the $165m Mars Polar Lander.
992
993 In September, the $125m Mars Climate Orbiter mission also failed, after an embarrassing mix-up over metric and imperial measurements.
994
995 The history of failed missions to Mars stretches back 40 years, so are they a waste of taxpayers' money, which would be better spent on more earthly needs?
996
997 Is the cost of space adventure unjustified while poverty exists in many parts of the world?
998
999
1000------
1001
1002
1003by CNN saying report about to be released:
1004
1005http://www.cnn.com/TECH/space/9911/10/orbiter.01/
1006 NASA set to release report on loss of Mars orbiter
1007 November 10, 1999
1008
1009 From Correspondent
1010 Miles O'Brien
1011
1012 (CNN) -- Six weeks after controllers lost the Mars Climate Orbiter, NASA is poised this afternoon to release a scathing report of what went wrong and what needs to be done to ensure the Orbiter's sister ship -- the Mars Polar Lander -- makes it safely to the surface of the red planet.
1013
1014 Months before the Climate Orbiter reached Mars, controllers at NASA's Jet Propulsion Laboratory saw signs of trouble but failed to understand the implications until it was too late.
1015
1016 "When the earlier trajectory work was being done it was noticed that the predictions … were not as exquisitely accurate as they normally are," said mission controller Ben Clark of Lockheed Martin Aeronautics.
1017
1018 Such computer predictions are crucial because Earth-bound antennas offer controllers a one-dimensional and thus incomplete picture of where to find a spacecraft. But the discrepancies were so slight, the team simply discounted the predictions.
1019
1020
1021report says:
1022
1023 ...Four months were used to fix the file problems and it was not until April 1999 that the operations team could begin using the correctly formatted files. Almost immediately (within a week) it became apparent that the files contained anomalous data that was indicating underestimation of the trajectory perturbations due to desaturation events. These file format and content errors early in the cruise mission contributed to the operations navigation team not being able to quickly detect and investigate what would become the root cause.
1024
1025 In April 1999, it became apparent that there was some type of mismodeling of the AMD maneuvers. In attempting to resolve this anomaly, two factors influenced the investigation.
1026
1027 ...It was clear that the operations navigation team did not communicate their trajectory concerns effectively to the spacecraft operations team or project management. In addition, the spacecraft operations team did not understand the concerns of the operations navigation team.
1028
1029
1030which is true? maybe CNN right that at least some discounted the problem--why else decide not to boost?
1031
1032CNN goes on to talk about the flywheel, thrusters correcting, what they actually do
1033
1034
1035 ...The prime contractor for the mission, Lockheed Martin, measured the thruster firings in pounds even though NASA had requested metric measurements. That sent the Climate Orbiter in too low, where the $125-million spacecraft burned up or broke apart in Mars' atmosphere.
1036
1037
1038this article a lot better than e.g. http://www.nytimes.com/1999/12/08/us/experts-warn-of-mars-on-the-cheap.html
1039
1040actually says who was responsible for units problem, not just "there was a mixup"
1041
1042CNN doesn't have anything on report after this
1043
1044
1045------
1046
1047
1048Oberg on why the MCO crashed:
1049
1050
1051https://spectrum.ieee.org/aerospace/robotic-exploration/why-the-mars-probe-went-off-course
1052 1 Dec 1999 | 19:33 GMT
1053 Why the Mars Probe went off course
1054
1055 Far more was at fault with the Mars Climate Orbiter than a simple mixup in converting metric and British units
1056
1057 By James Oberg
1058
1059 Aircraft accident investigators have a special term for a particularly insidious type of accident--CFIT, or controlled flight into terrain. It occurs when human error in the cockpit, in the traffic control tower, or in the flight planning process in effect flies a perfectly good airplane right into the ground.
1060
1061 In the past 40 years, space flight has encountered all sorts of failure modes. Propulsion systems have leaked and exploded. Power systems have short-circuited. Observation instruments have failed to work or have been pointed in wrong directions. But until this year no CFIT had occurred in outer space.
1062
1063 Then, on 23 September, through a series of still-baffling errors, flight controllers at the Jet Propulsion Laboratory, a California Institute of Technology facility under contract to NASA, sent erroneous steering commands to the Mars Climate Observer as it neared the target planet. Obeying blindly like all true robots, the probe, metaphorically speaking, marched off the cliff and was destroyed.
1064
1065
1066Oberg says NASA sent commands to send the probe crashing into Mars
1067
1068which is technically true
1069
1070
1071 NASA assigned three separate teams to investigate the embarrassing, US $125 million debacle and determine its cause. Preliminary public statements faulted a slip-up between the probe's builders and its operators, a failure to convert the English units of measurement used in construction into the metric units used for operation.
1072
1073 After six weeks, on 10 November, NASA officials released their preliminary findings. However, an IEEE Spectrum investigation had been going on separately, using unofficial sources associated with the program and independent experts. Spectrum quickly learned that far more had gone wrong than just a units conversion error. A critical flaw was a program management grown too confident and too careless, even to the point of missing opportunities to avoid the disaster.
1074
1075
1076Oberg and IEEE did their own investigation and found "far more had gone wrong thatn just a units conversion error." They found management was too confident and careless, and missed "opportunities to avoid the disaster."
1077
1078isn't that what the report said already?
1079
1080
1081 As reconstructed by Spectrum, ground controllers ignored a string of indications that something was seriously wrong with the craft's trajectory, over a period of weeks if not months. But managers demanded that worriers and doubters "prove something was wrong," even though classic and fundamental principles of mission safety should have demanded that they themselves, in the presence of significant doubts, properly "prove all is right" with the flight. As a result, the probe was about 100 kilometers off course at the end of its 500-million-kilometer voyage--more than enough to accidentally hit the planet's atmosphere and be destroyed.
1082
1083
1084says some ignored indications something was wrong with trajectory, acted skeptical to "worriers and doubters."
1085
1086
1087 In an internal memo dated 18 October, not intended for non-JPL readers, a laboratory official summarized the way thinking was developing: "There might have been some overconfidence, inadequate robustness in our processes, designs, or operations, inadequate modeling and simulation of the operations, and failure to heed early warnings." No, it was not a simple mistake at all, as NASA finally explained in detail on 10 November.
1088
1089
1090JPL memo says some got cocky
1091
1092
1093 ...The Mars Climate Observer--the first weather satellite for another planet--had not been intended to enter Mars's atmosphere. With an orbital observation program several years long stretching ahead of it, the aim had been for the probe to swing around behind Mars just above its atmosphere. Then its directional rockets were to fire for 15 minutes, to slow it down so that it was captured by the planet's gravity.
1094
1095 Next, a series of skimming motions across the upper atmosphere were to lower the probe's orbit and adjust it till it became circular
1096
1097
1098from report released 3 weeks earlier:
1099
1100 Nine and a half months after launch, in September 1999, MCO was to fire its main engine to achieve an elliptical orbit around Mars. See figure 1. The spacecraft was to then skim through Mars’ upper atmosphere for several weeks in a technique called aerobraking to reduce velocity and move into a circular orbit. Friction against the spacecraft’s single, 5.5- meter solar array was to have slowed the spacecraft as it dipped into the atmosphere each orbit, reducing its orbit period from more than 14 hours to 2 hours. On September 23, 1999 the MCO mission was lost when it entered the Martian atmosphere on a lower than expected trajectory.
1101
1102
1103it was supposed to enter the atmosphere. I think a lot of probes do this.
1104
1105why does Oberg say this?
1106
1107
1108 ... As mentioned, accurate orbit determination and targeting any course changes required a complete knowledge of all forces acting on the probe. With this, the computer programs generate both an ephemeris and an error estimate for the flight path based on how widely scattered the navigation marks are. It was at the first step--where the forces acting on the spacecraft were input in the wrong units, according to preliminary NASA explanations after the debacle--that the computers were led astray. But what doomed the spacecraft, experts believe, was errors in human judgment that led to a poor grasp of the navigation uncertainty, in general, and of the meaning of the great range that existed in the calculation of the altitude over Mars, in particular.
1109
1110
1111Oberg says the real problem was that NASA made bad judgement calls, didn't grasp the significance of the error on the probe's position
1112
1113It seems like this could be true for the decision not to boost--they got corrected measurements saying the probe was lower than their nav models predicted, and on top of that they knew there was additional error they hadn't figured out yet. So they knew at that point there was uncertainty in the measurement of the probe's position, and they knew it was making it lower than it was supposed to be.
1114
1115
1116 ... At the 10 November news conference, NASA revealed even more damning information about this issue. Fully aware of the issue of asymmetric torques, the original designers of the probe had planned to neutralize them by slowly spinning the spacecraft about its long axis, perpendicular to the sun (the so-called barbecue mode). Some time later, concern over a potential shortfall in the electric power budget caused the design team to change this balancing spin to a constant face-onto-the-sun orientation. There were no navigation experts on the team at this point (they were in fact not added until two months before launch, and had no significant knowledge of the spacecraft's peculiarities even then). So the change was made without opposition.
1117
1118
1119Oberg mentions the design change decision that led to more thruster firings
1120
1121apparently it was mentioned in a NASA press conference on nov 10, when the report was released
1122
1123this is a summary of the conference:
1124
1125
1126
1127http://www.cse.psu.edu/~gxt29/bug/localCopies/marsOrbiter.html
1128 Contractor takes blame for math goof that crashed Mars probe
1129
1130 Copyright © 1999 Nando Media
1131 Copyright © 1999 Associated Press
1132
1133 By PAUL RECER
1134
1135 WASHINGTON (November 10, 1999 6:02 p.m. EST http://www.nandotimes.com) - For nine months, the Mars Climate Orbiter was speeding through space and speaking to NASA in metrics. But the engineers on the ground were replying in non-metric English.
1136
1137 The mathematical mismatch that was not caught until after the $125 million spacecraft, a key part of NASA's Mars exploration program, was sent crashing too low and too fast into the Martian atmosphere. The craft has not been heard from since.
1138
1139 "We were on the wrong trajectory and our system of checks and balances did not allow us to recognize that," Edward Stone, director of the Jet Propulsion Laboratory, said Wednesday. The NASA center in California was in charge of the Mars mission.
1140
1141 Noel Henners of Lockheed Martin Astronautics, the prime contractor for the Mars craft, said at a news conference that his company's engineers were responsible for ensuring that the metric data used in one computer program were compatible with the English measures used by another program. The simple conversion check was not done, he said.
1142
1143 "It was overlooked," Henners said.
1144
1145
1146this says something--LMA told the press it was a problem originating with them in giving data from one software program to another, but apparently that got lost in translation
1147
1148
1149 The Mars Climate Orbiter was launched Dec. 11 and spent nine months coasting toward Mars.
1150
1151 Art Stephenson, director of the Marshall Spaceflight Center and head of a NASA investigation team, said that the spacecraft was not symmetrical and that pressure from the sun caused it to slowly twist or roll as it sped along. On-board gyroscopes partially controlled the motion, but eventually rocket firings were needed to stabilize the craft, he said. This happened 12 to 14 times a week over the nine-month voyage.
1152
1153 Engineers on the ground calculated the size of the rocket firing using feet-per-second of thrust, a value based on the English measure of feet and inches. However, the spacecraft computer interpreted the instructions in Newtons-per-second, a metric measure of thrust. The difference is 4.4 feet per second.
1154
1155 "Each time there was a burn (rocket firing) the error built up," said Stephenson.
1156
1157 As the spacecraft approached its rendezvous with Mars and the engineers prepared for a final rocket firing, there were indications that something was seriously wrong with the navigation, but no corrective action was taken, Stephenson said.
1158
1159
1160also told press they knew something was "seriously wrong" as it approached Mars, one of the most interesting stories about this
1161
1162
1163 When the Mars Climate Orbiter did fire its rockets, the craft went too low into the planet's atmosphere instead of into a safe orbit. Communication signals stopped when the craft passed behind Mars and have not been heard since.
1164
1165 "We entered the Mars atmosphere at a much lower altitude (than planned)," said Ed Weiler, NASA's chief scientist. "It (the spacecraft) either burned up in the Martian atmosphere or sped out (into space). We're not sure which happened."
1166
1167 Stephenson said that the problem was not with the spacecraft, but with the engineers and the systems used to direct it.
1168
1169 "The spacecraft did everything we asked of it," said Stephenson. He said the mathematical mismatch was "a little thing" that could have been easily fixed if it had been detected.
1170
1171 "Sometimes the little things can come back and really make a difference," he said.
1172
1173
1174this article has a lot of the details others missed
1175
1176I think this is right:
1177
1178https://www.reddit.com/r/explainlikeimfive/comments/3jy4kl/eli5_why_do_so_many_news_agencies_cite_reuters/
1179 Every newspaper/radio- or TV-station uses the service of press agencies like Reuters, AP and a couple of others (AFP is based in France, dpa in Germany, etc...) and you pay a certain fee to use their reports for your stories. Then you usually rewrite the thing/throw in a few of your own sources/ do some more research / mix the information from several press agencies.
1180
1181
1182AP sort-of a standard source for other news, like they'd research the AP reporting first
1183
1184adds to the mystery of how these details got lost in other reports
1185
1186even without AP, all this info was in the NASA press conference anyway (and the report).
1187
1188But they can't really make the excuse that the report was too long or confusing (not a light read)--it looks like some important bits were summarized in the press conference and AP.
1189
1190
1191Oberg's report is most detailed I could find so far
1192
1193
1194https://spectrum.ieee.org/aerospace/robotic-exploration/why-the-mars-probe-went-off-course
1195 ...The corresponding British unit was the pound force. If the values provided by the spacecraft engineers at Lockheed Martin Astronautics Co., Denver, Colo., had been in pound force, they would have been too large by a factor of 4.45.
1196
1197 But in engineering terms, these two values are still of the same order of magnitude. In other words, there is no really gross mismatch in the scale of calculations made with the one or the other--the kind of mismatch that can provide an intuitive hint that something doesn't add up. These two units were close enough in magnitude that the unintentional substitution of one for the other apparently rang no warning bells.
1198
1199 According to a JPL spokesman, every maneuver intended to dump momentum added a velocity error of about 0.001 meter per second, on a probe that was traveling at a rate of tens of kilometers per second. These deflections themselves were not the problem, but their incorrect modeling was, when the computer was told the spacecraft had received a force of four or five times as great as it really had.
1200
1201 "Every momentum dump will introduce an additional bogus force into the navigation software," Spectrum's interplanetary navigation consultant explained. "Ultimately the curve fit will be warped by this constant application of a non-existent force."
1202
1203
1204explains how the measurement error would sneak-up on them
1205
1206
1207 ... At the 10 November press conference, chief investigator Arthur Stephenson explained why the initial trajectory had appeared healthy. There had been a software error in the implementation of the small forces model that was not fixed until April. This delay had forced JPL to make its own estimates of the course deviation from the AMD maneuvers, and they had done this accurately. Because of the rush to get the small forces model operational, the testing program had been abbreviated, Stephenson admitted. "Had we done end-to-end testing," he stated at the press conference, "we believe this error would have been caught." But the rushed and inadequate preparations left no time to do it right.
1208
1209
1210Stephenson says JPL succeeded in accurately modelling the small forces on the craft by hand. Says testing on the model LMA delieved was skimped since it was finished half-way through the mission.
1211
1212if the nav team was getting it right by hand and knew something was wrong with the small forces software, why did they keep using it and let errors build up?
1213
1214
1215 ...Spectrum's investigation uncovered a report that one navigator wrote a memo describing some vague uneasiness he had about the trajectory as planning began for the TCM-3 burn on 23 July. Possibly, the unease was due to an intuitively correct appreciation of higher-than-expected course-error estimates.
1216
1217 The size of the midcourse burns was large, unlike the experience with the Mars Global Surveyor two years earlier. In that case, the third midcourse correction was canceled entirely, and the fourth and final one was only 0.29 m/s. But TCM-3 for the climate observer was 3.3 m/s, and TCM-4 was almost 2 m/s.
1218
1219 NASA confirmed this navigational "unease." According to Stephenson, within days of the activation of the small forces model, navigators began to suspect something was not quite right with the spacecraft's ephemeris. "We even had a big meeting in Denver about this," he told the press conference on 10 November. "But the issue was never resolved."
1220
1221
1222Stephenson says NASA met LMA (in Denver) about "unease" with the small forces model, but "the issue was never resolved."
1223
1224
1225 Although the navigators continued to express concern about the spacecraft trajectory, NASA's Stephenson explained why there had been no management response. "They did not use the existing formal process for such concerns," he stated. JPL has a special form to invoke a so-called incident surprise and analysis procedure, and the navigators did not follow the rules about filling out that form to document their concerns.
1226
1227
1228this seems off
1229
1230According to Stephenson, navigators were continually "concerned" and expressing "unease" about the navigation errors, even arranged meeting about it, but there was no management response since they didn't fill out the right forms?
1231
1232the second report (which sucked) says the incident with not filling out the incident form was "heart of the issue"
1233
1234
1235 Stephenson did admit that inadequate navigation team staffing was a contributing factor to the accident because they were responsible for three separate missions at the same time. In addition, their training in team operations was inadequate, according to Stephenson. "This was a problem of transition from the era of very large teams to when teams are very small," he explained.
1236
1237
1238Stephenson "admits" the nav team was overworked, understaffed, undertrained
1239
1240
1241 ...But even if the targeting software worked perfectly--and some experts believed it did--the maneuver doomed the probe because the spacecraft was not really where the navigators thought it was. The trajectory errors induced by the wrong units in the small forces model made Earth think the spacecraft was several hundred kilometers from where it actually was. This mistake would displace the point of closest approach to Mars by about the same amount. This was confirmed by NASA on 10 November.
1242
1243
1244is Oberg suggesting it could be an error in the targetting software too?
1245
1246
1247Oberg talks about the decision not to boost the orbit.
1248
1249
1250report says:
1251
1252 Mars orbit insertion was planned on September 23, 1999. During the weeklong timeframe between TCM-4 and MOI, orbit determination processing by the operations navigation team indicated that the first periapse distance had decreased to the range of 150-170km
1253
1254 During the 24 hours preceding MOI, MCO began to feel the strong effects of Mar’s gravitational field and tracking data was collected to measure this and incorporate it into the orbit determination process. Approximately one hour prior to MOI, processing of this more accurate tracking data was completed. Based on this data, the first periapse altitude was calculated to be as low as 110km. The minimum periapse altitude considered survivable by MCO is 80 km.
1255
1256 ...Finally, after the fact navigation estimates, using all available data through loss of signal, with corrected values for the small forces ∆ V’s, indicated an init ial periapsis (lowest point of orbit) of 57 km which was judged too low for spacecraft survival.
1257
1258
1259Oberg says:
1260
1261 ... Reportedly, the first trajectory measurements after the fourth burn showed the probe to be right on course for a 193-km close point (they had aimed for 224 km, just to be on the safe side). Still, an error of 30 km when the expected error was less than ±10 km may have been slightly disturbing.
1262
1263 In the following days, as additional tracking marks were accumulated, the navigation programs began to show the estimated closest point sinking lower and lower. It looked as if the probe was drifting off course, an impossibility if all the forces on it were properly modeled. What was really happening was that as the probe fell towards Mars, its increasing speed helped refine the probe's true path. But instead of believing that the newer values were closer to reality, controllers apparently chose to trust the earlier navigation, and suspect instead that something was now going wrong with the navigation software. The spacecraft itself, they assumed, remained on a safe trajectory.
1264
1265
1266Oberg says controllers "apparently" ignored warnings that probe was closer to Mars than nave software said
1267
1268
1269 No final trajectory correction
1270
1271 The 4 October issue of Aviation Week magazine provided detailed navigation data from this period, based on its inside sources at JPL and confirmed by other experts who talked anonymously with Spectrum. According to the magazine, "Several days after the TCM-4, the navigation calculations had relatively poor convergence. The new numbers were trending to 150-180 km--but with uncertain confidence." The phrase "uncertain confidence" probably means that the calculated error bounds had gotten quite high. Yet even though some predictions were up to 70 km off the aim point, navigators still acted as if they believed the aiming accuracy was within 10 km.
1272
1273
1274Oberg and Aviation Week's inside sources at JPL said the nav calculations had "uncertain confidence" on approach to mars, but navigators still acted like the confidence in the probe's trajectory was high.
1275
1276
1277 According to Aviation Week, project management decided on 19 September to forgo TCM-5, because of a flyby range predicted at 150 to 180 km and a belief that the space craft would be safe to 85 km.
1278
1279
1280Aviation Week said project management decided to forego a boost since nav estimates put the probe at 150-180 km high
1281
1282lines up with what's said in report, decision being on Sep 19 in the "weeklong period" mentioned
1283
1284something off here--Oberg's article and Stephenson blame the nav team for not raising concerns with management, while the report blames management for not consulting everyone for the decision not to boost.
1285
1286and the report says a request was made to boost which was not executed--if that happened, it seems hard to believe whoever in charge of that decision didn't know something was wrong
1287
1288
1289 But the 85-km figure was not based on actual engineering analysis. According to a control center expert who has spoken with Spectrum, JPL operators put their faith in a 160-km fly-by altitude until a few hours before encounter. Then a recomputation showed that the altitude would be 110 km. When Lockheed Martin engineers were asked to examine the effects at this altitude, they were frightened. Atmospheric drag would probably be enough to tumble the spacecraft and overheat it. As the spacecraft passed behind Mars, navigators came up with a new estimate: 95 km. This would generate heating equivalent to a bank of propane torches.
1290
1291
1292report says 110 km computation came "approximately one hour" before entry, not "a few hours" (why the difference?)
1293
1294Oberg says even the too-high estimate of the entry would be enough to crash the craft
1295
1296If this is right, Lockheed engineers were asked about the surviability of the craft within 1 hour before entry and told them it probably would crash at 110 KM
1297
1298but this didn't end up in the report--it just said "the minimum periapse altitude considered survivable by MCO is 80 km"
1299
1300maybe "minimum survivable" a low bar?
1301
1302would a boost have still been possible at this point?
1303
1304
1305 A sense of gloom descended on the controllers as they began listening for the reacquisition of radio contact on the other side of Mars. By then, JPL navigators had an even newer estimate--57 km, where the heating would be 10 times as bad as at 95 km.
1306
1307
1308kindof misleading--they got the new estimate after finding the units bug a week later
1309
1310
1311 Spectrum has been told that this decision to forgo the TCM-5 correction was flawed: "Given expected errors in altitude targeting of about 10 km, a spread of values over a 100-km range [from 70 to 180 km] should have people screaming down the halls," one navigation expert told us. "This tells you that you have no idea where your spacecraft is, and therefore your trajectory has an unacceptable probability of intersecting the planet's atmosphere. To me this says 'aim high' and put another 200 km in there to be safe."
1312
1313
1314this seems right, and basically what report says: they had a lot of uncertainty and should have boosted
1315
1316Oberg took and already damming story and embelished it a little (more time after 110KM estimate, less time before 57 KM estimate). Why?
1317
1318
1319 Reportedly there were more indications of trouble. After the last burn, when navigation was hinting at a 100-km range of uncertainty of true position, somebody at JPL ran the data through the 1998 Mars Pathfinder navigation code (different from the Mars probe code). It showed the spacecraft was off course by hundreds of kilometers, which turned out to be correct.
1320
1321 Asked about these indications by Spectrum at the 10 November press conference, JPL officials denied any knowledge. When Spectrum then asked if the main actors in the navigation misjudgments could now be allowed to speak freely with the press, JPL director Edward Stone stated coldly that "everyone's full attention must be on the December 3 [Mars Polar Lander] landing." Another journalist jumped on what he called this "artfully unresponsive" answer and got Stone to concede that perhaps some time next spring, after the final report is issued, the investigation gag order might be lifted.
1322
1323
1324interesting--according to Oberg, JPL director Stone put a gag order on the MCO team talking to the press. Insinuates trying to cover it up.
1325
1326was that second report the "final report?" released in March 2000
1327
1328
1329 Rumors even assert that the leader of the navigation team, Pat Esposito, had recommended making the TCM-5 burn to raise the fly-by range "just in case." He declined comment when phoned by Lee Dye, retired Los Angeles Times science editor and now a contributor to the abcnews.com science Web page. Dye then asked Thomas Gavin, deputy director of JPL's space and earth science directorate, about rumors that Esposito had personally called Gavin to urge making the burn. Gavin vehemently denied it. "I thought the telephone line was going to vaporize," Dye reported.
1330
1331
1332Oberg raises that it was nav team leader who requested burn. For whatever reason declined to comment to press. Gavin denied nav leader called him and reporter Lee Dye says Gavin was angry about it.
1333
1334this says similar things, pulls quote from Lee Dye report: http://www.spacedaily.com/news/mars-mco-99f.html
1335
1336talks about JPL team members wanted to pummel each other
1337
1338
1339 ... NASA presented a strikingly different version of the dispute. Stephenson admitted that in fact the navigation team had verbally requested the TCM-5 maneuver be performed. Permission was denied-- "properly," Stephenson asserted--because the team was not prepared to perform the burn; no procedures for the fast response needed had been put in place or practiced. Besides, Stephenson continued, "the navigation team was not clear on what the problem was, but they did not see it as a case of possible loss of spacecraft." He concluded: "Unfortunately, maybe they didn't see it as a big issue, so they didn't make it into a big issue."
1340
1341
1342weird
1343
1344Stephenson (guy in charge of writing report) goes to Oberg to confirm nav team asked for boost
1345
1346but then gives sketchy-sounding excuse for why it was denied "oh it didn't sound like a big deal"
1347
1348the report also did sound a little too much like making excuses for the section on why they didn't do a boost
1349
1350
1351 ... JPL director Stone endorsed this. "Even on the day of encounter it was not clear to anyone that we were on the wrong trajectory," he insisted. "No one."
1352
1353
1354but their own report says they figured out they were twice as close they should have been...
1355
1356
1357 ... This is the remaining inconsistency between NASA's official version of what happened and the one reconstructed by Spectrum. Our conclusion is that adequate doubts had been raised to require the TCM-5 burn, even in an emergency mode. Further, according to participants in this tragedy of errors, by the time the probe reached Mars, those most "in the know" were persuaded it was already doomed by its sick trajectory--but by then it was too late.
1358
1359 All this information has modified perceptions of the widely published images of glum space controllers waiting hopefully for the probe to regain contact after it passed behind Mars. They were not astonished, or caught by surprise by an anomaly whose cause they could only guess. They had seen the most recent tracking data, which spelled doom, and they were hoping not for the spacecraft to emerge but for a miracle, which never materialized.
1360
1361
1362basically Oberg is saying NASA is covering up their weird decision not to boost
1363
1364I wonder what a conspiracy theorist in 1999 would say... were the Washington Post and NY times covering up this decision too by ignoring it?
1365
1366to take a guess, the whole way this story was reported might have kept it out of the mainstream and pushed it into the realm of crackpots
1367
1368
1369 ... Even if what ruined the Mars Climate Observer mission can be overcome, it should not be forgotten. The analogies with the Challenger disaster are illuminating, as several direct participants in the flight have independently told Spectrum.
1370
1371 In that situation, managers chose to cling to assumptions of "goodness" even as engineers insisted the situation had strayed too far into untested conditions, too far "away from goodness." The engineers were challenged to "prove it ISN'T safe," when every dictum of sound flight safety teaches that safety is a quality that must be established--and reestablished under new conditions--by sound analysis of all hazards. "Take off your engineering hat and put on your management hat" was the advice given to one wavering worker, who eventually went along with the launch decision.
1372
1373
1374Oberg compares the failure to the challenger disaster
1375
1376
1377 ...In an analysis done by the spacecraft's builders at Lockheed-Martin, the spacecraft was almost certainly destroyed when its hydrazine propellant tank heated to the point of self-ignition of the remaining fuel. "There was enough explosive force there to level a city block," one engineer told Spectrum, so the spacecraft probably was blown apart into shards of scrap metal that soon burned up in the Martian atmosphere.
1378
1379 Whether the debris burned up, fell to the surface, or grazed the atmosphere fast enough to have passed out into space on the other side of Mars is not known. For now, it is truly "lost in space."
1380
1381 But if space mission operators learn their harsh lessons better, the exploration of Mars will push ahead. Sometime in the next century, or later, voyagers with far greater range and far sharper sensors than our own may find the observer's remains and come to know exactly how it died. It's enough, now, to know just why it died.
1382
1383
1384...
1385
1386if active measures, what is significance of this?
1387
1388Obergs report goes into detail on the major story the rest of the news missed, and comes off as a little conspiratorial. He alleges NASA is covering something up but hints they're covering up incompetence.
1389
1390With the 2011 Roscosmos failures, mainstream outlets would point to incompetence and corruption, while fake news sites would point to aliens, HAARP, secret bio weapons and coverups (e.g. the "aliens shot down phobos grunt" stories coming out right after they got telemetry)
1391
1392but this is different
1393
1394IEEE isn't exactly a fake news outlet, and Oberg is often battling conspiracy theorists, not agreeing with them (see https://dorkmission.blogspot.com/2011/12/review-of-ken-johnston-on-coast-to.html)
1395
1396
1397------
1398
1399space daily reporting on the decision not to boost:
1400
1401
1402http://www.spacedaily.com/news/mars-mco-99f.html
1403 JPL's assumed that a calibration table sent to them from MCO's manufacturer Lockheed Martin, and used in JPL's navigation calculations, listed the thrust of the spacecraft's small maneuvering thrusters in Newtons (less than 1/4 pound), when in reality it listed the thrust in pounds.
1404
1405 The result of the mix up was that JPL's navigation software miscalculated the spacecraft's position, and so JPL ended up inadvertently ordering MCO to carry out a final TCM (or Trajectory Correction Maneuver) eight days before its arrival at Mars that actually caused it to fly over Mars as low as 57 km. This was deep enough into the Martian atmosphere for air friction to impact the probe adverserly - causing it to either crash or left it drifting in Martian orbit as a charred and crippled wreck.
1406
1407
1408space daily says the final trajectory correction maneuver (TCM 4) caused the probe to enter the atmosphere too low
1409
1410aviation week and the first report say they started seeing much lower than expected altitude a few days after the TCM 4 on Sept 19, mars insertion sep 23
1411
1412the report says at least part of this is because Mars' gravity helped them get a more accurate measurement (how?)--they got this data sep 22, and incorporated it in their model to get insertion as low as 110km.
1413
1414But a few days before that, they were getting an altitude range of 150-170 km when they expected 226 km.
1415
1416Oberg says this was because as the probe approached mars, its increasing speed "helped refine the probes true path" and they started getting readings "lower and lower."
1417
1418was the TCM 4 bad?
1419
1420most reports say it would have been correct if they had the right data in the first place, and they had the wrong data because of the units problem.
1421
1422
1423...
1424
1425
1426here's the Oct 4 aviation week report :
1427
1428https://imgur.com/a/us5x5
1429
1430 Engineers have discovered that use of English instead of metric units in a navigation software table contributed to, if not caused, the loss of Mars Climate Orbiter during orbit injection on Sept. 23.
1431
1432 As of late last week, investigators had not determined whether the switch was enough to fully cause the 100-km. (62-mi.) lower-than-expected closest approach, but they believe it is the prime culprit (AW&ST Sept. 27, p. 24).
1433
1434 ...The table is used to remove the effect of thruster firings to reduce the residual error in the trajectory calculations. The calculations converge more rapidly when the initial residual error is small.
1435
1436 ...The Velocity change of these firings was tiny, a few millimeters/sec., said Sam W. Thurman, flight operations manager for the JPL Mars Surveyor operations project, which includes Mars Climate Orbiter (MCO). But they occurred more than 11 times per week for the nine-month cruise to Mars, and it can take several weeks of clean observations to determine a trajectory to the desired accuracy. Position in the sky is ideally resolved to 50 nanoradians, or 10 km. at the 200-million-km. distance to Mars.
1437
1438
1439This is saying the small forces model (the program with the untits problem) isn't actually necessary to figure out where the probe is, it just speeds up calculations of where the probe is.
1440
1441
1442https://books.google.com/books?id=sElVDQAAQBAJ&pg=PA29-IA113&lpg=PA29-IA113
1443 To navigate a spacecraft like CosmicSea to Mars, we need to know two things: how far the spaceship is from Earth and its location in space. NASA can locate spaceships that send a time-coded signal to at least two of the three radio telescopes on Earth at the same time. The Deep Space Network (DSN), as it's called, currently consists of three radio telescopes place approximately 120 degrees apart, or one-third of the way around Earth. This positioning ensures that there is always at least one telescope that can send and receive signals facing any point in space at any itme.
1444
1445 Using the Deep Space Network, we can compare how far a spaceship is from any two radio telescopes and any other known object in space, like a bright star or a nebula. Using the three locations--two telescopes and a star--a technique called triangulation helps us calculate a spaceship's location.
1446
1447"basics of space flight" from 2001:
1448
1449http://www.reading.ac.uk/physicsnet/units/1/ph1005/b/home/resources/Basics_of_spaceflight_01.pdf
1450 Since the Earth's own orbital parameters and inherent motions are well known, the measurements we make of the spacecraft's motion as seen from Earth can be converted into the sun-centered or heliocentric orbital parameters needed to describe the spacecraft's trajectory. The meaningful measurements we can make from Earth of the spacecraft's motion are:
1451
1452 its position in Earth's sky,
1453
1454 its distance or range from Earth, and
1455
1456 the component of its velocity that is directly toward or away from Earth.
1457
1458 Some spacecraft can also provide a fourth type of data,
1459
1460 optical navigation,
1461
1462 wherein the spacecraft uses its imaging instrument to view a target planet or body against the background stars. By repeatedly acquiring these three or four types of data, a mathematical model may be constructed and maintained describing the history of a spacecraft's location in three-dimensional space over time. The navigation history of a spacecraft is incorporated not only in planning its future maneuvers, but also in reconstructing its observations of a planet or body it encounters. This is an essential to constructing SAR (synthetic aperture radar) images, tracking the spacecraft's passage through planetary magnetospheres or rings, and interpreting imaging results. Another use of navigation data is the creation of predicts, which are data sets predicting locations in the sky and radio frequencies for the Deep Space Network, DSN to use in acquiring and tracking the spacecraft.
1463
1464 ...These techniques, combined with high-precision knowledge of DSN Station positions, a precise characterization of atmospheric refraction, and extremely stable frequency and timing references ( F&T , which is another one of the DSN data types), makes it possible for DSN to measure spacecraft velocities accurate to Basics of Space Flight Section II. Space Flight Projects http://www.jpl.nasa.gov/basics/bsf13-1.html (3 of 6) [4/5/2001 12:06:45 PM] within hundredths of a millimeter per second, and angular position on the sky to within 10 nano-radians.
1465
1466
1467I think 10 nano-radians translates to around 1 KM accuracy around mars... what's up with that?
1468
1469
1470https://www.scientificamerican.com/article/how-do-space-probes-navig/
1471 The final component of accurate navigation takes all of these other elements and, using models of the forces acting on a probe and orbital dynamics, estimates its location. By taking regular measurements over a period of time, a probe's position and velocity can be determined. For example, Cassini's location is typically determined to a kilometer or less relative to Saturn. Using a probe's known position and velocity, its future positions can be worked out. Navigators compare these positions to the predicted location of the target body--based on the ephemeris--to determine when a probe will reach its target. Then, all that's left to do is to collect the flyby data, take a deep breath, and go on to the next encounter.
1472
1473
1474if I read this right, I think this means the "small forces" program with the units bug wasn't required to track the probe's position.
1475
1476they can keep getting position data from range and doppler to DSN and refine its position from that (apparently accurately to 1 KM?)
1477
1478it wouldn't make sense if they needed to track every force on the probe to know where it is. What if solar flare/micrometeor hit it? alien poked it? or thruster burn uneven? There would be things they can't predict, and every measurement would have error anyway. So it can't be that error can just builds up indefinitely. It can't be that they try to calculate exactly where it would end up ahead of time. They would need to measure directly where the probe is and use that, even if that measurement has some built-in error.
1479
1480not sure, but I guess what they do is keep reading its estimated position over time from DSN and then refine its position using that data and knowledge of orbital mechanics, and that could be part of the "navigation model" the report talks about. By working that way, it would use what is known about the probe's previous position to better guess it's current one (like "dead reckoning" or kalman filter). I think what Aviation Week is saying is that the "small forces" program helps give it even more information about the relationship between its previous position and current one, which helps that model filter out error and converge on where the probe most likely is.
1481
1482If this is true, that would mean the error that a non-working small-forces model could introduce on the probes position is bounded. I couldn't be any worse than the error of a single measurement of the probe's position by the DSN, which apparently is already pretty good (but maybe not good enough?)
1483
1484Since the probe was firing its thrusters multiple times a day, basically it would mean that their data for where the probe is would be really noisy, and it would look like to the navigation team that the error filter based on the probes' trajectory history wasn't working well.
1485
1486
1487I think I get what the Aviation Week report is saying: they need weeks worth of "clean" position history (e.g. without any unusual outside forces hitting the probe, just known things like gravity from sun and planets) to cut out the error and get an accurate reading of the probe's position. But the design change to how the probe corrects its orientation made it fire its thrusters 11 times a week. So without accurately modelling these thrusters, they will never get enough "clean" history to accurately know the probe's position.
1488
1489Aviation week says the "position in the sky is ideally resolved to 50 nanoradians, or 10 km. at the 200-million-km. distance to Mars." So apparently the error they get on the probes' position was higher than that.
1490
1491
1492also predicting where it will end up would be more sensitive to small errors in velocity than position--they need to predict where the orbiter will end up when it hits mars. Thruster firings were on order of 1mm/s iirc. (1 mm/s * 1 week) = 0.6 KM. I guess they use position over time to estimate velcoity, on top of doppler readings and other things.
1493
1494
1495...
1496
1497might be what MCO's position error plot looked like:
1498
1499https://www.researchgate.net/publication/289647043_Early_navigation_results_for_the_new_horizons_mission_to_PlutoCharon
1500 Navigation modeling of 3AX mode is more complicated than in spin mode due to the thruster pulses used to maintain and change spacecraft orientation. This presents a special challenge for Navigation, as each paired thruster firing (coupled or otherwise) produces a small residual ∆V due to thruster inefficiency and misalignment. This residual ∆V is much more pronounced in 3AX mode, due to the constant thruster firing associated with deadbanding. Figure 5 shows 2-way Doppler residuals for the first month after launch with and without small force modeling for post-launch spacecraft events, including spin-up, spin-down, deadband clamps, etc. This is compelling evidence that even coupled thruster firings can result in non-trivial residual ∆V’s, in this case up to several millimeters per second. The plot on the right of Figure 5 shows post-fit residuals after estimating the residual ∆V’s.
1501
1502
1503https://trs.jpl.nasa.gov/bitstream/handle/2014/19492/98-0902.pdf?sequence=1&isAllowed=y
1504
1505this says mars '98 mission (including MPL and MCO) needed error in initial orbit error less than 20 km
1506
1507pg 5
1508
1509 The navigation strategy for both the orbiter and lander is based on that devised for the Mars Pathfinder mission [3]. In general, this involves collecting radio metric tracking data at regular intervals for flight path estimation, predicting the arrival conditions, and performing occasional midcourse maneuvers, when needed, to correct the flight path such that the required arrival conditions are achieved.
1510
1511 Tracking Data
1512
1513 Both vehicles will be tracked at regular intervals from ground stations of the Deep Space Network (DSN) located in California, Australia, and Spain. The DSN's 34-m antenna subnet will be used to perform two-way Doppler tracking and ranging, in conjunction with command and telemetry operations, at X-band frequences (7.2-8.4 Ghz) during cruise.
1514
1515
1516 ...Orbit Determination
1517
1518 The flight path estimation, or orbit determination, process involves estimating the spacecraft's flight path and a set of force model and measurement error model parameters from the tracking data. This is accomplished with JPL's multi-mission navigation software system, which employs a discrete sequential filtering algorithm for data reduction. The achievable accuracy depends upon data quantity and quality, observation geometry, and ultimately, the fidelity of the underlying models.
1519
1520 Parameters affecting the flight path include the initial state vector, solar radiation pressure model, planetary ephemeris and gravity modelling, and spacecraft, which produce measurable effects on the flight path given sufficient time. These perturbations must be modeled carefully in the estimation process, in order to achieve acceptable navigational accuracy. These momentum unloading events are expected to occur approximately once every seven days during cruise.
1521
1522
1523this is saying even if the AMD events were occuring at the expected rate, modeling them right would be necessary for "acceptable navigational accuracy."
1524
1525goes on to say they compare error in oribters trajectory to calculate insertion, and have a +/- 20 KM acceptable error for initial orbit. I think this is saying they would have been able to calculate the probability the final burn would have put the oribter in the right orbit, or at least known if there was uncertainty here.
1526
1527
1528 ...In Figure 10, the lander's 2-signma guidance error after TCM-4 is slightly larger than the 0.25 degree flight path angle error requirement. This indicates that, although there is a high (>90%) probability of a <0.25 degree error in flight paht angle after TCM-4, TCM-5 is still required as a contingency in order to provide the 95% probability confidence required by the mission. Figure 9 shows that the orbiter's 3-sigma guidance error after the fourth TCM is about 75% larger than the B-magnitude limits. This result implies a 91% probability of meeting the +/-20 km periapsis radius error, not eh 99% probability required by the mission. This is not a great concern for two reasons. First, the orbiter guidance analysis used a conservative error model for momentum desaturation, which contributes significantly to the overal guidance uncertainty. Additionally, these results do not assume that near-simultaneous tracking is used for the orbiter's approach. The project is planning to use this technique for the oribter (using MGS as the orbiting reference spacecraft) as a means to further rehearse for the navigation-related procedures required for the lander's approach.
1529
1530
1531They were planning ahead of time to use TCM-5 for the lander, and knew ahead of time that might need one for the orbiter. Makes the excuse in the report that they weren't prepared to do a TCM-5 even more strange.
1532
1533it says the error predicted on the orbiter's momentum near entry isn't a great concern since they were modeling small forces--woops!
1534
1535
1536 ...Summary
1537
1538 Pre-flight analysis of interplanetary guidance errors for the Mars Surveyor '98 orbiter and lander have been performed for mission design. Both vehicles will rely on Doppler and range radio metric data collected at regular intervals, as well as careful modelling of the forces acting on the spacecraft. Following the orbiter's fourth midcourse maneuver at Airrival-10 days, the 3-sigma uncertainty in capture orbit periapsis radius is expected to be 35 km. This is larger than the +/-20 km error requirement imposed by the project. However, it is anticipated that the error will be reduced with a more accurate error model for the momentum wheel desaturation maneuvers.
1539
1540
1541they knew the error was larger than target ahead of time even with AMD events once a week, and were planning to rely on the small forces model. This makes the decision to increase AMD events to 11 times a week and LMA being late getting the model working even more strange.
1542
1543article dated August 1998
1544
1545
1546so the small forces model was supposed to reduce the error on their tracking model
1547
1548they knew ahead of time that without it working they would have large values on error for trajectory that are not "acceptable"
1549
1550they knew right away that it wasn't working when LMA finally finished it--they were getting large errors on their trajectory
1551
1552and they knew they wouldn't have acceptable error bounds on the TCM-4 without the small forces working
1553
1554after the TCM-4, they knew their trajectory was too low as it was getting refined with better data
1555
1556all of this points to serious negligence somewhere in NASA--it wasn't a small error they missed that built up over time and snuck up on them, it was that they knew they couldn't track exactly where the probe was going the entire time
1557
1558and even when they got more refined data and finally figured out better where it was, someone decided not to boost the probe
1559
1560
1561...
1562
1563some reports give better idea of how it works, like Aviation Week and these:
1564
1565http://www.cnn.com/TECH/space/9911/10/orbiter.01/
1566 Months before the Climate Orbiter reached Mars, controllers at NASA's Jet Propulsion Laboratory saw signs of trouble but failed to understand the implications until it was too late.
1567
1568 "When the earlier trajectory work was being done it was noticed that the predictions … were not as exquisitely accurate as they normally are," said mission controller Ben Clark of Lockheed Martin Aeronautics.
1569
1570
1571https://spectrum.ieee.org/aerospace/robotic-exploration/why-the-mars-probe-went-off-course
1572 "Every momentum dump will introduce an additional bogus force into the navigation software," Spectrum's interplanetary navigation consultant explained. "Ultimately the curve fit will be warped by this constant application of a non-existent force."
1573
1574
1575NASA officials confused things more:
1576
1577http://www.cse.psu.edu/~gxt29/bug/localCopies/marsOrbiter.html
1578 Contractor takes blame for math goof that crashed Mars probe
1579
1580 Copyright © 1999 Nando Media
1581 Copyright © 1999 Associated Press
1582
1583 By PAUL RECER
1584
1585 ..."We were on the wrong trajectory and our system of checks and balances did not allow us to recognize that," Edward Stone, director of the Jet Propulsion Laboratory, said Wednesday. The NASA center in California was in charge of the Mars mission.
1586
1587 ..."Each time there was a burn (rocket firing) the error built up," said Stephenson.
1588
1589 As the spacecraft approached its rendezvous with Mars and the engineers prepared for a final rocket firing, there were indications that something was seriously wrong with the navigation, but no corrective action was taken, Stephenson said.
1590
1591 ..."The spacecraft did everything we asked of it," said Stephenson. He said the mathematical mismatch was "a little thing" that could have been easily fixed if it had been detected.
1592
1593 "Sometimes the little things can come back and really make a difference," he said.
1594
1595
1596------
1597
1598
1599Sam Thurston on oct 4 Aviation Week report says even with the small forces model not working, they only got "medium levels of jitter" until after the final burn:
1600
1601https://imgur.com/a/us5x5
1602 ...There also were four trajectory correction maneuvers (TCMs), ranging from 1.0 to 3.3 meters/sec. The effect of the thruster firings on navigation error is complex and still being studied, THurman said. "The table was a substantial contributer to estimation error" but it is not clear if it produced all the error that manifested itself in the final hours of the mission.
1603
1604 Compared with other missions, controllers observed a medium level of jitter in the measured cruise flight path. "It wasn't atypical," Thurman said. "Some missions are very smooth, and others are worse. We had stable nav for periods. Looking back, we may have been able to see signs of error, but didn't for lots of complex reasons."
1605
1606 Several days after the final 1.4-meter/sec TCM-4 on Sept. 15, the navigation calculations had relatively poor convergence. The orbit injected flyby was targeted at 224-km. altitude, and the new numbers were trending 150-180 km--but with uncertain confidence. The flight team considered using a preplanned TCM-5 to increase the altitude by 40 km. out of concern for the second periapsis, which would be 16 km lower.
1607
1608 Project management decided on Sept. 19 to forgo TCM-5, which was set for Sept. 20, reasoning that the risks of an added maneuver, such as a possible "safe mode" halt, outweighed the benefits of high altitude based on shaky navigation data.
1609
1610
1611but isn't the shaky navigation data the reason they wanted to boost in the first place?
1612
1613this report says the flight team requested the boost, but project management rejected
1614
1615
1616 The minimum survivable altitude was 85-98 km, which gave adequate margin to the predicted 150-180 km. Since the problem would likely be with the lower second periapsis, it would be easier to fix it with planned firing in the intervening 15-hr. orbit, Thurman said. From Sept. 18-21, the calculated periapsis remained at 150-180 km. with apparent +/-10-km error.
1617
1618
1619Thurmann says they didn't see convergence problems until a few days after TCM-4. When they did, the uncertainty was +/- 10 km.
1620
1621
1622 "The hard part is assessing navigation results and how much to believe them," Thurman said. "It only takes hours to run the data, but it takes much longer to assess and confirm it."
1623
1624 Up to the last day, Earth-based radio navigation data came from one condition--interplanetary cruise. But then Mars' gravity provided a second condition, measuring trajectory with respect to the planet. As gravity became stronger, this method rapidly become more precise--but with no time to do anything about it. At 6-8 hr. before the orbit injection firing, enough Mars-influenced Earth radio data had been collected to tell that the altitude was going to be a lethal 57 km.
1625
1626
1627why does the report say they didn't figure this out until after they found the units problem? Report says Mars gravity gave them an altitude of 110 km one hour before insertion, not 57 km 8 hours before.
1628
1629
1630 But it took time to process these data and, along with delay induced by a shift change, project managers were not aware of their predicament until an hour before orbit injection--too late except to sit and watch.
1631
1632
1633ok, it's saying the had enough data to figure out the insertion was 57 km 8 hours before insertion, but didn't finish processing until 1 hour before (why did report say 110 km?)
1634
1635the report also makes it sound like they wouldn't have been able to figure out it was 57 km without finding the units bug--is this true? by then they would have known they had all the info they needed from telemetry anyway.
1636
1637
1638 Mars Polar Lander (MPL) is in cruise for a Dec. 3 direct entry to the planet's surface, and officials want to know with confidence that hte problem has been fixed. MPL is also in the Surveyor project and will use navigation programs similar to MCO's. So far, the problem is with MCO spacefract data, but the 1-million-line program itself is being thoroughly scrubbed, Thurman said.
1639
1640
1641investigators were also looking at the nav program itself
1642
1643makes sense--if it was a problem with the AMD data only, why didn't it manifest strongly until after the fourth burn?
1644
1645
1646 ...Other questions being asked include:
1647
1648
1649 ...How the estimate of trajectory error remained low when it was in fact high. Was there a systematic bias that was undetectable from the single method of cruise navigation, that only became apparent when the second technique of Mars' gravity came into play?
1650
1651
1652
1653a lot of this doesn't make sense
1654
1655pre-flight analysis said they would not have acceptable error without the small forces model, even with burns once a week
1656
1657the small forces model wasn't working (aviation week says off by 22%) and was firing 11 times a week
1658
1659still, their estimates only saw "medium error" until a few days after the fourth burn (about a week before entry), when they got +/- 10 km error on altitude estimate. Pre-flight says they wouldn't be able to make +/- 20 km error bound without in part additional tracking info and small forces modelling. They are ~50 KM lower than they thought they would be, solutions not converging.
1660
1661By 6-8 hours in, they have enough data to know they're actually off by 100's of kilometers for altitude.
1662
1663
1664------
1665
1666according to space daily:
1667
1668http://www.spacedaily.com/news/mars-mco-99f.html
1669 There have been many such slips before: the missing hyphen that caused the booster rocket for the Mariner 1 Venus probe to veer off course; the erroneous temperature figure in a single manufacturer's memo that -- three years later -- led to the Apollo 13 disaster; the single misplaced decimal point that led to the loss of the billion-dollar Milstar satellite only six months ago.
1670
1671 In a technologically complex world, these things will happen sometimes.
1672
1673 And this seems to be the public impression that JPL and NASA are trying to leave of this accident. But over the past two weeks, it has become increasingly apparent that the loss of MCO was actually due to a bigger and less excusable mistake: namely the failure of MCO's mission control team to heed increasingly clear warnings - during the final days before its arrival at Mars -- that the craft might be seriously off course.
1674
1675
1676same deal as Oberg's report, alleging a NASA coverup of severe incompetence, NASA trying to cover up with PR spin
1677
1678which appears to be true comparing what happened in the failure and statements of some NASA officials
1679
1680
1681 Soon after the "mishap", SpaceDaily was contacted by an aerospace engineer who would prefer to remain anonymous, but whose contacts with JPL have provided him with ongoing reports as to the "true" story.
1682
1683 His conclusions have now been almost entirely confirmed by stories in the Oct. 4 "Aviation Week", and by an Oct. 5 report from science columnist Lee Dye on the ABC News Website.
1684
1685
1686Aviation Week might have been first to break story about the decision not to do final boost?
1687
1688kindof sounds like a tabloid in these paragraphs
1689
1690
1691 ...The erroneous operations table caused JPL's navigation software to underestimate the strength of MCO's small thrusters during its entire 10-month cruise to Mars, both during MCO's midcourse maneuvers and during hundreds of small thruster firings which it periodically made to allow reductions in the rotation speed of the motorized "reaction wheels" that MCO mostly used to control its attitude.
1692
1693 The errors were small -- especially since MCO carried out its bigger Trajectory Correction Maneuvers with its main rocket engine, which was not affected by the metric error. But they did add up, in a gradual way that was rather difficult to detect.
1694
1695
1696says errors added up (over what time period?) in gradual way
1697
1698
1699 ... Now --as you might expect -- precisely measuring the position of a spacecraft 200 million km from Earth is not easy. Instead of relying on a single radio tracking measurement to do so, you must average out the results of a whole series of tracking measurements -- often a week or more. Nevertheless, JPL expected to know MCO's actual Mars flyby altitude to within plus or minus 10 kilometers.
1700
1701
1702then says it averages out tracking measurements over a week
1703
1704
1705 ...The purpose of the thruster-strength data that was erroneous was to reduce the "residual error" in the trajectory calculations, allowing JPL's computers to converge somewhat more quickly on an accurate and consistent estimate of the spacecraft's position.
1706
1707 Under normal circumstances, JPL's computers should have been able to calculate the spacecraft's flyby altitude with an accuracy of 10 km within a few days after TCM-4.
1708
1709 Indeed, MCO's flight plan called in advance for a possible fifth and last Trajectory Correction Maneuver three days before Mars arrival if the ability to calculate MCO's true position indicated that the craft was dangerously off course.
1710
1711
1712this checks out
1713
1714
1715 However, within several days after TCM-4 the radio tracking data was indicating that the craft was 74-94 km off course -- far more than it should have been after such a maneuver -- and even those calculations of its path were far more uncertain than usual.
1716
1717 This should have alerted the flight control team that something was seriously amiss with their ability to calculate MCO's true position -- and, in fact, they did seriously consider carrying out TCM-5 in order to raise MCO's flyby altitude by an additional 40 km as a precaution.
1718
1719 But, to quote Aviation Week: "Project management decided on Sept. 19 to forego TCM-5 [the next day], reasoning that the risks of an added maneuver, such as a possible 'safe mode' halt [which was unlikely, but might have paralyzed the craft for a few days and kept it from carrying out its orbit insertion burn], outweighed the benefits of higher altitude based on shaky navigation data."
1720
1721 The reason for this decision was that, from Sept. 18 through Sept. 21, the "apparent" error in the periapsis calculations dropped back down to the usual 10 km, the calculated periapsis remained at 150-180 km, and "The minimum survivable altitude was 85-98 km, which gave adequate margin," quoted Aviation Week.
1722
1723
1724this says the error was high shortly after TCM-4 then "dropped down" to +/- 10 km.
1725
1726
1727 ...Two questions spring to mind. First, why didn't the flight managers pay more attention to that data shortly after TCM-4 showing both that the burn had been significantly off target, and that the trajectory calculations were unusually bad?
1728
1729 And secondly, why -- during the crucial period from Sept. 18 to 21 -- did the "apparent" accuracy of the trajectory estimates shoot back up to 10 km, although the actual error was 93-123 km?
1730
1731 (As an added oddity, JPL told Aviation Week that the uncertainties in its estimates of MCO's flyby altitude during this period were "plus-or-minus 10 km" -- but they then said in the same sentence that the total uncertainty was in a range of 30 km, not 20!)
1732
1733
1734yeah that's weird
1735
1736for most of cruise and sep 18-21, error was lower than it should have been given the small forces problems
1737
1738and the error estimates were wrong--it was actually off by 100's of km
1739
1740so it does point to systematic error
1741
1742why did the report conclude it was the units problem and not mention any of this?
1743
1744
1745 SpaceDaily's source says flatly that the only possible explanation for the last two statements is that, contrary to JPL's public statements, the "apparent accuracy" of plus or minus 10 km during this period wasn't derived from the actual radio tracking data, which -- if properly examined -- would clearly have shown that MCO's position was highly -- and dangerously -- uncertain.
1746
1747
1748maybe the nav software itself had a bug?
1749
1750if they have all that raw data somewhere shouldn't be too hard to find out
1751
1752
1753 Instead, it was based on a blind, overconfident assumption that MCO must really be on that accurate a course, simply because all of JPL's earlier missions had been that accurate. And this assumption was made in the very face of those earlier indications both that the craft had veered significantly off course during TCM-4 and that the tracking calculations were less accurate than usual.
1754
1755
1756space daily says managmeent were trusting this strangely-acting navigation software when there were signs they shouldn't have
1757
1758
1759related to this, Oberg reports:
1760
1761https://spectrum.ieee.org/aerospace/robotic-exploration/why-the-mars-probe-went-off-course
1762 In the following days, as additional tracking marks were accumulated, the navigation programs began to show the estimated closest point sinking lower and lower. It looked as if the probe was drifting off course, an impossibility if all the forces on it were properly modeled. What was really happening was that as the probe fell towards Mars, its increasing speed helped refine the probe's true path. But instead of believing that the newer values were closer to reality, controllers apparently chose to trust the earlier navigation, and suspect instead that something was now going wrong with the navigation software. The spacecraft itself, they assumed, remained on a safe trajectory.
1763
1764
1765which makes no sense. same report he cites says "From Sept. 18-21, the calculated periapsis remained at 150-180 km. with apparent +/-10-km error." then says they didn't figure out true altitude until too late.
1766
1767but both Oberg and Aviation weekly suggest some at NASA suspected the nav software itself had problems
1768
1769
1770http://www.spacedaily.com/news/mars-mco-99f.html
1771 In short, MCO was lost not just because of that one metric conversion error, but also because of significant overconfidence on the part of its project managers.
1772
1773 Indeed, Lee Dye's report for ABC News indicates that the current tumult within JPL is so great that some sort of major blunder must have occurred. Dye reports that there has actually been one instance of fisticuffs in the halls between a navigator and a senior engineer, and that frantic finger-pointing and mutual accusations are now filling the institution:
1774
1775 "The lab has been rife with rumors that the navigation team determined 10 days before the accident that the spacecraft was on a course that would take it too close to Mars, ample time to make a course adjustment. According to lab lore, the navigators are being blamed to save the brass, which failed to heed the warning, wrote Dye.
1776
1777 "Pat Esposito, the head of the navigation team, declined comment, saying he had been ordered not to speak with the press. According to the rumors, Esposito personally had given warning of the needed course change to Thomas Gavin, deputy director of JPL's space and earth science directorate, which is responsible for the Mars program.
1778
1779 "When I asked Gavin about that, I thought the telephone line was going to vaporize.
1780
1781 "'That never happened,' Gavin said. 'I would have come unglued. I would have called a session and said, "OK, what the hell is going on?" We would have done something,'" reported Dye in his ABC News report.
1782
1783
1784Lee Dye says nav team and project management were fighting over whether or not to boost, nav team knew it would crash
1785
1786so if all this true:
1787
1788- nav team knew it would crash
1789
1790- frantically tried to get project management to boost
1791
1792- managment comes up with lame excuses
1793
1794- decides not to boost without consulting everybody
1795
1796- puts gag order on speaking to press
1797
1798- denies everything while putting PR spin that blames nav team for missing a simple error
1799
1800
1801 ... In short, this was not a subtle or excusable error -- somebody's heads are going to roll at JPL, although it's not yet certain whose. To quote our source: "There are two possibilities: either the navigation team were wimps and didn't make a good case, or they made a good case but the management ignored them."
1802
1803
18041st report, led by Stephenson, tried to make case nav team weren't clear in communicating concerns
1805
18062nd report layed it on thicker
1807
1808so they're setting up a fight between project management and nav team--who's to blame for the crash?
1809
1810and reporting, official reports make it look like a cover-up by management
1811
1812are some media reports presenting it this way intentionally? NASA officials? the second MCO report was ridiculous, as were some quotes by project managers. Seems familiar--act like your covering things up, but doing a poor job of it (Kerensky did something similar with his handling of Rasputin or some royal scandal iirc)
1813
1814most mainstream reports didn't go into any of this intrigue at all, appeared to mostly blindly follow NASA's press statements.
1815
1816if active measures, I guess the effect of this would be that the general public would learn nothing more than "it crashed because NASA was careless with units," while the more conspiratorial-minded would see a cover-up, and blame NASA.
1817
1818Did they want crackpots to cry foul about coverups? not sure if fake news was the same in 1999. Maybe listen to talk show radio on it or something.
1819
1820The Oberg and Space daily reports allege a coverup in a way that doesn't sound completely crackpot-ish.
1821
1822assuming its active measures, recruiting conspirtards with this might not be the point--they'd simply realize they'd have a harder time fooling nerds and space enthusiasts with the same narratives as more mainstream publications. So they'd need a separate narrative for the nerds. Could hope to rely on nerd arrogance for them to suspect mainstream reports not disingenuous, but just not as smart as they are.
1823
1824
1825 ... He suspects the last, on the grounds that many of the flight managers were senior former navigation engineers themselves and had excessive confidence on the basis of JPL's earlier precision during their own tenure. Moreover, since the calculation errors resulting from the multitudes of earlier small thruster firings during MCO's flight to Mars were cumulative, JPL should have noticed some time earlier that the uncertainties in the calculations of its trajectory were increasing abnormally.
1826
1827
1828is this source reliable? the thruster firings were cumulative, but DSN was getting a read on its position and velocity at regular intervals, which would include any effects that had accumulated since last read.
1829
1830
1831 ... The final likely conclusion, though, is that MCO's $125 million demise was not so much an example of the disasters which a single subtle error can produce in our complicated technological world, but rather yet another demonstration of a far older principle that has bedeviled the human race from the beginning: pride goeth before a fall.
1832
1833
1834------
1835
1836http://publicism.info/science/wrong/11.html
1837 ...Unlike the Mars Global Surveyor, which had solar panels on either side of the spacecraft, the Mars Climate Orbiter had all of its panels on one side of the craft. Because of this asymmetrical design, the main effect of solar radiation was a tendency to spin the spacecraft around on its axis.
1838
1839 ...If these thruster firings had only affected the spacecraft’s rotation, they would not have disturbed its trajectory through space and would, therefore, have been irrelevant from a navigational standpoint. Unfortunately, design considerations led to the positioning of the thrusters in such a way that the AMD would also kick the spacecraft sideways by a small amount, and the frequent repetition of these events over the course of the flight would cause the spacecraft to deviate by a significant degree from its planned trajectory – enough to prevent the craft from entering the Mars orbit correctly.
1840
1841
1842not designed so well?
1843
1844
1845 ...To make this possible, the subcontractor who manufactured the thrusters sent paperwork to Lockheed Martin that documented how much thrust was generated when each thruster was fired. This manufacturer was accustomed to working in British units (pounds, feet and so on). NASA generally requires the use of metric units throughout its operations and those of its contractors, but it makes exceptions in cases where ordering a change of units may be unduly burdensome, or where it increases the risk that the contractor will make some kind of mistake. NASA made an exception of this kind for this subcontractor, so the paperwork received by Lockheed Martin listed the thrusters’ performance in units of pounds of force.
1846
1847 ...According to Jolly (who presumably would have been more knowledgeable about the matter), the engineer did know that pounds needed to be converted to newtons. The reason he failed to make the conversion, Jolly told me, had to do with the ‘heritage’ issue. The Mars Global Surveyor had used similar navigational software, and the plan was to save money by having the Climate Orbiter ‘inherit’ or reuse it. The new spacecraft used different thrusters, however, so the portion of the software relating to thruster performance was excised, and the engineer’s task was to replace that portion with code incorporating performance data for the new thrusters. Unfortunately, he assumed that the code that made the conversion of units was left in the unexcised Global Surveyor software, whereas in fact it was in the excised portion. The conversion was represented simply by the number 4.45 in an equation, without any comment as to its purpose, so it was easy to miss. Thus, in writing the new code, the engineer left the units in pounds, thinking that the required conversion would be made by the pre-existing software.
1848
1849
1850so which was it--NASA "making an excemption" for a subcontractor or the engineer having a mix-up with old MGS software?
1851
1852something off with this article
1853
1854it's abnormal to write software and not test it against spec, which included units in metric, and the onus would be on LMA to do that. but a lot of articles miss this
1855
1856
1857
1858 ...By a bitter irony, the spacecraft’s own computer did in fact possess software to make this calculation independently, and these files correctly specified the resulting thrust in newtons. ‘You can imagine how many times I wake up at night thinking about that,’ said Jolly. In fact, the spacecraft was even programmed to radio the output of these calculations to the ground, but the navigators did not know this so no one looked at the incoming data packets or compared them to the output of the erroneous calculations being performed on the ground. If they had done so, the error would have been quickly detected. Even when I spoke with him in 2006, after NASA’s official inquiry had established and published the fact, Jolly said that he didn’t know that the spacecraft had been transmitting the correct data to Earth.
1859
1860
1861but they did do this...
1862
1863for most of this I assumed they were looking specifically for problems in small forces model. Maybe whole time they were looking for something different? they way it behaved isn't consistent with a problem in the small forces model
1864
1865they did say they were looking at other things, nav software afterward
1866
1867it's weird the report and NASA officials pointed to this as THE problem when signs point to other things going on
1868
1869
1870 ...Teams at JPL and Lockheed Martin, led by JPL flight operations manager Sam Thurman, monitored and controlled the spacecraft during its journey. Part of the team was a group of four JPL navigators, led by Pat Esposito, whose task was to determine the spacecraft’s trajectory and calculate the required corrections during the flight. The team was also responsible for two other spacecraft, however – Mars Global Surveyor (which was orbiting Mars) and Mars Polar Lander (which was launched on January 3). Only one team member, Eric Graat, could give his undivided attention to the Mars Climate Orbiter. This was a low level of staffing compared with previous and subsequent missions: the successful Mars Odyssey mission of 2001, for example, boasted a 15-member navigation team. Although neither Esposito nor Graat agreed to speak with me, Sam Thurman told me that they were very much overworked.
1871
1872
1873but sam Thurman could have also told him all the stuff he got wrong in his article
1874
1875
1876 ...About every 17 hours during the flight, the spacecraft automatically performed angular momentum desaturation (AMD) procedures, firing its thrusters for a few seconds to allow the reaction wheels to be decelerated. The navigators had not been expecting the AMD events to occur so frequently, because when they came onto the job they were not familiar with the Orbiter and they did not realise that its asymmetrical design would cause an increased tendency to spin under the influence of solar radiation.
1877
1878
1879so would just reading the report...
1880
1881
1882 ...Finally, in mid-April, the ground software was delivered and put into operation. Now the effects of the AMD events (including those that had already taken place) were incorporated into the navigational calculations. But for each AMD event the software told the navigators that the spacecraft had been deflected by an amount that was nearly five times larger than what had actually occurred, thanks to the poison pill that was the units error. Still, each individual navigational solution looked good, because the error was in an unobservable dimension perpendicular to the line of sight.
1883
1884 Only over time, as more and more solutions were calculated along the spacecraft’s curving path, did Graat become aware that the individual solutions didn’t quite mesh together to form a coherent trajectory. And calculations of the spacecraft’s current position that were derived from different data sets (for example, those based on range or Doppler measurements, or those that were based on different parts of the spacecraft’s trajectory) gave a fuzzy cluster of solutions instead of a single, unanimous answer.
1885
1886
1887I think what he's saying is the error didn't show up until they took a bunch of readings and put it through their model
1888
1889
1890 Graat discussed this navigational problem with the leader of the navigational team, Pat Esposito. According to John Casani, the problem should have been entered as a formal written record known as an Incident, Surprise, Anomaly form – or ISA – which guarantees that a problem is followed up to a satisfactory resolution, but that’s not what happened. ‘The navigator here at JPL sent an email message to someone at Lockheed Martin, saying, “Take a look at this, there’s something funny going on that we don’t understand.†That never got entered into the formal record, which is our normal practice. So someone received this at the Lockheed end and said, “I’m going to work on this,†and then he got some other task that came along that either he thought was a higher priority, or his boss thought was a higher priority, and he got deflected, and this problem that was communicated to him by email just fell off the table, so to speak. If the form had been filled out, that could not have happened.’
1891
1892
1893this is weird
1894
1895I think this is saying Graat is the guy who found the model/telemetry data discrepency?
1896
1897and he brought it up with Pat Esposito, who supposedly called Gavin to boost
1898
1899in the end management blames Graat for not reporting it on formal channels
1900
1901the article likewise says not reporting on ISA the big problem
1902
1903that was also cited by the 2nd report as the "heart of the problem"
1904
1905at the same time, other reports say nav team and management are screaming at each other
1906
19071st report says nav team didn't "communicate clearly"
1908
1909this article maybe doesn't do the story justice
1910
1911
1912goes on to quote Thurman saying they were disctracted from this problem by other problems with the spacecraft
1913
1914
1915 ...Meanwhile the navigators worked almost non-stop to refine their trajectory calculations. As the days passed, the predicted altitude of the first periapse gradually decreased to about 150 kilometres – a safe altitude, but only if their prediction was correct to within a few tens of kilometres. During the final two or three days before arrival, the navigators got a serious case of cold feet, and they brought up the possibility of making yet another, fifth course correction to raise the spacecraft’s fly-by to a safer altitude.
1916
1917 This was the last thing that Sam Thurman, the flight operations manager, wanted to hear. The possibility of conducting a TCM-5 was written into the mission’s contingency plans. However, the optional TCM-5 was mainly planned for a different circumstance, namely for a situation in which the second periapse (on the first aerobraking orbit) would be at the wrong altitude. Reprogramming a TCM-5 to alter the first periapse altitude would be a major task for a relatively small spacecraft team, especially when it was so far behind on its preparations for the aerobraking phase.
1918
1919 ‘I remember that the nav team chief was nervous,’ Thurman told me. ‘He said, “I think we ought to bump up [the altitude],†and other people [were] saying, “If we do that, it’s going to screw up our aerobraking preparations.â€â€™ Thurman himself sided with the latter group. “[Doing a TCM-5] would have put at risk our ability to get into the correct science mapping orbit a few weeks later,’ he said. ‘It would have jeopardised our ability to transition people over to preparations for the Lander’s arrival. We needed the same people at Lockheed Martin to do that as well as to get aerobraking started.’
1920
1921
1922Thurman says there was argument over whether or not to boost
1923
1924says Esposito was nervous, navigators "cold feet"
1925
1926one camp (where?) said it was too hard? or would screw up aeorbraking some way
1927
1928another camp said ... article doesn't say there argument, presumably that they weren't sure high enough
1929
1930Thurman sided with first camp
1931
1932
1933 ...Given the lack of consensus, it was left to Thurman (perhaps in conjunction with other leaders of the mission) to make the decision, and they decided against a course correction. ‘I think that was an error of judgment,’ John Casani told me, ‘but that’s easy to say.’
1934
1935
1936article cites Thurman as the one who made the decision
1937
1938
1939 ...The group could follow the slowing of the spacecraft for the first four minutes of the planned 16-minute burn, but at 2:04 and 56 seconds the spacecraft’s carrier signal began to break up, and six seconds later it disappeared completely. This was the expected effect of the spacecraft passing behind the planet into its radio shadow. ‘At this time we are in our occultation period,’ announced Irish.
1940
1941 The spacecraft actually went into occultation 52 seconds before the event was expected. Less than a minute early – surely that tiny error could be of no significance after a nine-month voyage? Irish’s voice didn’t betray any surprise, and the NASA-TV commentators continued their chatter about the details of aerobraking. But to Thurman, it was a bad omen. He started staring intensely at a sheet of paper that he was holding in his left hand, then glancing up at the screen in front of him.
1942
1943 ‘I remember I had a plot next to me that our mission engineer had made that allowed us to correlate the time that the spacecraft headed behind the planet. He’d come up with a clever scheme where, by looking at the time of loss of signal, we could get a guesstimate of the actual altitude. The lower the altitude, the earlier the loss of signal. I had this very quick way of watching data on the screen, seeing the loss-of-signal time, and then looking at the data sheet for the guesstimate. I remember we had the signal – it was very early, and I remember thinking, Uh-oh, this is not good. That really – yeah, my anxiety level shot up, I tried to hold my composure since there were seven guys with cameras in front of me.’
1944
1945 The NASA-TV commentator eventually seemed to pick up on the early occultation, because without mentioning the fact that it was early he attempted to explain it away. ‘The signal can be refracted by the atmosphere,’ he said. ‘The accuracy is not as deterministic as we’d like.’ Meanwhile, the eight men in the glass booth sat or stood there, fidgeting, looking at one another’s screens, well aware that they could do nothing but wait 21 minutes for the scheduled re-emergence of the spacecraft from occultation. By that time the spacecraft should have terminated its burn and turned so that its signals could be picked up by the 70-metre Tidbinbilla deep space antenna near Canberra, Australia.
1946
1947 At 2:26am, the predicted time for the spacecraft to re-emerge from occultation, there was dead silence in the control room, and even the television commentator stopped talking. The silence went on for minutes, while the men in the glass booth became increasingly fidgety, staring at their watches or at the computer screens, standing up and sitting down again for no apparent reason, or glancing into one another’s tired, anxious faces. Sam Thurman kept adjusting his wedding ring, as if its precise alignment was crucial for the spacecraft’s destiny. ‘Waiting for acquisition of signal,’ said the disembodied voice of Kelly Irish. And a minute later: ‘Still haven’t seen anything; stand by.’
1948
1949
1950heh
1951
1952is the author enjoying this?
1953
1954
1955 After a few minutes, Thurman stood up and began to talk with the other members of the JPL team, including the Mars ’98 project manager, Richard Cook. His words weren’t audible on the TV broadcast, so I asked him whether he had been telling them that the spacecraft was probably lost. ‘I don’t say things like that,’ he said. ‘That can be devastating to a team’s morale. You always have to hope for the best and do everything you can to make it come about. And there’s a little bit of lore called the 24-hour rule, which is, when something seemingly bad or scary happens, don’t overreact for a minimum of 24 hours, because more often than not what actually happened and what you need to do about it might be different from what you think at first. So I tried to reach inside and gather the presence to say – getting emotional doesn’t serve any purpose, so you have to say, “Here are the options. The vehicle survived and may be in safe mode; it might be tumbling; it might have ended up in a much lower or higher orbit than expected because of an overburn or an underburn.†Remember, the ground stations need fairly accurate predictions of what the transmitter frequency is going to be in order to tune their receivers properly to hear a spacecraft 120 million miles away, orbiting another planet. So it could be the spacecraft was there and transmitting fine, and the set of predicts we used to drive the antennas were off.’
1956
1957
1958why is Thurman trying to explain away his reaction
1959
1960
1961 About 30 minutes later, Richard Cook came out of the glass booth and spoke with NASA-TV. He outlined the possible factors that might have caused the spacecraft to go into safe mode. ‘At this point, we’re still very confident that we’re in orbit at Mars,’ he said, ‘and we’re going to see the spacecraft signal sometime in the next few hours.’ With that, NASA wrapped up its live television coverage for the night.
1962
1963
1964Richard Cook again
1965
1966
1967 ...JPL’s John Casani was quickly appointed to head an internal investigation of the mishap. There was considerable time pressure, because the Orbiter’s companion spacecraft, Mars Polar Lander, was fast approaching the planet and was due to arrive on December 3. There were many similarities between the two spacecraft, and investigators wanted to ensure that whatever doomed the Orbiter would not also affect the Lander.
1968
1969 All the focus was on the navigational problems. Every piece of navigational software was scrutinised line by line, and on September 29 an engineer identified the crucial error: the lack of a conversion factor to change pounds of force to newtons in the Small Forces software. Once that error was identified, all the problems with navigation were readily explained.
1970
1971 To my knowledge, Casani’s report only circulated internally and was never published. ‘In my subjective opinion, it focused too much on all the technical details of what didn’t get done, or didn’t get done well enough,’ commented Thurman, ‘and too little on how the lab got itself in that position to begin with.’ That deficiency was quickly made up for by a second investigation, headed by Art Stephenson, Director of NASA’s Marshall Space Flight Center in Alabama. While agreeing with Casani that the units problem was the root cause of the mishap, Stephenson’s report put far more emphasis on the numerous contributing factors – inadequate training, testing and communication; the failure to resolve the anomalous navigational solutions or to report the problem through the proper channels; the failure to execute TCM-5, and so on – that together amounted to a systems-engineering or even programmatic failure. While not exactly criticising Goldin’s ‘Faster, Better, Cheaper’ philosophy, Stephenson urged that it be practiced under a set of guidelines that he summarised as ‘Mission Success First.’
1972
1973
1974Casani did a report that wasn't released
1975
1976replaced with one (Stephenson report) more in line with what "blame FBC and management problems" narrative
1977
1978why wasn't casani's report released?
1979
1980
1981 ...By the time the Lander reached Mars, it looked like all the kinks had been ironed out, but after it began its descent through the atmosphere nothing more was heard from it. Once again, Richard Cook was delegated to give the waiting press an upbeat assessment of the situation. ‘I’m very confident the Lander survived the descent,’ he said, five hours after it should have been broadcasting its first images.
1982
1983
1984Richard Cook again
1985
1986
1987 ...In any event, Lockheed Martin Astronautics went through a tough period after the Mars ’98 failures. The company suffered financial losses, NASA cancelled a contract for a follow-up mission and many astronautical engineers left the company, at least temporarily.
1988
1989
1990LMA roiled after failures, despite blame-shifting in press
1991
1992
1993 ...Although Lockheed Martin lost some employees after the Mars ’98 failure, the hapless young engineer who made the units error was not fired; in fact, he’s still with the company. ‘He has a lead position; he’s in the critical path for all the flying missions that we have,’ said Jolly. ‘You know, that’s the noble way to do it. Engineers do not walk in and say, “I’m going to make a mistake today.â€â€™
1994
1995
1996hmmm
1997
1998
1999
2000this isn't some quick and dirty article (WaPo: "smaller, better, faster" not working, NASA scientists forget to convert units!)
2001
2002this guy is trying to explain the whole thing in detail
2003
2004but softens or ignores a lot of the weird things about this incident
2005
2006also brings some up
2007
2008
2009
2010------
2011
2012
2013preliminary flight analysis says error unnacceptable without small forces modeling, even with firing once a week.
2014
2015Stephenson says they were manually inputting small forces into model correctly for first half of trip
2016
2017Thurman says "medium jitter," "not atypical," "stable nav for periods", didn't see alarming error (problems with "convergence") until after TCM-4
2018
2019Thurman also reportedly the one to decide not to boost, said nav team had "cold feet" and esposito "nervous" after 4th burn
2020
2021report and other sources say they knew something was wrong as soon as the small forces model was running
2022
2023space daily source says error fluctuating, spiked after TCM-4 then stablized to +/- 10 km Sep 18-21, Stephenson report likewise gives 20 km range for that period
2024
2025true error was actually 196 km
2026
2027
2028this points to a problem with the nav software more generally, and not just the small forces model
2029
2030unmodeled forces would show up as large error on the trajectory, but this didn't happen.
2031
2032report also says range and doppler were giving different solutions, and they couldn't resolve this discrepency, pointing to problems beyond unmodelled perturbations, and investigators were reviewing the software.
2033
2034
2035------
2036
2037
2038this goes over the behavior of the nav software as the probe approached mars:
2039
2040
2041http://web.mit.edu/16.070/www/readings/Failures_MCO_MPL.pdf
2042
2043 ...Trajectory Correction Maneuver 1
2044
2045 ...TCM 1 was unnaffected by the small forces anomaly due to the small number of desats since launch, the different radiometric observation agnles associated with early cruise, as well as the fact that the AMD files were not being used at this time by the Navigation team.
2046
2047 ...Trajectory Correction Maneuver 2
2048
2049 ...the quick look burn reconstruction did indicate execution well within the 1 sigma B-plane error ellipse.
2050
2051
2052First two TCM's were good since they didn't use the flawed AMD data
2053
2054
2055 ...Trajectory Correction Maneuver 3 (MOI- 59 days)
2056
2057 TCM 3 was the turning point for the Flight Team in that observable data began to show that something was amiss in the design and execution of the maneuver. Up to this point the Navigation team had been using estimates of the small forces delta-V as discussed before -- without the flawed SPAS-generated data. But finally all the format issues had been resolved, the packet consolidation S/W tested and in place, and on or about April 19, 1999 the SPAS data was available for the Navigation Team. In fact, the Flight Team ran all the previous small forces packets through the system and delivered them to the file server. So the design of this burn used the flawed data.
2058
2059
2060by third burn, they had replaced previous calculations with the new but flawed data
2061
2062
2063 The Mars Climate Orbiter spacecraft successfully performed TCM 3 on July 27, 1999 which was 59 days from MOA. ...All execution errors were well within expectations.
2064
2065 However, following the slew to the earth-point communications attitude, the spacecraft did experience a safe mode entry while commanding the solar array two-axis gimbal to articulare the array out of its passive restraint.
2066
2067
2068they did a burn that succesfully obeyed commands, but afterwards the hinge on the solar panel got stuck (or somethig like that)
2069
2070
2071 In addition, the quick look data following the burn, using several hours of tracking was much worse than results from the previous two TCM's - but was attributed to too short a time of tracking and was expected to improve. The data indicated the actual trajectory was well outside the 1 sigma target ellipse, and in fact, was "below" the Mars horizon. So even at this early time before MOI, the orbit determiniation (OD) process was indicating solutions biased toward a low altitude at closest approach.
2072
2073
2074a quick check on the new trajectory (using hours intead of weeks of tracking data) was worse than previous burns. The trajectory put it crashing into Mars.
2075
2076 Unfortunately, the anomaly that resulted in the safe mode entry completely diestracted the small MCO flight team, including scarse ACS personnel resources, and because of the flurry of activity associated with the safe mode entry and the potential threat to MOA and Aerobraking, the Flight Team and Navigation Team failed to ascertain the importance of this "symptom." For the next 6 weeks the spacecraft team was engaged in troubleshooting the anomaly and replanning MOA and Aerobraking in case the spacecraft were to suffer again the same anomaly. No less than 3 flight troubleshooting exercises were performed, preceded by heavy test laboratory preparation and data review. The final passive restraint exercise was successfully executed on Friday, September 3 as planned.
2077
2078
2079the team was distracted by fixing the hinge and planning for what to do if it happened again. Someone scheduled 3 exercises to prepare for this, which didn't finish until September 3.
2080
2081
2082 Using this data the team was able to map a trajectory into and out of the restraint to hopefully avoid the anomalous condition, and was now ready for TCM 4 and MOI.
2083
2084
2085they plotted a new trajectory to use in case it happened again
2086
2087
2088 Trajectory Correction Maneuver 4 (MOI - 8 days)
2089
2090 TCM 4 was successfully completed on September 15, 1999. A 1.3742 m/s delta V was recorded by the spaceraft's IMU against a target of 1.3711 m/s. All spacecraft indications were nominal. Overall pointing error (RSS of x and y) was 27.2 mrad, well within requirements.
2091
2092 The solar array articulation was nominal. The revised restrain/unrestrain trajectory was used, along with the new gain values that were demonstrated in the recent flight verifications. The team felt they were a "go" for MOI and Aerobraking. (Mitigation strategies were still in place onboard the spacecraft, to prevent mission loss in case the array got stuck in, or near the passive restraint.)
2093
2094
2095they used some new parameters from their recent exercises
2096
2097
2098 But the initial Navigation tracking (based on only 2 hours) indicated an underburn, showing only 1.33 m/s. This initial trajectory estimate showed that MCO may be out of its 3 sigma burn target ellipse.
2099
2100
2101quick look for TCM-4 showed it was outside of 3-sigma (99.7%) error estimates
2102
2103quick look for first two were on target, 3rd was outside 1 sigma, and 4th outside 3-sigma
2104
2105
2106 This would correspond to a post-MOA periapsis (P2) altitude of 137 km periapsis instead of the target of 210 km. MIA periapsis altitude (P1) was 16 km higher. However, over the next 24 hours, the NAV team reported that the solution showed that the periapsis altitude was 172 km.
2107
2108
2109the true trajectory was 57 km altitude
2110
2111the quick look right after TCM 4 showed 137 km
2112
2113a longer calculation after 24 hours showed 172 km.
2114
2115
2116 Daily telecons continued between the Navigation team and the Spacecraft team as they monitored the situation. It was agreed that if TCM5 was necessary, the decision to execute it would be made on Sunday, September 20 (1430 MDT). But to make the decision to execute, required that the TCM5 products by fully tested and reviewed. So the team decided to design the TCM5 using the same burn attitude as did TCM 4. The magnitude was changed to 0.23 m/s (corresponding to about a 30-50 km increase in periapsis altitude), and the epoch updated for a Monday, September 20 execution. All products were fully tested in the STL on Friday night, with the full data review to follow on Saturday.
2117
2118
2119The nav team got into daily calls with the spacecraft team (LMA I think?) while monitoring situation. They planned on possible TCM 5 for Sep 20. They tested out how to do it Friday and had a review scheduled for saturday.
2120
2121
2122 The Navigation and Flight Team continued to monitor the predicated periapsis altitude for MOI resulting from TCM4. Because of the observation geometry from Earth to the spacecraft's velocity vector it was talking longer to confirm the resulting trajectory and make sure it was high enough ton insure that periapsis would be outside the Martian atmosphere.
2123
2124
2125they kept trying to get better measurements on how high it would enter, but solutions took longer due angle the probe was going
2126
2127
2128 Trajectory Correction Maneuver 5 and Post-MOA Options Developed and Tested
2129
2130 MOI - 6 days: Friday, September 17, 1999
2131
2132 It was reported that tracking solutions were beginning to cluster, the post-MOI P2 mean had climbed to 164 km with the 3 sigma low at about 140.
2133
2134
21352 days later, they get tracking solutions converging on 164 km with 140 km lowest.
2136
2137
2138 Late in the day as plans were firmed up for how to test TCM5 and the resulting MOI wihch follows, the Flight Team began to raise the possibility of reducing risk to MOI by not performing TCM5, but instead preparing plans for post-MOI that would alleviate MCO from going into Safe mode during its first post-MOI periapsis passage (referred to as P2 since P1 is MOI) if it were to encounter aerodynamic torques. ...These discussions were held because the data indicated that a TCM5 was not strictly required.
2139
2140
2141they firmed up plans for the TCM 5, but the Flight Team raised that it might be better to do TCM 5 after the first loop around Mars, since the new solutions predicted that a TCM5 was not strictly needed.
2142
2143
2144 ...Finally, a decision was made to hold a Saturday morning special telecon with management and the MCO standing review board to outline the TCM 5 risks and options. It was expected that 24 more hours of Nav tracking would give more confidence in the decision to be made the following morning.
2145
2146
2147I guess the argued about it and agreed to go to management Saturday to "outline risks and options." By some reports the nav team wanted to make a case for TCM 5, by others they didn't make their case strongly enough.
2148
2149They thought 24 more hours of tracking data would give them more confidence to make the decision on Sunday.
2150
2151
2152 MOI - 5 days: Saturday, September 18, 1999
2153
2154 The latest solutions were reported as 173 +/- 18 km (p1, 3 sigma). TCM5 was outlined and discussed but the decision was to proceed with options for post-MOA as described above
2155
2156
2157the new solution with extra data showed even tighter bounds, 173 +/- 18 km 3 sigma
2158
2159
2160 ...The team was to proceed with TCM5 until results from the 12 Noon Navigation team meeting could be assessed. Following the Noon meeting, at 2:30 PM the solutions were reported from range measurements to indicate a mean of 161 km and a tight variance of no lower than 150 km (3 sigma low), but solutions using doppler data were reported to be all over the map. The worst single solution for the doppler data was 140 km and was felt by the Flight Team to be survivable with margin. (The data did not indicate as of yet the true altitude of periapsis, but the data was indicative of a problem of some kind.) TCM5 was then called off and the Flight Team was refocused on building and testing the post-MOI options a and b discussed in the telecon.
2161
2162
2163they had a TCM5 planned, but after looking at solutions decided to cancel
2164
2165range solutions were 161 with with 150 km 3 sigma low, but doppler "all over the map," lowest was 140 km which Flight Team felt was survivable with margin. The data indicated a problem, but they decided to cancel TCM5 anyway.
2166
2167
2168 MOI - 2 days: Tuesday, September 21, 1999
2169
2170 The decision criteria for using the post-MOA options was presented and approved at the 9AM Mission Design Meeting. At that time it was agreed that MCO should perform the OTM (option b) for sure at the at the first Apoasis ...On this date the best solutions were indicated to be 161 +/- 11 KM. By the time that Thursday, September 23 arrived, the predictions were still about 150KM.
2171
2172
21733 days later solutions still showing tight bounds, 161 +/- 11 KM.
2174
2175
2176 Mars Orbit Injection: September 23, 1999
2177
2178 The final flight products for the Mars Climate Orbiter were successfully uplinked on Friday, September 17 and the injection sequence was initiated. The sequence remained in a holding pattern (as planned) until 17:00:44 UTC (ERT) on 9/22/99 when, as planned, it issued a series of commands to disable most of the spacecraft's fault protection. The remainder of the countdown to 9/23/99 went smoothly with zero alarms.
2179
2180 All telemetry and ground data indicate that on board Master Sequence and MOI Block were executing properly at Mars occultation.
2181
2182 ...During the countdown to MOI ignition, the Flight Team received word from the JPL that navigation solutions were now showing a post-MOI periapsis of only 110 km agains the prediction earlier that day of 150 km. The Flight Team immediately began altering the existing post-MOI apoapsis Orbital Trim Maneuver (OTM-1) to doube the firing time so an orbit raise of the post-MOI periapsis of +80 km could be achieved. (This burn would be loaded following MOI and executed at the first apoapsis.)
2183
2184
2185Everything going fine, then JPL calls of flight team and says navigation solutions now showing 110 km altitude. The flight team immediately entered a command that would boost the orbit after the first loop around Mars (??).
2186
2187Sam Thurman was head of flight team
2188
2189
2190 ...At 9:00:46 UTC (ERT) the Main Engine was ignited on time and JPL reported that the doppler residual indicated the burn was taking place. All indications were that the spacecraft was in control and holding the burn attitude while the main engine was firing.
2191
2192 At 9:05:02 UTC (ERT), 39 seconds earlier than predicted, the spacecraft carrier dropped out completely as apparently Mars occultation had begun. Occultation was expected to end at 9:26:25 (using the actual occultation time) as reported by JPL. Unfortunately, the signal never appeared again.
2193
2194
2195everything else was working until signal cut out 39 seconds early, never appeared again
2196
2197
2198 While the Flight Team continued to build the post-MOI burn predict during the occultation period, the navigation solutions continued to drop until finally reaching 57 +/- 1 km for MOI (P1) and 40 km for post-MOI periapsis (P2).
2199
2200 It was now apparent that the navigation predictions prior to approaching Mars apparently had an error about 100 km at MOI closest-approach from where the spacecraft actually was, at closest approach. As the spacecraft began to more strongly experience the Martian gravity, the Navigation team was able to calculate more accurately the MOI closest-approach distance. The most confident calculations were the last ones before occultation, resulting in the estimated 57 km altitude.
2201
2202
2203while computing the trajectory with data up to the loss, the Flight Team found the signal dropping lower and lower until reaching 57 +/- 1 km for initial entry altitude. The Navigation team were able to use Mars gravity somehow to get more accurate solutions.
2204
2205
2206 Thermal subsystem predictions show that the spacecraft would mos likely have exceeded critial temperature limits at about 98 km. Attitude control authority would be exceeded at about 85 km altitude.
2207
2208
220998 km was probably low enough for it to burn up and stop working
2210
2211goes on to say they tried to blind fire boost commands but they never saw a trace of the probe
2212
2213
2214...
2215
2216Thurman lead flight team and made decision not to boost. The nav and flight team disagreed. nov 10 Report is telling:
2217
2218ftp://ftp.hq.nasa.gov/pub/pao/reports/1999/MCO_report.pdf
2219 In the wake of the MCO loss and the subsequent augmentation of the MPL team, the Board observed that roles and responsibilities of some individuals in MSOP are unclear. A recurring theme in the Board’s deliberations was one of “Who’s in charge?†Another such recurring theme was one of “Who’s the mission manager?†The Board perceived hesitancy and wavering on the part of people attempting to answer this question. One answer was that the Flight Operations Manager (FOM) was acting like a mission manager, but is not actually designated as such.
2220
2221 ...The MCO project did not have a defined set of Go–No Go criteria for using TCM-5. There was no process in place to review the evaluation and decision criteria by the project and subsystem engineers before commitment to TCM-5. Polling of the team by the MCO Flight Operations Manager should establish a clear commitment from each subsystem lead that he or she has reviewed the appropriate data and believes that the spacecraft is in the proper configuration for the event.
2222
2223
2224the report above says:
2225
2226 Late in the day as plans were firmed up for how to test TCM5 and the resulting MOI wihch follows, the Flight Team began to raise the possibility of reducing risk to MOI by not performing TCM5, but instead preparing plans for post-MOI that would alleviate MCO from going into Safe mode during its first post-MOI
2227
2228 ... The latest solutions were reported as 173 +/- 18 km (p1, 3 sigma). TCM5 was outlined and discussed but the decision was to proceed with options for post-MOA as described above
2229
2230
2231and then says they had plans for a TCM-5 anyway, after the first loop around Mars
2232
2233
2234but the nov 10 report says:
2235
2236 A request to perform a TCM-5 was discussed verbally shortly before the MOI onboard procedure was initiated, but was never executed.
2237
2238 Several concerns prevented the operations team from implementing TCM-5. Analysis, tests, and procedures to commit to a TCM-5 in the event of a safety issue were not completed, nor attempted. Therefore, the operations team was not prepared for such a maneuver. Also, TCM-5 was not executed because the MOI maneuver timeline onboard the spacecraft took priority. This onboard procedure did not allow time for the upload, execution, and navigation verification of such a maneuver. Additionally, any change to the baselined orbit scenario could have exceeded the time for the MCO aerobraking phase when MCO was needed to support the communications of the MPL spacecraft. The criticality to perform TCM-5 was not fully understood by the spacecraft operations or operations navigation personnel.
2239
2240
2241it wasn't just a "request"--they had a plan and were getting ready to execute, until they were overruled by Thurman.
2242
2243
2244...
2245
2246
2247recap:
2248
2249they knew something was wrong when they plugged in the small forces data around the time of the 3rd burn. Unlike previous burns, a quick look at the trajectory after third burn exceeded 1 sigma error and plotted the craft crashing into Mars.
2250
2251But they didn't have much time to investigate--something was wrong with the solar panel, which the team fixed in 6 weeks, and then spent the remainder of the time until 4th burn doing "no fewer than 3" exercises and replotting in the event the problem happened again.
2252
2253A quick look at the fourth burn put it outside 3 sigma error and entering atmosphere at 137 km instead of 226 km. A series of calculations with more data then showed it at 173 km and 164 km with bounds around +/- 10. The nav team was monitoring the situation and calling the spacecraft team daily.
2254
2255A TCM 5 boost as planned and prearations were made. But the flight team, headed by Sam Thurman, pointed to the latest nav solution and argued a TCM-5 wasn't needed. Thurman overruled the plans to do a TCM-5 amid concerns from the nav team.
2256
2257As the probe approached Mars, the nav team contacted the flight team with a new estimate that the probe would enter 110 km high. The flight team immediately entered commands to boost the oribt but these would be executed after the first loop (it might have been that they didn't have enought time to boost the orbit beforehand).
2258
2259
2260This report is dated Feb 4 2001
2261
2262
2263------
2264
2265
2266Here is another report from JPL dated Nov 11, 1999 that I somehow missed this entire time:
2267
2268
2269https://trs.jpl.nasa.gov/bitstream/handle/2014/38186/D-18441.pdf?sequence=1
2270
2271pg 6
2272
2273 ...error was revealed when, on September 29, 1999, Lockheed-Martin Astronautics (LMA) reported that the files containing the magnitudes of the small forces impulses applied to the spacecraft had been delivered in English units (pounds-force seconds) instead of metric units (Newton-seconds). The interface agreements specify metric units.
2274
2275
2276I like this report better
2277
2278
2279 ...Over the 9-month mission, the cumulative effect of these twice-a-day impulses would have produced an error at Mars of more than 10,000 km. Most of this error was removed by the four trajectory correction maneuvers (TCMs) performed during cruise to Mars. After the fourth and final TCM, performed on September 15, 1999, the remaining effect of the small forces error was 169 km. Post-failure analysis has shown that the spacecraft's minimum altitude would have been 57 km. The 169-km difference between this prediction and the actual first periapsis altitude represents a major discrepancy in the mission's navigation.
2280
2281
2282the TCMs were correcting for some of the error, which would have been more than 10k km otherwise
2283
2284goes on to say they should have caught it before launch, during launch, and mitigated its effects
2285
2286chair of this review was Dr. Frank Jordan
2287
2288pg 8
2289
2290investigated
2291
2292 - Failure of the spacecraft while it was occulted from view.
2293 - Failure of the Deep Space Network (DSN).
2294 - The possibility that the spacecraft entered the Martian atmosphere and was destroyed.
2295
2296
2297concluded that it entered too low due to the small forces modelling error.
2298
2299they found three causes:
2300
2301 - The spacecraft was on the wrong trajectory,
2302 - The fourth trajectory correction maneuver (TCM-4) was wrong,
2303 - The contingency maneuver (TCM-5) was not performed.
2304
2305 These are the reasons why the MCO entered the Mars atmosphere.
2306
2307
2308Why couldn't the other report be this straightforward?
2309
2310
2311pg 10
2312
2313 ...The Spacecraft was on the Wrong Trajectory
2314
2315 The MCO was on the wrong trajectory because the MCO navigation process did not correctly account for the small forces acting on the spacecraft.
2316
2317 Two separate but related factors contributed to the error in small forces modeling:
2318
2319 - The omission of the English-to-metric conversion factor int he software program used to generate the AMD files.
2320
2321 - The LMA software program used to generate the AMD file also incorrectly formatted the data on the files, rendering the files unreadable by the navigation software. The format problem was not corrected until mid-April 1999, with the result that the small forces inconsistency was not uncovered until more than four months after launch.
2322
2323 ...The anomalous results yielded by the navigation process promted the Navigation Team and the Spacecraft Team to investigate the sources of the problem. Contrary to established practice, the investigation was not carried to completion. For example, the telemetry data included an onboard calculation of the delta-V resulting from the small forces; had this data been analyzed, the small forces units error would have been quickly detected.
2324
2325
2326says they dropped the ball on investigating it
2327
2328
2329 Without explicit knowledge of the reason for the discrepency, the MCO navigation analyses should have yielded significantly larger uncertainties for its estimate of the target point. This did not happen. As a result, the wrong targeting uncertainties contributed to the other two causative factors
2330
2331
2332nav analysis should have had larger uncertainties--is this something that should have been in the computer model already, or added by hand?
2333
2334
2335 ...TCM-4 Was Wrong
2336
2337 ...TCM-4 was wrong because the MCO navigation analysis yielded the wrong trajectory and because the aim point chosen for TCM-4 was too low, given the true uncertainty in the knowledge of the trajectory.
2338
2339
2340this makes sense--if they are predicting the small forces wrong, they can't predict how to aim it. the 2001 report said the small forces data was used after TCM-3.
2341
2342They also aimed it wrong given the "true uncertainty"
2343
2344
2345 ...TCM-5 Not Executed
2346
2347 The third causative factor was the decision not to execute TCM-5. The mission plan called for the execution of a fifth contingency maneuver (TCM-5) in case of trajectory errors following TCM-4. However, this maneuver was never executed. In spite of anomalous orbit determination results following TCM-4, the mission managers decided not to execute TCM-5, the risk-reduction contingency maneuver. Again, a major factor in this decision was project management's lack of knowledge of the true trajectory uncertainty. The other major factor in deciding not to execute TCM-5 was project management's decision to change the baseline risk-management strategy in the days immediately preceding Mars orbit insertion.
2348
2349
2350mission plan called for TCM-5 in case of errors
2351
2352got errors
2353
2354management decides not to do TCM-5
2355
2356changed "risk-mangement stratengy" in days before mars insertion
2357
2358major factor was their lack of knowledge of true uncertainty in orbit
2359
2360
2361goes on to say found problem with training and software dev process
2362
2363 ...Contrary to typical practice, there was not a requirement for a navigation (end-user) representative to be present at any of the walkthroughs or the acceptance test.
2364
2365 The Sm_forces software program was misclassified as non-mission critical, which reduced the number of reviews done on the software compared to mission critical software.
2366
2367 ...there were no minutes of the walkthrough meetings, action items from the review were not published, and the interface with navigation software was not tested.
2368
2369
2370goes on to say should have been considered mission critical
2371
2372
2373 ...The Failure REport on AMD file format problem was not generated until on April 6, 1999. The problem was observed December 12, 1998. No ISA was written.
2374
2375
2376
2377 ...- The navigation Plan was not updated to accurately reflect the desaturation frequency or delta-V magnitudes (see section 4.12).
2378
2379 As a consequence of an inadequate navigation performance plan ... the navigation Team was surprised after launch. However ...the project did not undertake to redo the navigation analysis, thereby leading to continued underestimation of trajectory uncertainties.
2380
2381
2382I think this was saying they were supposed to re-do navigation analyisis when they learned the magnitude and frequency of AMD events weren't what they thought, but didn't for some reason.
2383
2384
2385 The MCO project did not pursue either the small forces discrepency issue or the orbit determination issues to a point that would have forced resolution. The relationship between the small forces and orbit determination problem was recognized. MCO navigation and project management believed that the risk management strategy (raising the aim point as necessary to keep P2 above 150 km 3 sigma (see section 4.13)) would compensate for orbit determination uncertainties.
2386
2387 Project management did not seek an outside peer review of the orbit determination problem (see section 4.9). Instead, a single navigation consultant was commissioned. The consultant focused on data quality issues.
2388
2389 A hastily organized meeting of a small subset of the MSOP Standing Review Board convened on September 18, 1999, to discuss the need for TCM-5. Contrary to the typical practices of other flight projtects, the Navigation Team was not represented at this meeting. The utilization of a small subset of the Operational Readiness Review Board to make recommendations on the operational decisions without all viewpoints is innappropriate.
2390
2391 MCO project management did not communicate to the Navigation Team a major change in the TCM-5 go/no-go decision strategy. Specifically, the Go/No-Go meeting scheduled for Sunday, September 19, 1999 was superceded by the provisional decision made Saturday morning described above. The project management, in consultation with the Spacecraft Team and the attending members of the Review Board, made a provisional decision not to proceed with TCM-5 provided the navigation results were substantially the same as those of the previous day. The Navigation Team was not informed of his change in decision-making strategy.
2392
2393
2394...
2395
2396
2397 ...The Navigation team identified errors in the small forces impulse data. During May and June there were several informal e-mail messages back and forth between the Navigation Team and the Spacecraft Team in an attempt to resolve the errors.
2398
2399 However, the investigation was never completed because it was not considered mission critical and many "higher priority tasks" were determined to exist. The MCO project did not escalate the issue with LMA.
2400
2401
2402...
2403
2404
2405 ...Navigation Process
2406
2407 Findings
2408
2409 The MCO navigation process did not correctly identify the actual trajectory. Orbit determination solutions exhibited a large spread and were changing in a manner that strongly suggested modeling errors in the process. The source of modeling errors consistent with the solution spread was not identified.
2410
2411 The MCO navigation process did not correctly bound the uncertainties in the trajectory. As a consequence, the MCO mission managers did not understand the magnitude of the uncertainty in their knowledge of the trajectory, which, subsequently, led to incorrect decisions regarding TCM-4 and TCM-5.
2412
2413 In investigating the trajectory error, the MCO Navigation Team did not comply with the Orbit Determination Operational Procedure, which requires that a peer review be conducted to help select eh best solution for subsequent action.
2414
2415 ...No one person had end-to-end responsiblity for the navigation process
2416
2417 ...Reccomendations
2418
2419 ...must provide further guidance in the procedure regaridng the strategies for additional compute runs that should be carried out prior to initiating a peer review and must clarify the procedure. Specifically, the procedure should be changed to make it unambiguously clear that peer reviews must comply with the Laboratory policy on Reviews and that peer reviewers must be knowledgeable experts external to the project.
2420
2421
2422did someone show them a bogus before TCM-4 and TCM-5 without telling anyone else?
2423
2424
2425 ...Team-to-Team Communication and Understanding
2426
2427 ...The anatomy of the MCO mishap showed many instances where simple communications between people were not followed to their conclusions.
2428
2429...
2430
2431
2432 ...Contrary to typical practice, the MSOP Navigation Team did not ...re-do the navigation accuracy analysis, either in preparing to take over the mission or after the desaturation frequency was revealed to be two times per day, instead of once per week.
2433
2434 Spacecraft design and trade studies were undertaken during development without the benefit of input, review, and assesment from the Navigation and Fligth Mechanics Section. Examples include, the choice of unbalanced solar array and uncoupled desaturation thrusters, the choice of thrust direction to perform desaturation maneuvers, and the introduction of the "barbecue mode" and the subsequent dropping of this mode.
2435
2436
2437...
2438
2439
2440 ...The mission plan called for TCM-5 two days before Mars Orbit Insertion (MOI), if needed. However, TCM-5 had not been included in the missio operations timeline. Specifically, TCM-5 had not been tested for compatibility with a running MOI sequence during the couse of implementin gand preparing for MOI. Neither was TCM-5 part of the Operation Readiness Testing. As a consequence, when the time came to execute TCm-5, it became a major risk decision for the project to make at a critical point in time. The project waived the requirement of 150 km (3 sigma) altitude at P2 withotu following its established change mangemenge procedures.
2441
2442
2443didn't prepare for TCM-5 before mission, even though plan called for it if needed. Conditions were that it was needed, but this was waived.
2444
2445pg 33 lists team members for this review
2446
2447Appendix 1 says not modelling small forces, which built up a lot of systematic error, cause dthem to aim TCM 4 wrong. They aimed it so that the probe crashed into the Martian atmosphere.
2448
2449Appendix 2 on how their nav software works
2450
2451 ...The errors were sufficient to result i a trajectory displacement of approximately 169 km in where the spacecraft would pass Mars at its closets approach. Unfortunately, the consequences of this total error were in the direction closer to Mars, causing the spacecraft to take too deep a cut through the Martian atmosphere and, thereby, fail to survive.
2452
2453
2454...
2455
2456 On April 14, the Navigation Team Leader visited LMA to provide a tutorial on non-gravitational forces and emphasized the improtance of non-gravitation force modeling on trajectory reconstruction
2457
2458 ...On April 26, the Navigation Team sent an e-mail ... the e-mail also indicated that in several AMDs that had occured while the spacecraft was being tracked, the Doppler measured value of the AMD component along the Earth direction was several times larger than the value from the AMD file. The navigation Team asked the Spacecraft Team to look into the problem. No ISA was written at JPL or at LMA.
2459
2460 After receiving the Navigation Team memo, the Spacecraft Team initiated the investigation into the discrepency in AMD delta-V magnitude. The Spacecraft Team, on May 6, 1999, sent an e-mail to the Navigation Team indicating that the delta-V discrepencies would be investigated and requested more data. The Spacecraft Team indicated that it would compare the value of the Doppler-derived deta-V with the value calculated in the flight software and sent to the ground in the telemetry stream. On May 10, 1999, the Navigation Team delivered plots of the data to the Spacecraft Team. On May 12, 1999, the Navigation Team Leader visited LMA to conduct aerobraking discussions regarding the MCO. While the Navigation Team Leader was at LMA, he had a side meetin gwith some of the Spacecraft Team members to discuss the velocity differences epxressed in the Navigation Team memo. On May 14, the Spacecraft Team sent an e-mail to the Navigation Team acknowledging the receipt of the plots and asking for the data in tabulr form and in spacecraft event time. On May 21, the Navigation Team sent the requested data in tabular form. This was the last written communication between JPL and LMA on this subject until after the mission failure. Subsequently, the Navigation Team made a couple of telephone calls to the Spacecraft Team regarding the status of the investigation and was told they were still working on it. In this time frame, the Navigation Team Leader asked project management to escalate the priority of the AMD delta-V magnitude investigation at LMA.
2461
2462 Project management declined to escalate the priority. Instead, project management told the Navigation Team:
2463
2464 - That the Navigation Team needed to do the best they could with the data they had.
2465
2466 - That what the project really needed was the Navigation Team's best estimate of the uncertainty in the trajctory.
2467
2468 One June 9, 1999, the Attitude Control Group (ACG) at LMA was reoganized. ...The new MCO point of contact was apparently not told of the MCO AMD task.
2469
2470 ...In June, the estimates of the trajectory following TCM-2 had begun to change significantly. Believing that the velocity discrepency investigation would yield no new information because they had heard nothing from LMA, the Navigation Team began to solve for the components of the AMD maneuvers.
2471
2472 ...By early August, after TCM-3, the orbit solutiions, projected to periapsis altitude, dropped lower. In addition, the solutions using range-only data, Doppler-only data, and range-plus-Doppler data exhibited a spread far greater than the uncertainty in any of the individual solutiosn. The Navigation Team informed project management that the periapsis target altitude was too low given rpboelms they were having with fitting the AMD data and with the scatter in the solutions. By late August, numerous meetings relative to the scatter in the solutions were held between project management, the Navigation Team, and a consultant. However, ther was apparently no linking of the scatter in the solutions with the AMD velocity discrepency and, in fact, the AMD problem may not have even surfaced at those meetings.
2473
2474
2475...
2476
2477
2478 ...At a prior meeting on August 30, 1999, as a result of concerns regarding the orbit solution spread, the Navigation Team requested a TCM-4 aiming point corresponding to a P2 altitude of 250 km. Project management asked the Navigation Team to run more cases and report the results at the TCM-4 target-selection meeting scheduled for September 2, 1999. At the September 2 meeting, as a result of a Navigation Team's 3-sigma error quote of 60 km, a final P2 altitude of 210 km (p1 of 226 km) was chosen by project management. To allay concerns, the Navigation Team was told that TCM-5 could always be used to raise the orbit if such an action appeared warranted.
2479
2480
2481 ...The navigation Team again requested that project management furhter raise P2 to 250 km on September 13, because of the volatility of OD solutions. This time Navigation acquisced to not changing the targetting on the condition that there would be a TCM-5 and that the MOI maneuver would be modified as mentioned above. There was a misunderstanding between Navigation and project management regarding whether this condition was actually agreed to.
2482
2483...
2484
2485
2486 ...Based upon 2 hours of post TCM-4 tracking, the estimated P2 altitude was 138 km ...At or shortyl after this became known, it was decided to undertake the building of TCM-5
2487
2488 ...On September 16, 1999, the altitude estimate for P2 had moved back up to the region of 150 to 170 km. A TCM-5 delta-V magnitude was chosen that would raise P2 back up to the value of 210 km targetted at TCM-4 ...The date scheduled for TCM-5 was September 20, 1999.
2489
2490 A nominal MOI sequence had been sent to the spacecraft earlier as a precautionary measure in case the final upload could not be sent. On September 17, 1999, the project decided to delete the planned final MOI sequence load and to continue with the nominal sequence. Navigation as not aware of this decision until September 19, 1999.
2491
2492
2493...
2494
2495
2496 ...On Saturday morning September 18, 199, at the request of the former MSP98 Program Manager (also a board member), a meeting was held to inform the Board of the possibility of executing a TCM-5 contingency maneuver and to obtain the Board's advice. Present at this meeting either in person or by phone were a limited number (3 of 8) of the Mars Surveyor Operations Standing Review Board members. Navigation personnel were not invited or among the attendees.
2497
2498
2499management schedules meeting for "advice" day after deleting TCM-5 burn plan from probe
2500
2501only 3 of 8 board in meeting
2502
2503nav team not invited
2504
2505
2506 The Spacecraft Team had run the TCM-5 simulation in the STL on Friday mnight and felt that it would be safe to execute as proposed provided that the subsystem review was successfully completed. However, the project's position consited of noting that the P2 altitude was still safe and that it would be risky to perform a TCM-5 during the same time period as a clocking-out MOI sequence. The tentative decision was made that there should be no TCM-5 unless the navigation solutions later in the day showed a significant problem.
2507
2508
2509argue TCM-5 too risky with MOI sequence they had put on day before
2510
2511
2512 ...The Navigation OD solution available at the September 18 afternoon status meeting did not indicate any significant change in the altitude estimates. THe results of the morning meeting were not discussed at the afternoon status meeting.
2513
2514 At the scheduled TCM-5 Go/No-Go meeting on Sunday September 19, 1999, the project announced that the no-go decision had been made the day before. Navigation accepted the decision not to perform TCM-5 after project management told Navigation on September 20, 1999, that the decision reflected the unanimous consensus of the Board.
2515
2516
2517
2518lied to nav team, said decision was unanimous and had board had already decided to no-go TCM-5 day before
2519
2520
2521 ...No TCM-5 was performed on September 20, 1999. Had TCM-5 been performed and raised the P1 altitude from its subsequently determined value, of 57 km to perhaps 100 km, it is possible that the MCO spacecraft would have survived.
2522
2523
2524
2525
2526------
2527
2528
2529https://www.popularmechanics.com/space/moon-mars/news/a28632/the-dumb-mistake-that-doomed-a-mars-probe-in-1999/
2530
2531 The Math Mistake That Doomed a Mars Probe in 1999
2532
2533 Nobody's trying to make an idiotic error in space. But sometimes it happens.
2534 By David Grossman
2535 Oct 13, 2017
2536
2537 ...The core problem involved measuring in the Imperial system instead of the metric system—had been ongoing for nearly 10 months, yet had been undetected. Such a simple error, essentially a failure of translation, was traced to contractor Lockheed Martin Astronautics in Colorado. Using pounds when the NASA-JPL team was using the internationally recognized metric system proved to be a fatal mistake, sending the MCO to crash the surface of the Red Planet.
2538
2539
2540http://content.time.com/time/specials/packages/article/0,28804,1982672_1982673_1982667,00.html
2541 Mars Climate Orbiter
2542 By Kayla WebleyMonday, Apr. 19, 2010
2543
2544 NASA scientists are ridiculously smart, but even geniuses have bad days. Sept. 23, 1999, proved to be a particularly bad — and expensive — duh moment for NASA when a Mars orbiter crashed into the Red Planet's surface because of a simple conversion error: one engineering team had used metric units while another team had used feet and inches. After nearly 10 months in space, the navigation mishap pushed the Mars Climate Orbiter too close to the planet's surface. The crash effectively ended the mission — designed to help scientists study the history of water on Mars — on the day NASA had hoped to celebrate the craft's successful entry into Mars' orbit. NASA's associate administrator for space science said in a written statement at the time, "People sometimes make errors." This elementary mistake cost $125 million.
2545
2546
2547------
2548
2549https://www.wired.com/2010/11/1110mars-climate-observer-report/
2550 Nov. 10, 1999: Metric Math Mistake Muffed Mars Meteorology Mission
2551
2552 Strange Places on Mars: What Do You Want to See Next?The $125 million satellite was supposed to be the first weather observer on another world. But as it approached the red planet to slip into a stable orbit Sept. 23, the orbiter vanished. Scientists realized quickly it was gone for good.
2553
2554 "It was pretty clear that morning, within half-an-hour, that the spacecraft had more or less hit the top of the atmosphere and burned up," recalled NASA engineer Richard Cook, who was project manager for Mars exploration projects at the time.
2555
2556
2557thought Cook was saying it could still be saved hours later
2558
2559
2560 A NASA review board found that the problem was in the software controlling the orbiter's thrusters. The software calculated the force the thrusters needed to exert in pounds of force. A separate piece of software took in the data assuming it was in the metric unit: newtons.
2561
2562 "The units thing has become the lore, the example in every kid's textbook from that point on," Cook said. "Everyone was amazed we didn't catch it."
2563
2564 ...The whole thing could be written off as a miscommunication. Propulsion engineers, like those at Lockheed Martin who built the craft, typically express force in pounds, but it was standard practice to convert to newtons for space missions. One pound of force is about 4.45 newtons. Engineers at NASA's Jet Propulsion Lab assumed the conversion had been made, and didn't check.
2565
2566 But there was an underlying issue in the culture of NASA's space exploration at the time, Cook said.
2567
2568 "'Better, faster, cheaper' was the mantra at the time," Cook said. "Certainly that project was trying to do a whole lot for a limited amount of money."
2569
2570 The New York Times ran a preview article Sept. 21 titled "Beginning a Bargain-Basement Invasion of Mars," foreshadowing the disasters to come.
2571
2572
2573this article 20 years later is more or less same narrative as first reported
2574
2575NASA missed obvious error, blame cutting corners
2576
2577
2578 ...Cook said NASA made some "big-time" changes after that. Several planned missions, including a mission that was to bring Mars rocks back to Earth, were scrapped. The space agency went back to basics, rebuilding its Mars program based on conservative strategies and concepts that had already been tested.