“Small Changes Matter”

As an update to previous coverage on this blog of the Falcon 9 grounding written by Seth Haroldson, the Falcon 9 has resumed launches. On July 11, the vehicle was temporarily grounded following an attempted launch of 20 Starlink satellites. During that launch, a pressure sensor line had cracked, and the resulting excessive cooling and liquid oxygen leak disrupted the ignition fluid needed for the Merlin engine’s second burn. The ultimate outcome was that the desired orbit was not reached, and the satellites were stranded at an altitude that was too low for them to survive (Foust, 2024b). This anomaly comes with a massive price tag, as the updated V2 satellites cost $800,000 each to build, for a total of $16 million for the loss of 20 such spacecraft (Erwin, 2024). This figure does not include damage to the Falcon 9, specifically the Merlin engine, nor does it include any of the costs associated with investigating the recent anomaly and protecting against its recurrence.

To SpaceX’s credit, NASA characterized the company’s investigation and communication of its findings as “very transparent,” which is essential to restoring the Falcon 9’s launch readiness and broader confidence and trust in SpaceX as a launch provider (Foust, 2024b). Additionally, SpaceX appears to have cooperated with an FAA investigation ensuring that there were no risks to the public from the launch anomaly; the FAA was required to verify this before any clearance to launch again could be given to SpaceX (Foust, 2024a). However, Seth correctly noted in his initial coverage that anomalies like this can happen to even the largest and most successful space companies, and having options across the PACE (primary, alternate, contingency, and emergency) model is crucial. The two week delay already represents risks to time-sensitive launches and missions, as he outlined, and it likely at least interrupted the schedule for Starlink. While Starlink is an internal mission that would launch with SpaceX regardless, third parties such as NASA and the Department of Defense also largely rely on the Falcon 9 and other SpaceX technology. If those third parties lack viable alternative launch providers, rather than accepting indefinite grounding or schedule slip, they may create pressure on SpaceX to cut corners in its investigations or accelerate a return to launch.

While the failure in this instance was mechanical, as the root cause of the crack was a combination of fatigue and an ineffective clamp, similar anomalies could easily originate from cyber threats (Foust, 2024b). For instance, if an attacker could gain access to a sensor or line leading to or from the sensor, they could disrupt the contents of the data, such as pressure readings in this case, or they could set the frequency to an inappropriate value to cause either a denial-of-service attack from constantly pushing readings or block access to crucial data entirely. Similarly, if the attacker was instead able to gain control of a valve or other actuator relating to fuel, liquid oxygen, or the igniter fluid, it would be possible to cause leaks, failures to ignite, or even explosions by changing the timing or amount of propellant release. With so many sensors and actuators aboard one Falcon 9 in addition to the number of organizations likely involved in the supply chain for those parts, a bad actor would have a variety of options of cyber attacks that would create seemingly mechanical failures. 

Fortunately for SpaceX and every organization depending on them, the pressure sensor that failed earlier this month was determined to be largely irrelevant; its data was not needed, and it was not part of any flight safety systems. As a result, the short-term fix to allow a return to launch involved removing the sensor and line entirely (Foust, 2024a; Foust, 2024b). However, there are multiple situations in which this approach would not suffice. If the sensor were part of a crucial system, such as flight safety, either an alternative would need to be designed, built, and tested, representing a sizable time and energy expenditure and likely an extended delay before returning to launch. Similarly, the scale of the negative impact would likely change, particularly if crew members were on board and a failure jeopardized their safety. In any case, the decision to remove or replace a sensor should be made with great care. To aid in such evaluations, existing designs and configurations should be reviewed regularly to identify possible cyber risks early and to ensure that parts that are no longer used or relevant are removed. These reviews will allow mitigation of risk before a failure and reduce needless exposure to risk from parts that are not actively used anyway.

Steve Stich, NASA commercial crew program manager, offered valuable insight on this situation by explaining that “small changes matter.” A couple lines of code, a clamp, a pressure  sensor, a sense line, and more can all cause failures far beyond their role within the system (Foust, 2024b). Even a seemingly inconsequential change must be carefully examined for various forms of risk ranging from mechanical failures to safety concerns to cyber threats. SpaceX’s stellar track record in recent launches leading up to this anomaly may have led to complacency regarding changes that appear minor. While some level of confidence based on the company’s previous results is warranted, this incident serves as a sobering reminder that the tiny details and intricacies of a launch vehicle are not as small as they may seem with respect to risk.

Works Cited:

Erwin, S. (2024, May 9). Starlink soars: SpaceX’s satellite internet surprises analysts with $6.6 billion revenue projection. SpaceNews. https://spacenews.com/starlink-soars-spacexs-satellite-internet-surprises-analysts-with-6-6-billion-revenue-projection/ 

Foust, J. (2024a, July 26). Falcon 9 cleared to resume launches. SpaceNews. https://spacenews.com/falcon-9-cleared-to-resume-launches/ 

Foust, J. (2024b, July 27). Falcon 9 returns to flight with Starlink Launch. SpaceNews. https://spacenews.com/falcon-9-returns-to-flight-with-starlink-launch/