On rocket science and software testing

I recently read “Firing a Rocket” by James R. French. Mr. French is an engineer who was involved in the early American space program, working on developing engines and combustion devices used in the Saturn 5 rocket and Apollo space missions. He served as the Lead Development Test Engineer on the Lunar Module Descent Engine. The book is fascinating because most accounts of the early days of space exploration come from the famous astronauts and VIP’s. I found it very interesting to hear an account from a grunt in the trenches.

My favorite chapter in the book was the chapter about the testing of the F-1 engine. While I’m not a rocket scientist (or rocket engineer to be more precise), I found some of the lessons learned listed by Mr. French applied to the world I work in, software development and implementation.

1. “Think about how the device will be operated in real life and test it the same way.” The author tells the story how when the liquid oxygen pump on the F-1 was developed, each component was tested individually. In the individual tests, the start-up sequence was abrupt, but when the pieces were assembled, the start-up sequence was slower, which resulted in oxygen pump explosions. Similarly, with software deployments, if you test each subprocess individually but don’t test the end-to-end process in a real-world manner, you may pass the system test and wind up with a system that doesn’t work in the real world. An example of this is memory intensive processes or over-engineered forms that work in a clean room, but when used on real user’s computers with many other applications running, perform slowly.
2. “One should not let prior experience blind you to a problem, especially if you are operating outside the range of parameters in which the experience was obtained.” The author talks about how as rocket technology advanced, engineers with experience didn’t catch issues that had “never been an issue before” in other contexts. Experience is a good thing, but experience that blinds you to things that you don’t know is a bad thing.  In the Microsoft Business Applications world that I work in, things are rapidly changing, and things that were right yesterday are wrong today. This is where someone entering the workforce now may have an advantage over someone who has been working with Microsoft technology for 20 years, as it may be easier for them to learn the new way without having to overcome all of the things that they used to know.  Setting aside technology change, this can also be an issue as you work in different industries. If you develop and deploy software for a specific industry, like banking, you may come to see the world as a bank sees it. If you assume that a retail or manufacturing client needs the same solution that banks do, you will fail. The answer for “old timers” is to surround yourself with reminders that you don’t know everything. Seek feedback from others in your industry, and stay hungry to never stop learning.

There are key differences between rocket engineering and software development and deployment. For one, if the application test fails, things don’t usually literally explode, but there is a lot that we can learn from other industries. Like rocket engineering, many of the failures have nothing to do with the technology–they are caused by miscalculations and user error. By testing in real-world scenarios and being curious and open to new ideas, most of these issues can be identified and resolved.