Technology Manifesto, Part 3

A few years back, I (Zack) was working as a Sr. Website Administrator for Amazon.com. It was one of those exciting yet disruptive jobs, where as a group of elite Unix gurus we handled the troubleshooting and resolution of large scale outages of the primary website. The team was very solid and I was happy to be in such good company, but the job came with quite a downside…pager duty.

Not just any pager duty, but $20,000/minute pager duty. That was how much money Amazon was losing per minute of outage back then, so the conference calls were particularly emotionally charged. Good times.

I can remember one evening rather vividly. It was 3am and I had just fallen asleep after resolving a few other pages, when off goes the pager again. Buzzzzz…if you’ve ever been responsible for a pager before, you know what I mean. Even for several months after I transitioned off that job, I still had phantom pager buzzes waking me up in the middle of the night. But, this was a real page, 3am, and I got out of bed without destroying the pager, so I was off to a good start.

I called in to the number on the pager and joined the conference call already in progress. Guess what? The whole site was down, the main database was about to wander off into the weeds, and the ringing of the cash register had come completely to a halt. Amazon.com was down.

Believe it or not, this happened alot back in those days, so my blood pressure and adrenaline were at normal levels and with the excellent co-workers on the line, we managed to track down the root cause of the crash. Complex multi-threading issue in the database server? Nope. Super high-tech part get interference from cosmic rays? Nope.

In fact, it was all because of a typo in a Excel document.

You see, the Technology supporting the setting of a price was so difficult to use that no one bothered with it anymore and had instead built a workaround where they would create a list of prices in Excel and then just upload it into the annoying Pricing program. Excel munged the price, it went to the site incorrectly, and hordes of people tried to buy the drastically reduced but very popular item at it’s pennies on the dollar price. Everything crashed under the load.

This leads us to the second corollary of Principle 1: Ministry will be more effective when there are fewer problems to work around.

Anytime the weird and obscure ways of Technology force ministry to travel the side streets to get to their intended destination, the less ministry will get done. Just like the coffers at Amazon slammed shut with a website malfunction, so too can the reach of the Gospel be cut short when our systems go offline.

Reliability, redundancy, fault-tolerance, three-nines (a reference to the 99.9% of the time we want our websites to be up)…these are all words in the Christian Technologist’s repertoire. If the light in the lighthouse is off, ships will wreck on the shores.