Plumbers Are Worth Every Penny

By Avi Deitcher

There is a common pattern I see over and over again in fast-growth companies:

  1. Release the minimum product to get some traction. GOOD
  2. Get traction and revenue and grow. GOOD
  3. Focus on feature functionality to grow quickly, deferring "maintenance work." NOT SO GOOD
  4. Product begins to crack under the scale for which it wasn't designed. BAD
  5. Scream for help. NECESSARY

Every edifice requires basic plumbing work - at the outset and on an ongoing basis. It is true for a house; it is true for an apartment building; it is true for an office tower; and it is true for a company.

Poor plumbing will lead to leaks, and eventually risk the entire building.

This is precisely what happened to Knight Capital. Last week, the SEC filed an 18-page complaint - fascinating and required reading for anyone in the finance or technology business.

Knight has an automated trading system. In the summer of 2012, Knight deployed a new version of their software, and made two assumptions:

  1. The software had been properly deployed to all 8 of their servers.
  2. An old unused indicator called a "flag" (like a checkbox on a paper form) could be reused now that the software was deployed.

Both of these proved erroneous. The software had been copied manually to only 7 of the 8 servers, which meant that the flag would trigger massive unintended purchases and sales.

The SEC complaint states that the problem was Knight's risk controls, and in some ways they were right: having a second technician review the deployment probably would have caught the issue.

But at heart, the problem was poor plumbing. Knight did not invest in automation and testing so that turning the new software on or reusing the old flag became impossible unless everything was safe. It is highly likely Knight was reluctant to invest precious capital in "non-feature-functionality" technology, like testing and automated deployments, that would have no immediate revenue impact.

In short, it didn't want plumbers, it wanted fancy new toilets.

Plumbers are expensive, but they are far cheaper than exploded pipes that can bring down your building.