Liar, Liar, Samsung on Fire

It always amazes me when a company tries to cheat, somehow expecting not to get caught. Yesterday, Ars Technica published a detailed takedown as to how Samsung tried to deceive the performance benchmark tests in its latest Samsung Galaxy Note 3.

For the non-technical among the readers, benchmarks in the technical world are exactly what they are in the business world: standards-based tests that allow purchasers to compare multiple items. Computing devices have a series of performance benchmarks, so we can compare the performance of, say, the latest MacBook Air with the previous generation, or the iPhone 5S with the Google Nexus 5.

Like standardized testing, the temptation always exists to "teach to the test" instead of "teach to the student." Companies that sell devices based on performance - not everyone does; Apple cares about performance but it takes a backseat to user experience - will be torn between manufacturing the device so it best suits customer needs vs so it bests the performance benchmarks. These are legitimate business questions, and should lead device-performance-concerned companies to question if they should build for the benchmark, or fight the uphill battle to convince the market that its results are better than the benchmarks show. Some companies have taken the uphill battle in the past, either via marketing or by releasing new versions of benchmarks, which give additional credibility.

Samsung, however, didn't "teach to the test"; it cheated on the test. It released its phone with a special ability to detect if the benchmarking apps are running, and, if so, to put on an artificial boost that doesn't apply when other apps are running. It told its device, "behave much better when running one of the following tests than you will for any other application." This is the equivalent of giving someone a pill that boosts their knowledge artificially for exactly eight hours and then sending them into their Medical Boards or Bar exam... or taking a bottle of Felix Felicis.

The irony is that the Note 3 actually is, for the most part, the best performant of the competitors even without the cheat. So why did they cheat? Probably to make the distinction appear greater and drive sales from existing owners of other devices just behind - "20% better than my device?? maybe I should get it!" - as well as gain an additional 6-12 months of "head of the pack" for their expensive-to-develop devices.

Did they expect to get caught? Of course not. But the world is full of very smart people; it was almost inevitable. And there is no way it was a rogue operation. It required collaboration of engineering, product management and marketing, at the very least. That many groups coordinating to cheat means not just staff or management involvement, but executive involvement as well.

Samsung on Fire.