** Wed 06 April 2016
Summary: It’s much easier to create impressive demos than it is to create dependable, functionally-comprehensive production software. I discuss my thoughts on this topic.
Post-Conference Lows
Last week was the annual California Strata-Hadoop World conference. I’ve now been to some variant of the Strata conference 12 times since Fall 2011.
Having worked more than 8 out of the last 10 years on building production-grade data analysis systems, conferences like Strata have grown emotionally draining for me as the majority of the time I spend solving engineering problems that don’t lend themselves to especially exciting demos. So it’s easy to walk away from the conference feeling pretty worthless, like everyone else is “crushing it” while I’m spending all my time dealing with Unicode bugs and GroupBy optimizations.
Much software, both open source and proprietary, depends on the tireless efforts of many unsung heroes, folks like my fellow pandas team-member Jeff Reback. A lot of the development work that’s enabled pandas to become a dependable software component for hundreds of thousands of Python users is too esoteric to explain to most normal users!
Of course, I recognize that the fame I’ve accrued in the Python community is a rarity (and not to be taken for granted). I wish more of it could be spread around to my hacker-compatriots fighting the good fights to make the community of users as successful as possible.
It goes the other way, too. When I meet new software developers, rather than ask them for their polished pitch about the cool things their code does, I try to find out what kinds of down-in-the-muck problems they had to solve to make their code a serious tool for doing real work. If that doesn’t immediately dredge up horror stories about the most esoteric of bugs and software refactoring problems, it’s usually a clue that the software may not be as impressive as it seems in a demo.
Demos and Potemkin Villages
In high school I learned about the Potemkin villages constructed during the Russian Empire as a way to be impressive without doing that much work. I like using that as an analogy for so-called “demoware”, software crafted to show nicely in a slide deck, conference talk, or publication.
But, demoware is great! At least, it is until you sit down to do your day job with it.
Perhaps the most insidious trend I’ve seen in the industry is the Machiavellian tendency to build software solely for the purposes of impressing investors and prospective-customers. From a game theory point of view, this may be optimal behavior, especially in the early days of a project where solving the “hard problems” may have diminishing returns when you’re bootstrapping a project from nothing.
I personally have struggled with building purposeful demoware, because my conscience burdens me greatly when I show it to anyone. Some people have the “used-car salesman” bit in their brain that permits some freedom in the commitment to integrity and intellectual honesty (“she’s got a great engine, honest!”).
Most of the stuff I work on is more of the “submarine-and-periscope” model – you see a nice API on top, but there’s a complex and painstakingly-constructed unseen machine under the hood. I prefer it that way, to be honest.
Moving forward
If I’ve learned anything from my time on this planet, it’s that my personality and approach toward creating new things is unlikely to change all that much. So I’ll keep doing what I’m doing: identifying and solving the hard problems, even the ones that don’t seem obvious.
Luckily, I feel confident that when users venture below the surface (or look inside the proverbial Potemkin house), they’ll be happy with what they find.