By Terran Melconian, enterpreneur and consultant, and Trevor Bass, edX
Image courtesy of Intersection Consulting
When you think of “data science” and “machine learning”, do the two terms blur together, like Currier and Ives or Sturm and Drang? If so, you’ve come to the right place. This article will clarify some important and often-overlooked distinctions between the two to help you better focus your learning and hiring.
Machine learning versus data science
Machine learning has seen much hype from journalists who are not always careful with their terminology. In popular discourse, it has taken on a wide swath of meanings and implications well beyond its scope to practitioners. Machine learning refers to a specific form of mathematical optimization: getting a computer to perform better at some task, through training data or experience, without explicit programming. This often takes the form of building a model based on past cases with known outcomes, and applying the model to make predictions for future cases, finding ways to minimize a numerical “error” or “cost” function representing how much the predictions mismatch reality.
Notice that some important business activities appear nowhere in this definition of machine learning:
-
Assessing whether data is suitable for a purpose
-
Formulating an appropriate objective
-
Implementing systems and processes
-
Communicating with disparate stakeholders
The need for these functions led to the recognition of data science as a field. The Harvard Business Review tells us that the “key skills for data scientists are not the abilities to build and use deep-learning infrastructures. Instead they are the abilities to learn on the fly and to communicate well in order to answer business questions, explaining complex results to nontechnical stakeholders.” Other authors agree: “We feel that a defining feature of data scientists is the breadth of their skills — their ability to single-handedly do at least prototype-level versions of all the steps needed to derive new insights or build data products.” Another HBR article affirms, “Getting value from machine learning isn’t about fancier algorithms — it’s about making it easier to use…. The gap for most companies isn’t that machine learning doesn’t work, but that they struggle to actually use it.”
Machine learning is an important skill for data scientists, but it is one of many. Thinking of machine learning as the whole of data science is akin to thinking of accounting as the entirety of running a profitable company. Further, the skills gap in data science is largely in areas complementary to machine learning — business sensibility, statistics, problem framing, and communication.
If you want to be a data scientist, seek out an interdisciplinary education
It is no secret that data scientists are in high and increasing demand. Despite this, much of the most hyped educational programming in data science tends to be concentrated in classes teaching machine learning.
We see this as a significant problem. Many student have focused far too heavily on machine learning education over a more balanced curriculum. This has unfortunately led to a glut of underprepared early-career professionals seeking data science roles. Both of the authors, and several other data science hiring managers with whom they spoke when preparing this article, have interviewed numerous candidates who advertise their knowledge of machine learning but who can say little about basic statistics, bias and variance, or data quality, much less present a coherent project proposal to achieve a business objective.
In the authors’ experience, software engineers seem especially susceptible to the siren’s call of an education too rich in machine learning. We speculate that this is because machine learning uses the same type of thinking that already comes easily to software developers: algorithmic, convergent thinking with clearly defined objectives. An education that is hyper-specialized in machine learning offers the false promise of more interesting work without demanding any fundamental cognitive shifts. Sadly, the job market rarely delivers on this promise, and many who follow this path find that they are unable to make the career shift from engineer to scientist.
Data science demands learning a different style of thought: often divergent, poorly defined, and requiring constant translation in and out of the technical sphere. Data scientists are fundamentally generalists, and benefit from a broad education over a deep one. Interdisciplinary study is a far better bet than a narrow concentration.
Scalable versus non-scalable jobs
Most organizations will generate significantly more value by hiring generalist data scientists before machine learning specialists. To understand why this is the case, it is useful to appreciate the difference between scalable and non-scalable jobs.
Creating general purpose machine learning algorithms is a scalable job — once somebody has designed and implemented an algorithm, everybody can use it with virtually no cost of replication. Of course everyone will want to use the best algorithms, created by the best researchers. Most organizations cannot afford to hire top-tier algorithm designers, many of whom receive seven-figure salaries. Thankfully, much of their work is available to the public in research papers, open source libraries, and cloud APIs. Thus the world’s best ML algorithm designers have an outsized impact, and their work enables the generalist data scientists who use their algorithms to have a large impact in turn.
Conversely, data science is a far less scalable activity. It involves understanding the specifics of a particular company’s business, needs, and assets. Most organizations of a certain size need their own data scientist. Even if some other company’s data scientist had published their approach in detail, it’s virtually certain that some aspects of the problem and situation will differ, and the approach cannot just be copied in toto.
Of course, there are many highly worthwhile and interesting paths for a career other than data science. In case you are thinking of a career more specifically in machine learning, here’s one of the dirty secrets of the industry: Machine Learning Engineers at large companies actually do very little machine learning themselves. Instead, they spend most of their time building data processing pipelines and model deployment infrastructure. If you do want one of these (often excellent) jobs, we still recommend focusing a minority of your education on machine learning algorithms, in favor of general engineering, DevOps practices, and data pipeline infrastructure.
While the world’s best machine learning expert may be able to contribute more to the grand sum of human knowledge than the world’s best data scientist, a skilled data scientist can have an outsized impact in a much broader range of situations. The job market reflects this. If you are seeking employment, you will likely do best by consuming machine learning education as just one part of a balanced diet. And, if you are looking to make your company more data driven, you will likely do best by hiring a generalist.
Counter to the hype, amassing machine learning education beyond the basics without also skilling up in complementary areas has diminishing returns on the job market.
Bio: Terran Melconian has led software engineering, data warehousing, and data science teams and both startups and industry giants including Google and TripAdvisor. He currently guides companies starting their first data science efforts, and teaches data science (not just machine learning!) to software engineers and business analysts.
Trevor Bass is a data scientist with over a decade of experience building highly successful and innovative products and teams. He is currently the Chief Data Scientist at edX, an online learning destination and MOOC provider offering high-quality courses from the world’s best universities and institutions to learners everywhere.
Resources:
Related: