By Avneesh Sharma, Tech Product Management
Engineering allowed us to push the limits of human capabilities. We used our understanding of nature and utilized that to serve our purposes. Be it a high performant mechanical machinery or an encoded silicon chip. Computers have been by far one of the most intricate utilization of nature’s forces put to help humans in pushing their limits of capabilities i.e. many tasks which can be performed by computers can never be performed that quickly and efficiently by a human or a set of humans. As Steve Jobs would say, computers are like a bicycle for our minds.
I have been fascinated by computers from my childhood. I had my first interaction with computers back in 2001 when I wrote a program in BASIC to add two numbers. I was amazed that no matter how difficult the addition was, computer answered it instantly.
When I heard about Machine Learning I couldn’t contain the amazement. I was not able to get my mind around the fact, that unlike normal software programs - which I was accustomed to - I wouldn’t even have to teach a computer the “how” in detail about all the future scenarios up front. There was a way that computers can themselves learn how to solve a problem. It was a giant leap for mankind.
What is Machine Learning?
The term ‘machine learning’ isn’t new. It was coined coined by Arthur Samuel in 1959. It has borrowed understandings from different fields like Computer Science, Statistics, Linear Algebra, and Probability Theory and applied to solve practical problems. Machine learning enables the computers to learn from the historical data and formulate a solution which can be used to solve similar problems in future, without the explicit need of teaching computers all the combination of possible scenarios in the problem. And the practical application of machine learning is where it is not even feasible to articulate a definite mathematical solution to the problem.
A real-world problem is a candidate for the application of machine learning if -
Historical data exists in a huge amount A pattern exists in the data Extremely hard to pin down a solution mathematically
How is it different from conventional programming?
The approach of conventional programming is to feed the computer with a set of instructions for a defined set of scenarios. After that computer will utilize its computing capabilities to help human process the data faster and in an efficient fashion. Whereas in machine learning, a huge amount of data is thrown at the computer, which in turn processes all the data and comes up with something called trained model(solution). Then this model is used to solve the unseen problems of the real world.
Example
Let us take a toy problem to demonstrate the difference. The problem takes an input number and tries to divide by 3 and 5. If the number is divisible by 3 then it prints ‘fizz’, if it is divisible by 5 then it prints out ‘buzz’ and if it is divisible by both then it prints out ‘fizzbuzz’. If it is not divisible by any of the 3 or 5 then print ‘other’. It’s called a Fizzbuzz game.
Conventional Programming
It is extremely easy in conventional programming to feed the computer with a set of instructions because we have only 4 scenarios to be verified and print the output based on that. The python code can be written as below but you can skip reading code if you are not into coding.
Machine Learning
Supposed we already have a lot of numbers whose output is already known i.e whether it is ‘fizz’ or ‘buzz’ or ‘fizzbuzz’. All we need to do is now, write a machine learning code and feed (train) the available data. Then verify whether we have successfully created a model by verifying with testing with an unseen data. If model provides the output using trained model without actually calculating the result then we have achieved the purpose.
We will be using Google’s Tensorflow library for this purpose. Here are some code snippets from the implementation.
Again, you can skip the code if you aren’t into coding. The whole working code can be found here in my Github account.
a. Creating the model
b. Train the model
c. Test the model
d. Result:
Conclusion
We repeatedly fed the same data 5000 times, after re-shuffling it every-time. And after each run, we measured whether how accurate out model has become by measuring the error rate in test data. As it can be seen that there is 98 percent accuracy for the model. I have plotted the graph of error after each iteration. The closer it gets to 1 better our model has become. Here is what it looks like:
Final Words
As seen above, in this particular toy case - the conventional program will always give the correct output whereas machine learning might not reach 100% accuracy level.
Now imagine a scenario where you are running a company like Netflix which has millions of customers. Each customer has a unique set of preferences. For example, some person might love documentaries and comedy movies but he watches documentaries on Tuesdays and comedies on Sunday. While other person binge watches on Saturday nights and loves watching horror on Sunday and so on. You can imagine how complex it would be to document all the habits of each individual when there are so many variables which determine exact preferences. Because if you want to suggest each customer a movie which he/she is going to watch then you must know the exact preferences of that person.
For conventional programming, this problem becomes increasingly difficult because:
You don’t know what all things determine the watching habits of a person. Even if you know, then the solution will not scale to millions of user at a time because for each person you have to write a separate solution based on his/her habits.
This is where machine learning is implemented. Generally, machine learning will be trained on two premises in this case:
Based on your past data, which movies are most likely you are going to watch? What are people like you watching these days?
Do let me know your thoughts by commenting below.
PS: Full code used in the article can be found here.
References
https://www.amazon.com/Pattern-Recognition-Learning-Information-Statistics/dp/0387310738 https://www.youtube.com/watch?v=mbyG85GZ0PI&index=1&list=PLD63A284B7615313A
Bio: Avneesh Sharma is a Masters in Management Information Systems candidate at University of Buffalo (2018-19). He has 8 years of experience in developing Software Products/Features ground up. He loves solving problems which have direct impact on business and make end users happy. He believes that creating a technology in general is a means to solve an existing real world pain point to make life easy.
Original. Reposted with permission.
Related: