How can we use machine learning to predict stockprices? In this tutorial we will make Python scripts for doing sentiment analysis on Tweets and it is explained how to use it for making predictions.
As an example, suppose we had €1000,- at the first of January of 2014 and suppose we could use the algorithm which is described in this tutorial. Then it would generate €2901,- in total on the 22th of February, 2017! The total amount of money (cash + investments) is shown in the next figure:
Despite the patience you need to have, it will be worth the waiting time eventually. As mentioned in [1], moods in tweets are a good indication of the movement of closing prices on a stock market. In this article, we will only predict how positive or how negative a tweet is. But it turns out that this is giving predictive signals which is accurate enough for our purposes.
Looking for new interesting investments idea? Check out our Crypto Advisor service!
Data sources
First things first. Before we can analyze Tweets and stock prices, we need to obtain the data.
Bitcoin closing prices
For the Bitcoin prices, I used CoinDesk. Here, it is also possible to export the prices to the CSV format, which we then can use using one of the many Python libraries for parsing CSV files. I used the Bitcoin prices from 2014-01-01 up to 2017-02-22 and I removed the text which is appended by CoinDesk at the last couple of lines in the CSV file. I will not give Python code for importing your own datafile, since it depends on the structure of the file you use. For importing the CSV file, I used a Python package called Pandas. Importing the CSV file is as simple as the following line of code:
1 | |
Reading the Bitcoin datafile
If you are new to Pandas, this library is quite easy to use. Pandas is frequently imported as “pd” and dataframes are often prefixed with “df_”. With Pandas, it is easy to compute the mean and other statistics and grouping data.
Tweets
Unfortunately, Twitter has API rate limits, which limits you to a certain amount of API requests per 15 minutes. And luckily there is a Python library which slows down the requests such that you will not go over the rate limit. The downside of this, is that you will have to wait a long time in order to fetch a large number of Tweets. Therefore, I took only 50 tweets per day (which contained the keyword “bitcoin”) for the period 2014-01-01 up to and including 2017-02-22. I used the GetOldTweets-python library for collecting the Tweets. Then, I converted the tweets to the CBOR format because data can be analyzed directly by any other script. The CBOR format allows you to dump objects at the end of a file and process all dumped objects in parallel. Using CBOR files is optional, but I found it quite useful.
Sentiment Analysis
Now we have the Tweets and the closing prices, it is time for some analyzing. We want to know how “positive” or “negative” a Tweet is. Some approaches take a word of lists and define for all the words whether a word is positive or negative. Then, you can count the number of positive words and subtract the number of negative words, divide that by the number of words and you’ve got a simple score of how positive your tweet is. For example, take the sentence “I hate this song.” Suppose that the list of positive words contains the word “love” and the list of negative words contains the word “hate”. Then, the simple score of this sentence equals (0 – 1) / 4 = -0.25. The sentence “I love you” then has a score of (1 – 0) / 3 = 0.33. So, you can now say that “I love you” is more positive then the sentence “I hate this song”. Can you spot the issues with this approach? First of all, the number of words in a sentence influence the simple score. “I love you” would be more positive than “I love you very much” (score: (1 – 0) / 5 = 0.20). Secondly, some words are more positive than other words. For example, “happiest” would be more positive than “happy”. Furthermore, for some words, even humans do not agree on which is more positive or negative than an other word. And lastly, some word combination should invert the score! Consider “I do not hate this song”. Is it positive or negative? And how more positive or negative would it be compared to “I hate this song”? Many questions, and some of the problems are not tackled yet. I will use a Python package which does only simple Sentiment Analysis only using a word list. The package I used is called Afinn [2].
Machine Learning
Actually, this should be called human learning for now. I just looked at the results and set a cut-off point manually. Above the cut-off point, you should consider buying Bitcoins and below the second cut-off point, you should consider selling Bitcoins. The following graphs are the result of running the described algorithm:
I intentionally left out some important details for obtaining the results since I sell this piece of software.
If we look at the results, we can see the obtained sentiment in the top figure. There are clearly some local optima in the plot. Using the cut-off points, we get the second plot, which shows the sell signals (-1) and buy signals (+1). If you would apply these signals, you obtain the results of the first plot, so you would have made €2901,- (when started with €1000,-)!
Improvements
There are many possible improvements. For example, we could apply the sentiment analysis on other keywords and check out whether it has predictive powers for other stocks. The manual evaluation of the cut-off point could easily be automated. Furthermore, maybe there are some hidden signals which could be found by a machine learning algorithm. If you have any other improvements or suggestions, please let me know!
If you have any questions or if you would like access to the datasources, please send in a comment below the post!
References
[1] Bollen, J., Mao, H., & Zeng, X. (2011). Twitter mood predicts the stock market. Journal of computational science, 2(1), 1-8.[2] Nielsen, F. Å. (2011). A new ANEW: Evaluation of a word list for sentiment analysis in microblogs. arXiv preprint arXiv:1103.2903.