Demo of the chatbot
Last week my microwave stopped working so I went to Big W to get a replacement. As expected, they asked for the receipt which I did not have as it was my wife who bought it. I had to go through my bank transactions record to find the proof of purchase, which fortunately was not long since the purchase was recent. Then it occurred to me — since I’ve been building chatbots at work — why don’t I build a bot that would manage receipts for me. My requirement for the bot is minimal:
-
It should be able to take a photo of my receipt, extract the information automatically; and,
-
It should allow me to retrieve receipts later by the merchant name or the items in it.
This post describes how I build the bot. If you’re interested, you can check out the code on my github page.
The first thing to do was to design the system architecture: which components are required and how they link together. Here is the list of the components and the framework and service behind the implementation.
-
Chatbot — this is essentially a web server and is implemented using the Microsoft Bot Builder Framework
-
Text detection — extracting text from images with the Google Cloud Vision service
-
Organisation / merchant name extraction with ABN look up service by the Australian Business Register
-
Information extraction with regular expression
-
Information retrieval — this can be done by querying a database for merchant name or for items contained in the receipt. This was not implemented, for I did not plan to pay for the cost of hosting the bot, a database server, and storing images.
You can find a video demo of the chatbot in action at the top of this article. The interactional channel is a bot emulator, which you can think of as a web chat. There is some noise in the background and the response was a bit slow as I was on the train 
It works well functionally but clearly improvements can be made, in terms of the language and the look and feel of the bot — which is why building chatbot is a multidisciplinary effort just like other user interfaces. You may notice that the total value of the receipt is 0, and that is correct because that transaction was only an exchange for a new product so no payment was involved.
I imagine this bot can serve me well. But when it comes to transactional data, there are potentially many exciting applications. An interesting idea is an ‘automatic reward bot’: it reads transactions, identifies the merchants, then negotiate with them (or rather other bots) on behalf of the user for reward / loyalty points. The points can also be transferred or exchanged say within one’s social network. Dare I say this is a sexy combination of blockchains and chatbot, which is quite similar to what Kik is trying to do through this ICO.