To close out our series on building recommendation models using Sketchfab data, I will venture far from the previous posts’ factorization-based methods and instead explore an unsupervised, deep learning-based model. You’ll find that the implementation is fairly simple with remarkably promising results which is almost a smack in the face to all of that effort put in earlier.
We are going to build a model-to-model recommender using thumbnail images of 3D Sketchfab models as our input and the visual similarity between models as our recommendation score. I was inspired to do this after reading Christopher Bonnett’s post on product classification, so we will follow a similar approach.
Since our goal is to measure visual similarity, we will need to generate features from our images and then calculate some similarity measure between different images using said features. Back in the day, maybe one would employ fancy wavelets or SIFT keypoints or something for creating features, but this is the Era of Deep Learning and manual feature extraction is for old people.
Staying on-trend, we will use a pretrained neural network (NN) to extract features. The NN was originally trained to classify images among 1000 labels (e.g. “dog”, “train”, etc…). We’ll chop off the last 3 fully-connected layers of the network which do the final mapping between deep features and class labels and use the fourth-to-last layer as a long feature vector describing our images.
Thankfully, all of this is extremely simple to do with the pretrained models in Keras. Keras allows one to easily build deep learning models on top of either Tensorflow or Theano. Keras also now comes with pretrained models that can be loaded and used. For more information about the available models, visit the Applications section of the documentation. For our purposes, we’ll use the VGG16 model because that’s what other people seemed to use and I don’t know enough to have a compelling reason to stray from the norm.
Our task is now as follows:
-
Load and process images
-
Feed images through NN.
-
Calculate image similarities.
-
Recommend models!
The first step, which we won’t go through here, was to download all of the image thumbnails. There seems to be a standard thumbnail for each Sketchfab model accessible via their API, so I added a function to the rec-a-sketch crawl.py script to automate downloading of all the thumbnails.
Let’s load in our libraries and take a look at one of these images.