(c) 2018 by Thomas Wiecki
Imagine you have a machine learning (ML) problem but only small data (gasp, yes, this does exist). This often happens when your data set is nested – you might have many data points, but only few per category. For example, in ad-tech you may want predict how likely a user will buy a certain product. There could be thousands of products but you only have a small number of measurements (e.g. purchase decisions) per item. Likely, there will be similarities between each product category, but they will all have individual differences too.
Using classic ML methods is unlikely to yield good results in this setting. Firstly, ML is happiest with lots of data. If you want to be fancy and use deep learning, that usually requires even more data to give good performance. Moreover, we can’t really make use of the hierarchical structure (product-groupings) we know to be present in our data. So we can then either ignore it and try to learn one model for all categories which does not allow us to capture the differences between categories; or we can fit a ML model to each category separately but do not exploit the similarities between categories. (Technically, you could add the category as a feature and fit a single model but when I tried it on the data presented futher below it was at chance).
Hierarchical Bayesian models work amazingly well in exactly this setting as they allow us to build a model that matches the hierarchical structure present in our data set. Check out my previous blog post for a refresher.
In this setting we could likely build a hierarchical logistic Bayesian model using PyMC3. However, what if our decision surface is actually more complex and a linear model would not give good performance?
In this blog post I explore how we can take a Bayesian Neural Network (BNN) and turn it into a hierarchical one. Once we built this model we derive an informed prior from it that we can apply back to a simple, non-hierarchical BNN to get the same performance as the hierachical one.
This problem also occurs in multi-task learning. Neural networks trained to play multiple atari games have a very hard time because they tend to forget
In [1]:
The data set we are using are our battle tested half-moons as it is simple, non-linear and leads to pretty visualizations. This is what it looks like:
In [2]:
This is just to illustrate what the data generating distribution looks like, we will use way fewer data points, and create different subsets with different rotations.
In [3]:
In [4]:
In [5]:
As you can see, we have 18 categories that share a higher-order structure (the half-moons). However, in the pure data space, no single classifier will be able to do a good job here. Also, because we only have 50 data points in each class, a NN will likely have a hard time producing robust results. But let’s actually test this.
Classify each category separately¶
The code for the NN below is explained in this blog post.
In [6]:
Next, we have our function to create the BNN, sample from it, and generate predicitions. You can skip this one.
In [98]:
Next, we loop over each category and fit a different BNN to each one. Each BNN has its own weights and there is no connection between them. But note that because we are Bayesians, we place priors on our weights. In this case these are standard normal priors that act as regularizers to keep our weights close to zero.
We use NUTS sampling here because the hierarchical model further below has a more complex posterior (see Why hierarchical models are awesome, tricky, and Bayesian for an explanation) and I wanted the results to be comparable. All simulations in here work with ADVI as well but the results don’t look quite as strong.
In [ ]:
In [9]:
Train accuracy = 86.78%
In [10]:
Test accuracy = 83.78%
OK, that doesn’t seem so bad. Now let’s look at the decision surfaces – i.e. what the classifier thinks about each point in the data space.
In [11]:
That doens’t look all that convincing. We know from the data generation process as well as from the previous blog posts using this process that it give us a “Z”-shaped decision surface. So what happens is we don’t have enough data to properly estimate the non-linearity in every category.
Hierarchical Bayesian Neural Network¶
Can we do better? You bet!
It’s actually quite straight-forward to turn this into one big hierarchical model for all categories, rather than many individual ones. Let’s call the weight connecting neuron $i$ in layer 1 to neuron $j$ in layer 2 in category $c$ $w_{i, j, c}$ (I just omit the layer index for simplicity in notation). Rather than placing a fixed prior as we did above (i.e. $ w_{i, j, c} \sim \mathcal{N}(0, 1^2)$), we will assume that each weight comes from an overarching group distribution: $ w_{i, j, c} \sim \mathcal{N}(\mu_{i, j}, \sigma^2)$. The key is that we will estimate $\mu_{i, j}$ and $\sigma$ simultaneously from data.
Why not allow for different $\sigma_{i,j}^2$ per connection you might ask? Mainly just to make our life simpler and because it works well enough.
Note that we create a very rich model here. Every individual weight has its own hierarchical structure with a single group mean parameter and 16 per-category weights distributed around the group mean. While this creates a big amount of group distributions (as many as the flat NN had weights) there is no problem with this per-se, although it might be a bit unusual. One might argue that this model is quite complex and while that’s true, in terms of degrees-of-freedom, this model is simpler than the unpooled one above (more on this below).
As for the code, we stack weights along a 3rd dimenson to get separate weights for each group. That way, through the power of broadcasting, the linear algebra works out almost the same as before.
In [12]:
In [68]:
Auto-assigning NUTS sampler… Initializing NUTS using advi+adapt_diag… Convergence achieved at 54100 Interrupted at 54,099 [27%]: Average Loss = 522.66 Sequential sampling (1 chains in 1 job) NUTS: [w_2_out, w_1_2, w_in_1, w_2_out_grp_sd, w_2_out_grp, w_1_2_grp_sd, w_1_2_grp, w_in_1_grp_sd, w_in_1_grp] Only one chain was sampled, this makes it impossible to run some convergence checks
In [69]:
Train accuracy = 91.11%
In [70]:
Test accuracy = 89.78%
Great – we get higher train and test accuracy. Let’s look at what the classifier has learned for each category.
In [88]:
Awesome! By (partially) pooling the data for each individual category we actually manage to retrieve the non-linearity. This is the strength of hierarchical models: we model the similarities of individual categories and their differences, sharing statistical power to the degree it’s useful.
Of course, as we are in a Bayesian framework we also get the uncertainty estimate in our predictions:
In [89]:
Further analysis¶
There are a couple of things we might ask at this point. For example, how much does each layer specialize its weight per category. To answer this we can look at the group standard-deviation which informs us how much each weight is allowed to deviate from its group mean.
In [90]:
Interestingly, it seems that the specialization of the individual category sub-models is happening at the last layer where weights change most strongly from their group mean (as group variance is highest). I had assumed that this would happen at the first layer, based on what I found in this earlier blog post where the first layer acted as a rotation-layer.
Another interesting property of hierarchical models reveals itself here. As the group standard deviation is small for the weights in layers 1 and 2, it means these weights will be close to their group mean, reducing the effective number of parameters (degrees of freedom) of this model. This is different from the separate models where no similarities could be exploited to reduce the effective number of parameters. So from this perspective, the hierarchical model is simpler than the sum of the separate ones above.
Finally, I wondered what the group-model actually learned. To get at that, we can use the trace of $\mu_{i,j}$ from the hierarchical model and pass it to the non-hierarchical model as if we trained these weights directly on a single data set.
In [91]:
In [106]:
It seems like the group model is representing the Z-shape in a fairly noisy way, which makes sense because the decision surface for every sub-model looks quite different.
Correlations between weights¶
Usually we estimate NNs with MLE and BNNs with mean-field variational inference (like ADVI) which both ignore correlations between weights. As we used NUTS here, I was curious if there are meaningful correlations. Here, we look at the correlations in the first layer of the group distribution.
In [93]:
Out[93]:
It indeed seems like point or mean-field estimates miss a lot of higher-order structure.
Informative priors for Bayesian Neural Networks¶
Informative priors are a powerful concept in Bayesian modeling. Any expert information you encode in your priors can greatly increase your inference.
The same should hold true for BNNs but it raises the question how we can define informative priors over weights which exist in this abstract space that is very difficult to reason about (understanding the learned representations of neural networks is an active research topic).
While I don’t know how to answer that generally, we can nonetheless explore this question with the techniques we developed this far. The group distributions from our hierarchical model are providing structured regularization for the subnetworks. But there is no reason we can’t use the group distributions only in a hierarchical network. We can just use the inferred group structure and reapply it in the form of informative priors to individual, flat networks.
For this, we must first estimate the group distribution as it looks to the subnetworks. The easiest approach is to draw sample $l$ from the group posterior distributions ($\mu_l$ and $\sigma_l$) and, using this sample, draw realizations $x$ from the resulting distribution: $x \sim \mathcal{N}(\mu_l, \sigma_l^2)$. This is essentially sampling from the group posterior predictive distribution.
In [94]:
While there is no guarantee that this distribution is normal (technically it is a mixture of normals so could look much more like a Student-T), this is a good enough approximation in this case. As the correlation structure of the group distributions seem to play a key role as we’ve seen above, we use MvNormal priors.
Note that this code just creates a single, non-hierarchical BNN.
In [95]:
Again, we just loop over the categories in our data set and create a separate BNN for each one. This is identical to our first attempt above, however, now we are setting the prior estimated from our group posterior of our hierarchical model in our second approach.
In [ ]:
In [100]:
In [101]:
Train accuracy = 91.56%
In [102]:
Test accuracy = 87.11%
Holy mackerel, it actually worked!
As demonstrated, informed priors can help NNs a lot. But what if we don’t have hierarchical structure or it would be too expensive to estimate? We could attempt to construct priors by deriving them from pre-trained models. For example, if I wanted to train an object recognition model to my own custom categories, I could start with a model like ResNet trained on the CIFAR data set, derive priors from the weights, and then train a new model on my custom data set which could then get by with fewer images than if we trained from scratch.
Summary¶
In this blog post I showed how we can borrow ideas from Bayesian statistics (hierarchical modeling and informative priors) and apply them to deep learning to boost accuracy when our data set is nested and we may not have a huge amount of data.
If you want to play around with this notebook yourself, download it here.