This system detects in real time when the note is coming by using a piezo and accelerometer on the plectrum. In this demonstration, we play noise between the prediction and the actual onset of the note to show you how far ahead we knew the next note was coming. The controller was optimized to predict 50ms ahead. The prediction is done with reservoir computing for the sensor fusion (also known as echo state networks).
In order to have everything running in realtime on the Axoloti board, the weight matrices had to be kept sparse, posing additional constraints on the system. After all it had to predict an onset happening in 50ms on a microcontroller. The computations to come to that conclusion therefore need to be fast as well, otherwise you will need to predict even sooner as you lose time computing. Evenmore, everything had to be done on the 168MHz STM32F427 microcontroller.
The Axoloti board
Paper available here: http://dx.doi.org/10.1080/09298215.2017.1288747
Patent pending