SPOT-(AI)-FY

6 min readOct 20, 2020

How Spotify uses AI to rule the music streaming world.

Spotify, huh what’s that?

Spotify is a Swedish-based audio streaming and media services provider, launched in October 2008.
Spotify offers digital copyright restricted recorded music and podcasts, including more than 60 million songs, from record labels and media companies.As a freemium service, basic features are free with advertisements and limited control, while additional features, such as offline listening and commercial-free listening, are offered via paid subscriptions. Users can search for music based on artist, album, or genre, and can create, edit, and share playlists.
Spotify is available in most of Europe and the Americas, Oceania, and parts of Africa and Asia, and on most modern devices, including Windows, macOS, and Linux computers, and iOS, and Android smartphones and tablets.[8][9] June 2020, the platform has 299 million monthly active users, including 138 million paying subscribers.

If you use Spotify you might know about the Spotify’s Made For You.

Spotify creates some playlists for their users, one of such playlist is the Discover Weekly, it is one of the best services Spotify has to offer.
Each Monday individual users are presented with a customized list of thirty songs. The recommended playlist comprises tracks that users might have not heard before, but they might like them. These recommendations are generated based on the user’s search history pattern and potential music preference.

That’s cool but how???

Spotify developed a uniquely powerful music recommendation engine, mainly based on three previously known recommendation models:

- Collaborative Filtering:

User’s behavioral patters are compared to those of other users.
Just like Netflix which also uses collaborative filtering to power their recommendation models, using viewers’ star-based movie ratings to create recommendations for other similar users. While Spotify doesn’t incorporate a rating system for songs, they do use implicit feedback — like the number of times a user has played a particular song, saved a song to their lists, or clicked on the artist’s page upon listening to the song — to provide relevant recommendations for other users that have been deemed similar.
Spotify uses Matrices and Linear Algebra to implement this algorithm.

- Natural Language Processing (NLP):

NLP analyses human speech via text. Spotify’s AI scans a track’s metadata, as well as blog posts and discussions about specific musicians, and news articles about songs or artists on the internet. It looks at what people are saying about certain artists or songs and the language being used, and also which other artists and songs are being discussed alongside, if at all, and identifies descriptive terms, noun phrases and other texts associated with those songs or artists.
These keywords are then categorised into “cultural vectors” and “top terms”. Every artist and song is associated with thousands of top terms that are subject to change on a daily basis. Each term is assigned a weight, reflecting its relative importance in terms of how many times an individual would attribute that term to a song or musician they like. Spotify doesn’t have a fixed dictionary for this, but the system is able to identify new music terms as and when they come up — not just in English, but also in Latin-derived languages across cultures. Of course, spam and non-music related content is discarded through a filtering process.

- Audio models:

Audio models are used to analyse data from raw audio tracks and categorize songs accordingly. This helps the platform evaluate all songs to create recommendations, regardless of coverage online. For instance, if there is a new song released by a new artist on the platform, NLP models might not pick up on it if coverage online and in social media is low. By leveraging song data from audio models, however, the collaborative filtering model will be able to analyze the track and recommend it to similar users alongside other more popular songs.
Spotify uses convolutional neural networks to extract musical features directly from raw audio. Interestingly, convolutional networks have mainly been used with visual data(Computer Vision). As a result, data scientists have successfully applied them to image detection. This is achieved by feeding a dataset of images to the network, pixel by pixel, to train the model. Once trained, the algorithm is capable of classifying different objects that appear in images that are new to the network. In the case of Spotify, the network has been modified to accept audio data as the input instead of pixels.

Last week at the 2020 International Society for Music Information Retrieval Conference, Spotify open-sourced Klio, an ecosystem that allows data scientists to process audio files (or any binary files) easily and at scale. It was built to run Spotify’s large-scale audio intelligence systems and is leveraged by the company’s engineers and audio scientists to help develop and deploy next-generation audio algorithms.
The Apache Beam-based Klio enables organizations to create media processing systems that share tooling and infrastructure between production systems and research teams. The platform’s architecture encourages reusable jobs and shared outputs, ostensibly lowering maintenance and recomputation costs. Moreover, Klio supports continuous, event-driven processing of rapidly growing catalogs of media content, providing engineers a framework to productize processing jobs and organizations a way to process new content on ingestion.

How Spotify Uses Klio?

Spotify uses Klio to run its audio intelligence platforms, which powers the audio features, aspects of the audio ad platform as well as the audio research workflow. Further, the developers at the digital music platform use Klio for tasks like running Tensorflow models over audio files, creating new audio from existing audio in areas like audio fingerprinting and ad generation, and saving analysed data to databases.