A set of powerful algorithms use your music and your personal details to shape your entire listening experience

Spotify is doing everything it can to get you to listen to more music.

The company has created algorithms to govern everything from your personal best home screen to curated playlists like Discover Weekly, and continues to experiment with new ways to understand music, and why people listen to one song or genre over another.

While competitors like Apple Music, Amazon Prime Music, and Google Music rely on a mix of paid humans and community-created playlists, Spotify’s main differentiating factor is the level of customization and expansion of music knowledge offered to customers. Spotify needs to continue building out these algorithms because it’s the only way to create custom listening experiences for each of its over 200 million users. As Spotify struggles to grow its business, that differentiating factor needs to be a compelling reason to subscribe to the service.

The home screen of the Spotify app is a prime example of how algorithms govern a listening experience. Its goal is to quickly help users find something they are going to enjoy listening to, according to a presentation by Spotify Research director Mounia Lalmas-Roelleke at the Web Conference earlier this year.

She explained that the home screen is governed by an A.I. system called BaRT (“Bandits for Recommendations as Treatments”). The system’s task is to organize each home screen in a personalized way for each user. That includes the “shelves,” or rows of playlists, that follow a theme like “best of artists” or “keep the vibe going,” and the order the playlists appear on those shelves.

The BaRT system is Spotify’s central balancing act. Its whole purpose is to give you music that Spotify is confident you’ll like, based on your previous listening activity. But Spotify also has to drop in new, fresh music so you don’t get stuck in a loop of listening to the same thing all the time.

The system can be boiled down to two concepts: Exploit and explore. When Spotify exploits, it’s using the information it knows about you, the user. It takes into account your music listening history, which songs you’ve skipped, what playlists you’ve made, your activity on the platform’s social features, and even your location. But when Spotify explores, it uses the information about the rest of the world, like playlists and artists similar to your taste in music but you haven’t heard yet, the popularity of other artists, and more.

Just as important as Spotify’s ability to exploit or explore is how the app explains its choices to users. Each label for shelves like “Jump back in” or “More of what you like” tells the user why those specific playlists are being recommended. Spotify has found that explanation is critical to users trusting explanations, according to the 2018 research paper on BaRT.

Success for BaRT is measured by whether you actually listen to the music on these shelves, and for how long. When you stream a song for more than 30 seconds, the algorithm tracks that as getting the recommendation right, according to the presentation. The longer you listen to the recommended playlist or set of songs, the better the recommendation is determined to be.

Spotify’s sweet spot for understanding whether a person likes a song or not seems to be 30 seconds. In a 2015 interview with Quartz, Spotify product director Matthew Ogle, who has since left the company, mentioned that skipping before the 30-second mark is the equivalent of a thumbs down for the Discover Weekly playlist.

The company makes it clear in research that the success of all these algorithmic services is only possible because every action you make on the service is tracked and logged.

There isn’t much academic work detailing the mechanics of the Discover Weekly playlist in full. In the Quartz story, Ogle gives an overview of the system that jives with a slightly technical presentation Spotify employees gave in 2015 about the feature. Discover Weekly is a 30-song soup of playlists from other people with similar music preferences to your own, songs that literally sound similar to the music you like, and recent coverage from music blogs. Brian Whitman, who co-founded The Echo Nest, the startup Spotify acquired for $100 million in 2014 to get better at recommendations, wrote in 2012 that his service scoured more than 10 million music-related web pages a day in order to understand what was happening in the world of music.

“Every word anyone utters on the internet about music goes through our systems that look for descriptive terms, noun phrases, and other text,” Whitman wrote.

Spotify intern Sander Dieleman, who worked at the company in 2014 and did some foundational work analyzing the auditory similarity of music, also explained the audio analysis algorithm in a personal blog post. The original problem was that new music was uploaded to Spotify every day, but there was no system to recommend music if it wasn’t by a previously popular artist. Collaborative filtering, or the method of recommending music liked by people with similar musical interests to each other, didn’t work when nobody knew the artist in the first place.

Dieleman calls it the “cold-start problem.”

The solution was analyzing the audio itself and training an algorithm to learn to recognize different aspects of the music that might be desirable. Some experiments that Dieleman ran identified aspects of songs as concrete as distorted guitars, while others could recognize more abstract ideas like genre.

That system is now an important element of the Discover Weekly playlist, which is why you might see an artist being recommended that you’ve never heard of.

But algorithms can be tucked into all sorts of places on Spotify. While there are recommendation algorithms, like the ones that power the home screen and Discover Weekly, there are smaller tools that you’ve probably used but would never realize were the product of relatively cutting-edge A.I. research.

Take for instance, automatic playlist continuation. This feature analyzes the songs in a certain playlist and tries to predict the music that should come next — as if the person who created it had just kept adding music. Spotify wanted new ways to think about how it should be building that feature, so it released a “Million Playlist Dataset” of user-generated Spotify playlists that could be used to understand the traits of what humans considered a good set of tracks. The company invited other A.I. researchers to try and help them solve the problem and present their solutions at a 2018 industry conference. More than 100 academic and industry teams formed around the project, according to an analysis done by organizers after the competition. (We don’t know if the winners’ idea actually made it into Spotify’s app.)

Spotify researchers have also been working on ways to detect covers of songs on Spotify, which could play instead of the original version of the song that you actually wanted to hear. The resulting work is able to distinguish covers from the original track with high accuracy, especially instrumental covers and live performances. Jazz proved to be trickier, as there was typically more improvisation.

The team has also worked on aligning written lyrics to the moment in a song where the lyric is sung, which could not only help with the company’s Behind the Music feature that shows lyrics alongside popular songs, but also open up new opportunities for Spotify.

“Time-aligned lyrics can enrich the music listening experience by enabling karaoke, text-based song retrieval and intra-song navigation, and other applications,” Spotify computer scientists wrote earlier this year.

On top of all the research going into the Spotify platform, the company is also researching its users. Spotify studied data from more than 16 million users, tracking their listening patterns from December 2016 to February 2018, including how many times someone streamed a specific artist or song per day and what U.S. state they were in, according to a study published April 2019.

That data, coupled with users’ self-reported gender and age, allowed Spotify to study whether music taste changes after someone has moved to a different state, as well as how age impacts the kind of music a person listens to.

The team intuited a move to another state in a tricky way: Based on location data, Spotify users who went to another state for two of the three major holidays during the selected data — Christmas 2016, Thanksgiving 2017, and Christmas 2017 — were guessed to have moved from that location to the place where they typically stream from.

By studying the musical tastes of people in each state, and then contrasting the group of people who have moved to those different overall trends, the Spotify team concluded that over a long period of time, location does factor into musical taste in some small way.

“Relocation does appear to shift individuals’ tastes marginally towards those of their new environment. The size of this effect, however, is small, and listeners’ tend to more strongly resemble their past rather than present environments,” they wrote.

By studying age, they also suggest the music that is popular from ages 10 to 20 is the music that people will predominantly listen to in the future, having shaped their “musical identity.”

This all points to the vast amount of data that Spotify both has and needs to continue to collect on its users for its service to maintain its competitive edge. The 2015 presentation on Discover Weekly mentioned that Spotify logs one terabyte of user data per day.

However, that data is clearly used in service of its customers. The company makes it clear in research that the success of all these algorithmic services is only possible because every action you make on the service is tracked and logged.

And that might prove to be the secret sauce of music streaming. After all, Spotify has managed to thrive despite battling every tech giant, including music juggernaut Apple, which it’s beating by tens of millions more paying customers around the world.