Activision ran a secret experiment with 50% of Call of Duty's players

Call of Duty's multiplayer matchmaking uses a custom SBMM, or skill-based matchmaking, which has been praised and criticized in equal measure. Critics of SBMM in a game like Call of Duty claim that skill-based matchmaking leads to similar-feeling rounds where competition is high but unpredictability is low. SBMM, in a nutshell, is all about pairing players of a certain skill level with other players of a similar skill.

2

As part of a new in-depth paper and fascinating report surrounding the "complex and multifaceted domain" that is Call of Duty matchmaking, Activision confirmed that in early 2024, it ran a secret 'Deprioritize Skill Test in Call of Duty: Modern Warfare III' in North America. Activision put around 50% of the population into random matches without the usual skill check and grouping.

"For the treatment group, we loosened the skill constraints," Activision writes. "The other half of the population was left with the standard configuration." The test was run for a month, and the results were fascinating. Outside of the top 10% of players, the returning player rate metric was down by a staggering 90%.

Although there was only a marginal difference, Activision believes this will compound over time. "This test only deprioritized skill in the matching rules," the report writes. "If it were completely removed, we would expect to see the player population erode rapidly in the span of a few months, resulting in a negative outcome for all our players."

The most popular videogame franchise in the world would have this level of scrutiny and detail put into all aspects of the experience. However, it's still mind-blowing how detailed Activision's matchmaking system and 'Player First' Call of Duty philosophy is. Calculating and assigning a skill value to a player and putting them in a "Skill Bucket" involves tracking performance across kills, the kill/death ratio, and who carried out those kills. No, you cannot lower your skill ranking by jumping onto live grenades over and over.

"Players of all levels should have a fun and competitive experience with the game. Team balance is the first and most important reason to track skill. If we don't know how we expect players to perform in a match, then we can't provide a balanced in-match experience for players. This results in blowouts, which we know are not fun for players on the losing end. We have found that balancing skill against other matchmaking factors quantifiably increases the extent to which most players play and enjoy Call of Duty. When skill is utilized in matchmaking, 80-90% of players experience better end-of-match placement, stick with the game longer and quit matches less frequently."

Matchmaking Series: The Role of Skill in Matchmaking, Activision.

Yes, ultimately, it's all about getting players to come back - and if this report is anything to go by, Activision and its many studios working on Call of Duty are masters of creating the sort of environment and experience that fosters precisely that.