Xbox Chief Technical Officer Scott Van Vliet has followed up on the announced fix for yesterday’s Xbox Network issues by explaining the cause and what the company plans to do to ensure it doesn’t happen again.
The root cause per Van Vliet was a licensing service that sites outside Xbox’s own network, affecting all manner of licensing checks for sign ins, launching games, buying games and even just viewing your full library of games. Why that’s an external when Microsoft is one of the largest server infrastructure companies in the world is a little bizarre.
Van Vliet then explains how traffic was routed to healthier parts of the network infrastructure to bring service back to people as soon as possible, leading to the mixed reporting from Xbox gamers over who could and couldn’t play games.
Going forward, Van Vliet will look to stamp out single points of failure like this and, quite importantly, be more transparent about what’s happening when something does go wrong.
One of the early factors was that the Xbox Status page cheerily claimed everything was running well, even if it retroactively identified 4:10AM BST as the start of the issues, and even then the status page didn’t mark gaming as affected, when the social accounts were admitting that it was. But I’d add a further and deeper issue that should be rooted out as we get closer to an all-digital future: that Xbox DRM is still far too restrictive if it’s also able to knock out already downloaded games for users on their chosen ‘Home Console’, and even physical games. A change in 2022 reduced the number of DRM checks for games, but this should go further.
Scott Van Vliet in full:
Earlier today, many players began experiencing issues with sign in, enumerating their game libraries and playing games. The team has worked hard throughout the day to fix the issue, and we’ve restored service as of 2:30 PM PDT. Nonetheless, this is an unacceptable situation, and we must do better to support you.
Over the coming weeks, you’ll hear more from me about our services, our platform, and the work we’re doing to make XBOX better. And that starts with some transparency about this incident.
What happened. Late last night, a licensing service that sits outside of XBOX, but which XBOX depends upon, began failing. This caused some sign-in scenarios to fail, and it caused many scenarios that require an entitlement check to also fail, such as listing out your Full Library and launching games that you own. It also affected several of our publishing and store partners who depend on those same systems which is why some of you saw issues in specific games and not others.
What we did. Our on-call teams caught the first signals overnight through automated monitoring and declared a major incident. Once we isolated the failing infrastructure, we shifted traffic onto the healthy parts of the system while the team continued to work towards a root cause. Service came back unevenly across regions which is why some of you recovered hours before others.
What’s next. We’re running a full post-incident review. I care less about the one-line root cause and more about the real questions: why a failure in one service was able to take down this much, why recovery took as long as it did, and what we change so a single point of failure can’t ruin your night again. That means hardening the dependencies underneath sign in and game launch, improving how we detect and contain this class of failure, and being faster and clearer with you when something breaks.
Thank you for your patience, and for the reports that genuinely helped us narrow this down. We’ll do better.
Source: Scott Van Vliet
