Sony have tweeted that they have now fixed the glitch that allegedly bricked consoles. They also say that it wasn’t bricking them, instead it was getting them stuck in an infinite crash loop.
We've since fixed the issue, and it wasn't bricking consoles, just sending them into a crash loop that can be quickly fixed in under 5 minutes. Delete the message on the PS mobile app, go into Safe Mode, use Option 5, console back to normal. ^DB
— Ask PlayStation UK (@AskPS_UK) October 15, 2018
I didn’t notice any hotfixes for my PlayStation downloading last night so maybe they’ve done something behind the scenes.
Source: Twitter
camdaz
That didn’t take long.
tonyyeb
Shocking considering Sony’s track record!
MrYd
It might not actually be fixed. They’ve suggested a way to recover from it if you’re affected, but even that is reported to not always be successful.
It looks like someone on that 1 Twitter account might be confusing “here’s a possible workaround” with “it’s been fixed”.
I’d leave messaging (and friend requests) turned off for now.
tonyyeb
Other tweets from the account suggest it has been fixed, not just the workaround that could be used before the fix:
“Hi there! A background fix has been deployed, however if you encounter any issues don’t hesitate to get in touch. ^NL”
“Do you mean the malicious message causing consoles to crash? We identified the problem and applied a fix, working towards making sure it doesn’t happen again. Are you currently experiencing crashes due to it? ^DB”
“No, nothing like that. Some rascal discovered a way to cause crash loops on consoles with a weird message. We’ve since fixed that and we’re looking into preventing anything like this happening again. ^DB”
MrYd
“we’re looking into preventing anything like this happening again”
That suggests it hasn’t really been fixed. Just this particular case.
It may well be they’ve managed to filter out the messages currently causing issues on the server side. But give it a couple of days and there’s a good chance someone will find something similar that does the same thing. And then they can filter out that. And the whole thing repeats forever.
A firmware update would solve the problem permanently. No need to filter out dodgy characters in messages if they can’t do any damage.
And given that the “delete message in app and rebuild database in safe mode” fix doesn’t appear to be 100% reliable, I’m not going to trust 1 of Sony’s Twitter accounts just yet.
tonyyeb
Or I read it as ‘we’ve stopped messages causing this crash, now we are going to fix the thing they are triggering to make the PS4 crash’…
Data validation is always king when dealing with free text boxes, ask any half decent database developer.
MrYd
I think we’re agreeing here. They’ve stopped the current problems, but haven’t fixed the underlying cause yet, which probably needs to be done with a firmware update.
Not convinced it’s down to data validation though. It appears to be more of an issue with what happens with valid but unsupported data. If it’s using some flavour of Unicode, with 100k+ valid characters, and something that can’t be displayed is causing the problem. It seems to be making some effort to deal with unsupported characters (they show up as a little rectangle), but something’s going wrong.
You can’t really check each message against a list of however many supported characters there are for all the languages the PS4 supports. You just need to make sure unsupported characters are handled properly.
Which is why I suspect any fix they’ve got in place already is just filtering out whatever causes it to go wrong. And someone will quite possibly find another character or combination of characters that does the same thing. And a firmware update is needed to fix the problem once and for all.
So not really a data validation problem, more of a data display problem.
tonyyeb
“You can’t really check each message against a list of however many supported characters there are for all the languages the PS4 supports. You just need to make sure unsupported characters are handled properly.”
You can, this is how every database should be setup. Checking for invalid data is part of my job. Spurious characters are how hacks (of banks, online shops etc) happen because someone hasn’t ensured the validation is correct. Sony will have changed what is allowed as a valid message before writing it into the message table.
I think the message issue is resolved by having better validation. Yes a firmware fix is required to stop the triggering of the crash. But they’ve bought themselves time by preventing the triggering.
MrYd
No, you can’t check for invalid data. Not 100% anyway. Some things will have strictly defined values you can check for at input. If it’s expecting a number, check you’re getting a number and don’t allow anything else. But a text field? When using something like Unicode and you’re supporting many languages with weird characters?
What you need to do is not trust the data. Assume it’s going to be potentially dodgy and make sure that it can’t do any damage. And absolutely do not trust your own ability to validate the data at input. You’ll miss something, and without anything in place to catch it later, it could go horribly wrong.
tonyyeb
“No, you can’t check for invalid data. Not 100% anyway.”
Sorry but you are wrong. You really think a bank allows anything to be inserted as a record when making a BACS transaction? You think Amazon allows anything to be inserted as a record when putting in your credit card details? This is PART OF MY JOB, I’ve done this for years for various systems. I’m a DBA and we deal with lots of data entry by untrusted sources.
“You’ll miss something, and without anything in place to catch it later, it could go horribly wrong.”
Which is exactly what has happened here. Sony are now clearly validating that data, they have learnt the lesson or their checking wasn’t already good enough. End of story.
MrYd
Credit card details fall under the category of things you _can_ check. They only consist of numbers, and they’re a certain length. One or more of the digits are a checksum so you can check the whole number is valid.
A big free text field is going to pose problems though when supporting all those thousands of characters and variable lengths for a single character. Unicode is a mess to work with and easy to have something sneak through.
And all we know Sony have done is possibly fix this one case. Hopefully they don’t think “We’ve fixed that, end of story”. Because otherwise it’s pretty much guaranteed to come back to haunt them. A string of text in a message shouldn’t be causing the PS4 to get into such a mess. They need to stop it causing problems when the next cunningly formed message comes along. And until that happens with a firmware update, keep those messages turned off.
tonyyeb
Nothing better than someone on an internet comments section telling you how your jobs works. Sadly there is no respect when it comes to anonymity on a forum. Everyone remind me never to bother commenting again, especially on this guy’s comments.
Andrewww
I find it quite embarrassing such a stupid developer’s mistake can still occur on devices in 2018.
It’s ok that at least they fixed it quite quickly, but it should never have occurred in the first place.
MrYd
I’m sure it’s embarrassing for them. It always is when you’ve spent all that time building something and someone comes along and breaks it.
I wouldn’t call it a stupid mistake though. It took 5 years for someone to find it and it’s obviously not a simple thing to find. Until it happens, then it’s suddenly “FFS! Why didn’t we think of that in the first place?”
The whole “Stupid developers! Shouldn’t have happened. Fire them all. Out of a cannon” thing annoys me. I’m sure someone’s calling on Sony to give some form of compensation for it as well.
Andrewww
Well, it probably depends on the sample you have to deal with and which you base your views on. Part of my job is to review code. I don’t want to go into any details here, but because of that, I’m very sceptical of the quality of code in general, I have to admit.
Given that here the problem is very similar to others that got extensive press coverage recently, bricking other devices, and especially as this is something you can test for automatically, it’s just plain embarrassing.
MrYd
Not as embarrassing as your whole network going down for the past 3 hours just as I was about to start playing Firewall tonight.
Andrewww
That is indeed very bad luck, you have my sympathy. So you’re fully sold for Microsoft’s always-on streaming future, I guess…?
For now, you could always settle for a relaxing chapter of a classic like Alien:Isolation, which is reliably working with no internet access required at all..!
MrYd
That was the only time this year I’ve not been able to play what I wanted due to the PSN going down. Usually it’s fine if you’ve logged in recently.
They’ve also just buggered up the 3 months of Netflix offer. Just sent me the email this morning, and I’m apparently not eligible.
I don’t think Sony are having a good week, really.