--Update 12/05--
Hey Everyone,
Following our update on Friday, we're back with news!
Live Hotfix 12/05
This weekend's Insiders testing has generated sufficient data to verify the root cause of the issue affecting Diving to Experiences and our team has prepared a targeted fix for the Live game. We've rolled this fix out behind the scenes this morning into Live and have been monitoring the improvement in successfull server migrations as a result.
We'll be taking the game down for a short maintenance today (12th May) to re-enable Faction Battles and flush this hotfix out to all servers before bringing the game back up again.
The team will continue to monitor Diving as this fix settles into the Live game.
Looking Ahead
It's worth us stating here that the system managing server migration is a complex one, and there are many real scenarios that can cause a migration to another server to fail. This system often runs at ~98% success rate for migrations across the whole game - and while that is considered a pretty stable experience, migrations may still fail at times. The critical step here though is that in addressing the root cause of this issue today, crews should no longer become stuck on a black screen when diving to an experience and if their migration should fail for whatever reason - they will be returned to the world safely.
While investigating this recent Live issue the team have identified further improvements to the system to better protect from future issues like this impacting the experience. Given the critical nature of this system, we'll be scheduling this work as a priority area behind the scenes.
Thank you again for your patience as we've worked to address this issue, we appreciate the impact this has had on your experience over the last few weeks and we'll be taking steps to harden this system for the future.
Drew 'Sonicbob' Stevens
--Update 09/05--
Hey Everyone,
Following our update last week, we wanted to jump back in here and share an update on the steps we're taking to address the recent Live issues affecting Diving to Experiences.
Update from development
This week the team has been digging further into the migration system and working to identify the root cause behind players becoming stuck on a black loading screen after diving.
This is a pretty complex system that handles a wide range of scenarios across the entire game, but the team has used the Internal, Insiders and Live player data to hone in on the problem area.
As it stands today, we now have confidence that we've discovered the root of the issue and the team is working on a fix. We'll be using Insiders this weekend to monitor migration behaviour and to further validate this as the root cause before we prepare to bring a fix to the Live game.
Addressing the issue in Live
As we approach the launch of our upcoming Season and our release onto Battle.net®, our priority is to stabilise the Live experience ahead of time and ensure we have a steady foundation to launch from.
In order to do this, once we've built confidence from Insiders, we'll be scheduling maintenance and delivering an update that directly targets addressing this issue and in turn will also reenable diving to Faction Battles. We plan to target this early next week (Week commencing 12/05) and will confirm our timing as soon as we have those plans locked in.
I appreciate that we've taken the game down for maintenance multiple times since our last update and have had an extended window where Faction Battles have been disabled. However, we feel strongly that addressing this issue as our priority is the right approach and allows us to stabilise your experience in Live first and build our confidence ahead of our other updates scheduled for later this month.
Festival of Fishing
As our Live game has been impacted by the issues experienced with diving and recent downtime during the Festival of Fishing, all players have been provided an additional Hunter's Rare Fishing Map and Merrick's Recipe Request Voyage to use before the event ends on May 12th.
Thank you all for your patience as we've worked through these issues and we understand the impact that this has had on the wider game during this time. We'll share more on the details of our planned hotfix early next week, so stay tuned on our channels for more info as those plans get locked in.
Drew 'Sonicbob' Stevens
---Original Post---
Hey Everyone,
Following our April update on the 24th, we delivered a further hotfix this week to provide some improvements and address a number of high impact issues in the Live game. You can read more on the specifics of this hotfix here (3.3.2.1)
While this hotfix improved key areas of the experience, many were disappointed to find that the recent 'black-screen' loading issues encountered when diving to an experience were not fully resolved as part of this update. I wanted to jump in here and share more information on this issue and the steps we're taking to address this.
Diving To An Experience
In April's update we saw crews diving to an experience from the Quest Table or to a Faction battle from the War Map experienced both extended wait times and stuck loading screens.
After investigation the team discovered that server migration was failing behind the scenes in some scenarios, leading to players becoming stuck unable to exit once they began.
As this system manages the migration of crews across a range of different scenarios, we saw all of these player experiences affected such as Voyages, Tall Tales and Faction Battles.
We monitor server migration behind the scenes and saw our migration success rate drop from our usually very stable ~98% down to ~88%. While that doesn't sound like a large jump, given the sheer scale of migrations going on across the game at any time this is a significant impact.
The team identified a clear migration problem after reviewing the player reports and we prepared a fix to be delivered as part of the 3.3.2.1 hotfix. During internal testing we validated the improvement in migration, however knowingly at a significantly smaller scale of play. Players likely saw us call this fix out specifically as part of our social comms earlier in the week.
When we opened the game following the hotfix we saw players stepping into the game early still experience these stuck loading screens, however the scale of players affected was much less. Our overnight success rate is back up to ~93% but that is far from where we should be for such a critical part of the experience.
The team are back investigating the migration issues as a priority and i'll share more on our progress here when we have it.
Faction Battles
While I shared some success rate statistics above for migration all-up, the impact on Hourglass play has been much more severe. Due to the need for two crews to migrate successfully for the majority of battles, the success rate here is much closer to ~50% which is resulting in a significantly impacted experience.
Again while our hotfix provided an improvement, Faction Battles continue to be heavily impacted and in our eyes are not at an acceptable quality play experience right now.
We've therefore made the difficult decision to protect players from this poor experience in Faction battles by temporarily disabling diving into a battle which will take effect as part of a short maintenance window today (2nd May). Crews will find they are automatically returned to the seas after diving and these Faction Battles will remain disabled through the weekend until a resolution to the issues can be delivered safely.
As above, the team are prioritising efforts here to address the root cause of this migration issue in order to return Faction battles as soon as possible.
Kraken Encounters
In April's update the Kraken Encounter moved from a 'Main Event' in the World Event schedule to allow it to be present in the world alongside other events. This change would improve the rotation of world events, and provide a number of behind the scenes improvements to how servers are dynamically populated.
While this change intended to retain the same Kraken spawn frequency as before, it was very clear that was not the case in Live!
As part of today's short maintenance window (2nd May) we are rolling back the Kraken changes made, and in turn curtailing its recent aggressive spawn frequency. We still intend to make the previously shared improvements to this encounter, however will work on addressing the encounter frequency before you'll see this return.
Thank you for your patience here as we've worked to address these issues, the team are hard at work digging into the ongoing migration issues so that we can reenable Faction Battles and return to a more consistent diving experience as soon as we can.
Drew 'Sonicbob' Stevens
