Shadow of Cerberus EVE Community Forum

The War in Fountain - The Z9PP-H Node Crash

Everybody should be aware that there was a big-ass battle today in Z9PP-H in Fountain. It was all over the Twitters and forums and such. It was a battle that kept escalating for nearly six hours, at one point reaching 2200 pilots in system.

I was made aware of the battle about 4 hours into it. I figured it probably wasn't going to last much longer, and since it was three jumps from B-DBYQ, figured getting to it would be difficult. Running the gauntlet of gate camps, plus 10% TiDi. So I watched for awhile on Mad Ani's Twitch stream. He had 5000 viewers at one point and made it to the Twitch main page.

Z9PP-H was a fight that was reverberating across the gaming sphere. It was an Asakai level event, and it was drawing a lot of good attention to CCP and to EVE Online.

After about 45 minutes of watching the stream, I decided I needed to try to make it to the fight. Before logging in, I figured my best bet getting there unscathed would be in a stealth bomber. I'd bring a torp bomber. It would do good damage, if I got there alive. So I logged in. I needn't have bothered worrying about how to get to Z9PP, because there was a reinforcement fleet forming up. So I hopped into a Megathron and waited.

While formed up on a titan for a bridge, news came down that the Z9PP node had crashed. I tweeted:

Z9PP crashes. #tweetfleet

I just assumed that the node crashed due to load. Z9PP was not reinforced. These battles are usually organic, so there's really no predicting when they'll happen, thus submitting for reinforcement is usually a crap-shoot. Multiple timers are happening daily; most result in small fights; some result in no fight at all. You never really know if you're going to need it, and nobody wants to request reinforcement unnecessarily.

CCP, a lot of whom where watching the battle unfold from Iceland, decided to try to reinforce Z9PP live. Unfortunately, they made a boo-boo, as CCP Explorer admitted via Twitter:

Human error, forum thread incoming; we are very sorry for this mistake :(

I felt that was a brave admission. CCP could have just stayed mute and let everybody assume a load crash, but instead owned up to their mistake immediately. I wrote in their defense:

I give CCP credit for admitting human error. They could've said nothing and let us assume it crashed due to load.

As it turns out, they couldn't stay quiet, let us assume a load crash. The server message that appeared on the screens of those in Z9PP explained away the root cause of the crash:
CCP were trying to remap the node. And, well, they fucked up. Not only did they commit a spelling crime, they crashed their biggest bit of PR since Asakai. People don't decide to try EVE Online because of radial menus, they decide to try EVE when they see and hear of battles like this on the internet. Obviously CCP did not crash the node on purpose; CCP is not in the habit of shooting itself in the foot. (Besides, it is well-known that CCP will only crash a server to benefit the CFC, in this case it benefited Test Alliance. Obvious mistake is obvious.)

The tinfoiling is amusing as heck. Not for a minute do I suspect any foul play by CCP. They were simply trying to improve the battle, and they accidentally fucked it all up. Hell, I was going to poke fun at the tinfoil myself by photoshopping CCP Dolan turning off a server. (BTW, do not Google image search Michael BoltonIII, there are some very disturbing images out there of CCP Dolan, of which this is one [nsfw]. Yes, I am sharing my pain with you.)

Before CCP could get the full information into the public sphere, outrage was happening. Some comments by Darius Johnson were immediately interesting. He used to be CCP Sreegs, so he has some inside knowledge into how things work:

(6:58:56 PM) darius_johnson: you know you can't do that by mistake right?
(6:59:07 PM) darius_johnson: like it's not something you just do by misclicking

And then Mittens pinged the coalition:

Alright kids, that's the second time a hot node remapping has saved TEST in a fight. TEST has been ordered not to log back in, so they got away clean instead of losing their entire capfleet. We're going to grit our teeth and reach for our flaying knives, for now stay active in fleet until we figure out what just happened.

Needless to say, I am quite displeased, but I guess we'll just have to sate ourselves with 300+ BS kills, 30+ carrier kills, and fury. That was a fight bigger than Asakai, and CCP just ... stopped it.

*** This was a broadcast from the_mittani to all-all at 2013-07-04 22:55:06.751276 EVE, replies are not monitored ***

Even Montolio was annoyed (even though Test Alliance benefited):

This is fucked up CCP. I don't care if my side benefited this time, you've been doing shit like this the entire war and messing up the natural course of events for all sides.

You need to have some serious reviews to prevent these problems from impacting the sandbox gameplay you tout as a core feature.

Not long afterwards, CCP published their explanation of what went wrong:

We?d first like to explain the situation, and what exactly has happened so far.

Owing to the heavy load in Z9PP-H, caused by a fleet flight involving 2200+ pilots deeply embroiled in the Fountain War, the decision was made to remap all other systems away from the node on which Z9PP-H was located in order to improve responsiveness and playability. We were later than we wanted to be to try to reinforce the node (obviously) since it was a more organic-type battle without a fleet fight notification. Once we fully assessed the situation and what was at play, we figured it was time to go for it and the remap in preparation of what was set to be "Asakai 2.0".

This was put into play at 22:14 UTC, however due to an error in communication, all systems including Z9PP-H itself were moved from the node, which caused everyone who was in the system and was involved in the ongoing fleet fight to be disconnected from Tranquility.

The system of Z9PP-H was brought back online and made accessible as fast as possible, however we realize that the mass disconnect of everyone involved in the fight means that ships are potentially saved by their pilots not logging back in once they have been disconnected, or are lost due to other unforeseen consequences of the disconnect.

All pilots involved have our sincerest apologies for this error, and we are currently looking to review our policy of action in these types of situation to ensure that this is not repeated.

C'est la vie. Very unfortunate turn of events. There's no point dwelling on it too much, other than to make sure CCP implements some procedures to ensure that it never happens again. They screwed themselves as much as they screwed the players in Z9PP that day. This is going to be a bit of a black eye in the press for CCP, some press they'd rather not have.

So the war in Fountain will continue on. It could have very well ended today. Test Alliance were already down 300 battleships and 30 carriers. They had another 70+ carriers on field at very great risk of being lost. Many more battleships as well. If this fight had carried on to its expected conclusion, Test would have been decimated. Even if they could have afforded such a devastating loss, replacing that number of ships and getting them to the front line quickly? Crazy difficult. Replacing 100 carriers on short notice? Exceptionally difficult. Replacing 400 or so battleships? No small task. Test's ability to defend Fountain in the short-term would have been severely hampered.

The day wasn't a total loss. I did get this sweet killmail on the way over to Z9PP. Missed the big battle, but sometimes things still turn up roses..
Source: The War in Fountain - The Z9PP-H Node Crash