Education time! What happens behind the scenes when the servers need updated?

[Landshark]

Let me start off by saying that I've worked for Xbox Live at Microsoft for about 2 years. I worked as a Change/Release Engineer, which means I've worked with, quite literally, thousands of updates that have been implemented on the Xbox Live servers. These include game updates, patches, security fixes, etc, etc, ETC.

I thought I'd help describe what the process is for bug fixes and how it affects servers in a large online game environment, similar to what Defiance is being hosted on. I find this stuff fascinating and fun and maybe somebody out there will too.

Preface:
Server downtime is spent applying the fixes that were created in their test environments. What I mean is, servers aren't taken offline in order to create a fix, they are only taken offline to implement the fix. The fixes are created in a test environment. There are rare exceptions to this, of course.

This is how it usually/generally happens, and yes this is going to be detailed:

1) A bug is reported and logged officially in the back-end DB.
2) A PM (project manager) for that project/game gets the bug report list and discusses the changes with their CM (change manager) or RM (release manager) (and often the Change/Release Engineer, which is the job I have) to be prioritized.
3) The bug reports are then sent to the dev/test teams to confirm the bug and create the fix in their test environment (which is usually a very closed off environment that does not touch any outside internet)
4) The fix for the bug is reported to the PM (who has been tracking the progress the whole time).
5) The PM takes the fix and reports back to the CM or RM to schedule a time for the fix to be applied to the 'other' test environment, which is connected to the internet, but only very specific people/computers can access this network.
6) The CM or RM works with the Change/Release engineer(s) (this is the job that I have) to implement that fix into the TEST environment.
7) The test/dev teams (or in many cases the Change/Release engineer themselves) then confirm the fixes applied work correctly. Assuming they do work correctly, the next step is......
8) The CM or RM (or both) have a meeting (depends on the company, ours were weekly, but I imagine Trion is having these daily right now) discussing ALL bug fixes that need to be applied to the LIVE environment (LIVE meaning the game servers that the public connects to) and figure out which ones have priority and how to schedule them.
9) The CM or RM works with the Change/Release engineer(s) (this is the job that I have) to schedule the implementation of the fixes into the LIVE environment.
10) The Change/Release Engineer implements the fixes into the LIVE environment.
--This last step (step 10) is where the game servers are often taken down by the Change/Release Engineer. Depending on the server setup, they will take a subsection of servers out of rotation, apply the fix (rebooting them or keeping them off as necessary), then return the fixed server back into service before taking down the next set of servers.

Anyway, there's a small glimpse into how a game servers and fixes work.

[TheKidiKehT]

This is definitely appreciated knowledge. I knew the task of bug fixes and such were complicated, but wow. And I'm taking a guess that your list there is super simplified as to what all is completely going on at Trion's quarters.

[timothybeck1]

Very cool and detailed info. Thanks for describing the process!

[IIIZuLuIII]

I thank you for the time and info. Great to know a little more. Could you shed some insight on trions situation? If the studio is using a closed internet, how can they effectively know if the patches are working?

[Warlander]

Good job, sadly many will ignore it and continue to complain they want instant fixes and how unfair there lives are.

[Landshark]

This is definitely appreciated knowledge. I knew the task of bug fixes and such were complicated, but wow. And I'm taking a guess that your list there is super simplified as to what all is completely going on at Trion's quarters.

I thank you for the time and info. Great to know a little more. Could you shed some insight on trions situation? If the studio is using a closed internet, how can they effectively know if the patches are working?

Yes, the list above is super simplified. Of course, the list I made above is also generalized as each company does it a little different. However, in the software/game industry there are some very strict standards on how these processes are handled and MS, being a world leader in software, follows some of the most popular processes (actually thousands of companies look to Microsoft to learn how to do it).

I always see people making comments about server issues and how they'd want to work in the game industry, or how they could do things better, but very very few get a glimpse of how it really works. Honestly (and I humbly admit) I get jaded a lot when reading people's comments about how company XYZ sucks because of 'server issues', etc. There's so much going on that it's never a simple fix and it's hard to communicate that to the general gamer/public.

[S1L3NTKiLLaH]

Maybe we should somehow get this guy in there, maybe then they can get it stable...

[Taaltos]

Oh, just saw this, so will post here. Also, here's what a sample of code from a server looks like.

public class SampleServer
{
// instantiate a ServerManager object
public ServerManager manager = new ServerManager();

public SampleServer()
{
manager.setOPEndpointUrl("Http://my.openidprovider.com/server");
}

public String processRequest(HttpServletRequest httpReq,
HttpServletResponse httpResp)
throws Exception
{
// extract the parameters from the request
ParameterList request = new ParameterList(httpReq.getParameterMap());

String mode = request.hasParameter("openid.mode") ?
request.getParameterValue("openid.mode") : null;

Message response;
String responseText;

if ("associate".equals(mode))
{
// --- process an association request ---
response = manager.associationResponse(request);
responseText = response.keyValueFormEncoding();
}
else if ("checkid_setup".equals(mode)
|| "checkid_immediate".equals(mode))
{
// interact with the user and obtain data needed to continue
List userData = userInteraction(request);

String userSelectedId = (String) userData.get(0);
String userSelectedClaimedId = (String) userData.get(1);
Boolean authenticatedAndApproved = (Boolean) userData.get(2);

// --- process an authentication request ---
response = manager.authResponse(request,
userSelectedId,
userSelectedClaimedId,
authenticatedAndApproved.booleanValue());

if (response instanceof DirectError)
return directResponse(httpResp, response.keyValueFormEncoding());
else
{
// caller will need to decide which of the following to use:

// option1: GET HTTP-redirect to the return_to URL
return response.getDestinationUrl(true);

// option2: HTML FORM Redirection
//RequestDispatcher dispatcher =
// getServletContext().getRequestDispatcher("formredi rection.jsp");
//httpReq.setAttribute("prameterMap", response.getParameterMap());
//httpReq.setAttribute("destinationUrl", response.getDestinationUrl(false));
//dispatcher.forward(request, response);
//return null;
}
}
else if ("check_authentication".equals(mode))
{
// --- processing a verification request ---
response = manager.verify(request);
responseText = response.keyValueFormEncoding();
}
else
{
// --- error response ---
response = DirectError.createDirectError("Unknown request");
responseText = response.keyValueFormEncoding();
}

// return the result to the user
return responseText;
}

private List userInteraction(ParameterList request) throws ServerException
{
throw new ServerException("User-interaction not implemented.");
}

private String directResponse(HttpServletResponse httpResp, String response)
throws IOException
{
ServletOutputStream os = httpResp.getOutputStream();
os.write(response.getBytes());
os.close();

return null;
}
}

That's just server side, not game/client side.

[Zehka]

This guy is so awesome; I gave him a Sporeshot. As well, there are other things that could be at; worse case scenario, hardware failure.

[Landshark]

This guy is so awesome; I gave him a Sporeshot. As well, there are other things that could be at; worse case scenario, hardware failure.

Hey Zehka and thank you! I forgot to claim that sporeshot. I'm about to head to bed..it's like 1am here.