Migrating Orleans backend from Windows to Linux, AWS, and beyond

This is a draft of a post I composed in March, 2016. Only partially completed, but maybe it will be of some value to someone.

Our backend has been based on Microsoft Orleans (MS Research, GitHub) for some time. We're primarily running the entire backend- deployment and all- on Windows 7 mostly because:

1. It's convenient since all developers are running Windows on their desktops
2. It's easier for QA to test and operate playtests until everything is fully automated and we've got devops tools in place
3. Our company is mostly a "Windows shop" anyway

It's been a long time coming, but we've finally gotten around to looking at running the backend on Linux. A few components, namely Zookeeper and MongoDB are fairly straight-forward since they're arguably intended to run on Linux. That leaves Orleans itself and, of course, our own game server.

Orleans should "in theory" work fine on top of Mono. However, our Orleans projects are a bit of a mess, so I thought I'd just try running the binaries produced by Visual Studio. And shockingly (see "...debug everywhere"), it mostly worked; we only ran into the following problems:

• We use log4net and our App.config was specifying ColoredConsoleAppender which didn't work on Mono (switched to ConsoleAppender)
• We spawn our game servers using Process.StartInfo.UseShellExecute = true and caused an error "xdg-open: unexpected option ..." (we set UseShellExecute to false on Linux- Windows seems to require it)
• Our config file had a Windows "smell"
• It refused to startup with the version of mono that ships with Ubuntu 14.04, but installing the latest, stable version fixed that

Next came getting our game server to compile and run on Linux. Luckily, our WAF-based build system should "just work" on Linux (in theory). Most of the compilation and runtime problems fell into just a few categories:

• Hard-coded windows specific paths and commands
• Paths that were incorrect with a case-sensitive filesystem
• Asserts that were triggered by incomplete implementations

N.B.: At the time of writing we were using CryEngine 5.0. Since then the WAF buildchain has been retired in favor of cmake

Our initial round of deployment scripts where based on PowerShell 3.o and left a lot to be desired. PowerShell 3 (as opposed to 1 or 2) was chosen because it provided disconnected sessions allowing us to execute things on a remote machine and leave them running in the background without leaving a shell connected (or installing as a service). But, PowerShell 3 isn't installed by default on Windows 7 and then some other stuff is needed on remote machines to get remote commands working at all. Generally speaking, it was the overall idiosyncratic behaviour (Invoke-Command takes PSCredentials but old commands like "net use" don't- and there's apparently no modern replacement), and unfamiliar syntax (passing ArgumentList to Invoke-Command's ScriptBlock) that made the PowerShell prototype take longer than I expected. And there's still a few things that just don't work. Granted, I'm a completely new to PowerShell, and like all advanced tools there's a nontrivial learning curve. PowerShell is without a doubt leaps and bounds better than cmd.exe, and some aspects like getting auto-complete for commands and arguments in the ISE were welcome.

In a completely unfair comparison (given that I have non-trivial Linux experience), we were able to fully automate deployment to Ubuntu in under a day. The frustration of Invoke-Command/Credential/InDisconnectedSession was replaced with the relatively straightforward use of ssh and nohup. And, once you get password-less ssh and sudo setup, and embrace other commands like screen, working with multiple remote machines is a breeze.

All-in-all we were up and running on Ubuntu 14.04, rocking nightly builds in Jenkins, deploying to AWS, and doing playtests within a week. *opens a beer*