Legal

CCP Games

Media

Enter a living sandbox rife with challenge and opportunity

EVE Frontier

Discover

Build

Founder Access

LOG IN/ REGISTER

Upgrading to Python 3

Moving into the Future

We have just gone through a monumental moment in CCP’s history:

the very first (and likely only) major Python version upgrade. This achievement has also incorporated a staggering twelve minor versions - sixteen years worth of updates - in one jump.
 
 
Before we dive into what this has entailed, a special thank you to the talented engineers who made this possible across Platform and Team Gadget: your work has been truly heroic and we will be celebrating at Groska im sure.
 
 
If you’re not technically-minded, take note: this is a big change. Huge even.

EVE Frontier, like EVE Online, is based on the Carbon Engine. Carbon was developed as part of EVE Online and taking it into a separate project that supports multiple products posed a challenge in and of itself. 
 
 
Using Carbon and its framework has allowed EVE Frontier to stand upon years of excellent space video game technology but also landed it with years of technical debt. 
 
 
For example, Carbon originally ran a fork of Python known as ‘Stackless Python’. This was very useful in the days of old as it allowed developers to create massive amounts of tasklets (a way to run functions concurrently without using threads). This decision is somewhat controversial but at the time made sense for the initial development of EVE Online. Later, vanilla Python 2 would get this capability via gevent plus greenlets and Python 3 would continue this with asyncio as part of the standard library. 
 
 
Over the years, CCP has been a major contributor to the Stackless Python fork but it has been stuck on a maximum of Python 3.8 for some time now, which gave us two options if we were to continue the development of EVE Frontier as a game for the next 20 years (and beyond):
 
 
 1.) Maintain Stackless Python fork ourselves
 
 
 2.) Move towards vanilla Python 3

We chose option 2, giving us two immediate tasks: upgrade to Python 3 and move away from Stackless. Initially it seemed sensible to upgrade to Stackless 3.8 and then move away from it, we did begin work in this area. However as we soon realised, at no point would Stackless 3.8 be in a shippable state to underpin our efforts. 
 
 
Around 4 million lines of Python code and 1 million lines of C++ code make up the EVE Frontier client and server. Many of these needed to be updated or completely changed. 
 
 
Python 3 is a major upgrade but contained a number of backwards-incompatible changes. Many of these have a very simple migration path, such as changes to print statements which transforms to print "hello world!" to print("hello world!") . However others were more complicated, such as division changing from being an integer division to a floating-point division (in this case we can, at the cost of code readability and a small performance cost, do a safe transformation).

In addition, other challenges arose: some things no longer work. For example you can’t compare two types that aren’t comparable. This had historical reasoning but in our development we have come across code where sorting list of tuples sees the second element being not comparable. We also have C++ code that is exposed to Python and that interface changed quite a bit between Python versions. For example any code written and compiled for Python 2 does not work for Python 3. 
 
 
These and similar issues have already been found and fixed, however there may be more lurking in code that is seldomly executed or only executed in a production environment.
 
 
The networking module also proved particularly challenging. In the past we had made significant changes to Python’s socket module for enhanced performance. As the language has evolved we do not require those modifications anymore but we needed to modify the ssl module to be compatible with our un-modified socket module. We were left with a choice:
 
 
 1.) Maintain our own version of those modules
 
 
 2.) Make the socket module compatible with the standard socket module so we don’t need to modify the ssl module.
 
 
We opted for long term maintainability via option 2 at the cost of requiring us to make extensive modifications to our network stack.

At this point we were still not out of the woods in order to move to a version of Stackless Python later than 3.8. We considered a move towards Python’s asyncio, but this would have required a major modification to our code due it being conceptually different. Asyncio requires you to specifically mark functions that can yield control but our stackless implementation did not. We did not see a clear path of doing that upgrade so we opted for another route. We decided to go with greenlets but without gevent (owing to the fact that gevent is written in Cython and since we do interact with the scheduler through our C++ layer it was a poor fit). To solve this hurdle we wrote our own scheduler.

Due to the sheer amount of changes in the codebase the potential for conflicts was great. If one person modified a line of code to make it Python compatible and another modified it due to changes in gameplay the resulting conflict needed to be manually resolved. While not the most complex fix, it is highly manual and the risk of resolving it incorrectly always loomed near.
 
 
Estimating performance improvements in a complex product such as Frontier is difficult. When we do measurements on certain code paths we can see improvements in some and regression in others. However, what the net effect it has depends on how often those code paths are executed. 
 
 
You could get some numbers if we had consistent load before and after the Python 3 upgrade but that is not the case for Frontier. That being said, what we can clearly see in our numbers is that when we measure more complicated code paths, such as multiple users logging in or weapons being fired we are seeing between 10% to 30% performance increase.
 
 
An impressive result, especially when considering we are still developing EVE Frontier itself in a live environment! You could say we have been replacing the engine while the car is running. A massive undertaking the payoff is huge: EVE Frontier is now running smoothly on Python 3.12, setting the stage for all future development.

Moving into the Future

Keep exploring

dev blog

A First Look

dev blog

The Vast Frontier

dev blog

Welcome to EVE Frontier