These days, you don't have to be an IT professional to know the term "memory leak." For gamers, memory leaks are often associated with game lag, or even crashes. But what is a memory leak? What causes them, and how do we deal with them? It’s time for terrible Excel analogies!

Let's imagine that we have an Excel sheet in which each row is one game object, and each cell of that row is some element of this object. A cell can contain something like an HP value, a character's name, or a weapon texture. It can also contain a link to other objects, which in our example would just be the row number.

Fig. 1: Amiri object (row 1) refers to the Ginormous Sword object (row 5) in cell D1.

At some point, the object may become unnecessary, so we would like to free the memory it occupies by clearing its row. In programming languages without garbage collection, such as C++, this has to be done manually. At some point, someone has to say, "This row is no longer needed." In such languages, memory leaks often occur when the programmer forgets to do this for some reason, and the object continues to occupy that memory area indefinitely. If there are lots of these "forgotten" objects, we will eventually run out of memory. (After all, to use our example, there are only 65,536 rows in an Excel 97 spreadsheet!) The next time the game tries to find a place for another character or object, it will crash.

On the other hand, however, if the object is deleted while it’s still needed elsewhere (there’s a link to it in some other row), then the game may also crash when it tries to get the Ginormous Sword using this link and instead finds . . . an empty space at best, and at worst, a goblin.

Fig. 2: Now Amiri object (row 1) refers to Nok-Nok (row 5), and tries to use him as a weapon. Unfortunately, the computer is a bit more scrupulous than your DM at the table, and won't allow Amiri to do that.

Currently, there are many tools that help to both find and prevent such coding errors within a program being run. To do this, the tool simply has to force the program to tell us when the object was created, and when it was deleted. Then, at any time, we can see if there are unnecessary objects in the memory that shouldn’t be there. If there are such objects, we should deal with them using the traditional Russian questions, “Who is to blame?” and “What is to be done?”

However, for programming languages like C#, in which our game is written, things are both simpler and more complex.

C# has a feature called a "Garbage Collector.” This respected digital gentleman carries a big broom. From time to time he searches our spreadsheet, looking for the rows no longer referred to by a cell, so he can sweep them away. This approach eliminates the two problems previously listed: first, objects that are no longer needed are deleted, not forgotten. Second, no object can be deleted before the last reference to it is gone!

Fig. 3: Amiri (row 1) has thrown that goddamn Ginormous Sword away, and is now using the Flaming Two-Handed Sword +1 (row 7). The Ginormous Sword has been removed from the spreadsheet, and row 5 is ready to take on something else.

However, as old Heinlein used to say, "TANSTAAFL!” Such convenience has a cost: the game is hung up every time the Garbage Collector searches the spreadsheet. If it only takes a couple of milliseconds, that's great, and the player won't even notice. But spreadsheets can become huge in size, making the Garbage Collector’s work very noticeable, and causing some frustration for the player. Therefore, this digital gentleman does not work all the time, but only periodically, such as when the spreadsheet runs out of free rows. (After the Garbage Collector has gone through the spreadsheet, new empty rows will appear so that new objects can be created.)

A keen-minded reader will surely have noticed that no one refers to Amiri in our spreadsheet, and may wonder why the Garbage Collector has not deleted that object. This is what we call a chicken-and-egg problem. There’s an object or objects somewhere that no other object refers to, but which cannot be deleted, otherwise the spreadsheet will become empty! In the systems with garbage collection, such objects are called "roots" - they are marked so that the Garbage Collector does not delete them. For simplicity, we further assume that the root is the first row in our spreadsheet.

Furthermore, although the Garbage Collector eliminates the possibility of these types of memory leaks, other versions of the problem still remain. An example of this is when an object is no longer needed, but is still referred to by some other object.

Fig. 4: Amiri (row 1) no longer refers to the Ginormous Sword, but refers to Quest Number 5 (cell I1), which failed and (for some reason) refers to the Ginormous Sword. We should have deleted this Quest from Amiri's memory, but we forgot.

The Garbage Collectors have different features, and the one in the Mono platform used by Unity has a particularly problematic feature. It does not tell anyone after it deletes an object that is no longer needed, and forcing it to disclose information about this event is impossible. This greatly complicates the search for memory leaks when using this method, because although we know at what moment the object was created, we do not have the slightest idea about when the object was deleted. As a result, we have no way to know which objects are live at any given time.

There are built-in memory analysis tools in Unity that take a different approach. These tools compile and store a list of live objects at a given point in time, and you can simply look through the list to see if anything there is no longer necessary. In addition, it’s often useful to compare these lists at two different points in time, such as before loading a saved game from the menu, and after returning back to the menu from a loaded game. By doing this, we can easily see whether or not the game released all the objects that were clearly not needed in the menu.

Unfortunately, compiling this kind of list takes a long time, especially in Unity 2018. (The process was greatly accelerated in Unity 2019, but the amount of time needed to change the version of Unity for Pathfinder: Kingmaker makes it a nonviable option.) In fact, calling the process “long” is an understatement. In some cases, it can take up to ten hours in Unity 2018, and even in Unity 2019, it can’t be done for every frame.

That’s why we chose to go a different route. We decided to implement a leak detection method somewhat similar to the first one we discussed. To do this, we had to roll up our sleeves and get to work, plunging into the depths of Mono and Unity. As a result, however, we managed to make a tool that can show the game’s memory consumption in real time. (Well, more or less. To be honest, the game slows down A LOT when garbage is collected, but that’s okay because it only happens when our tool is running. Players shouldn’t worry about a slow down.)

For this method to work, there are two things we have to do:

1) Every time a new object is created (a new row in the spreadsheet), we remember the number of that row and the type of object that was created.

2) Every time the garbage is collected, we immediately repeat the virtual cleaning, but with our own code, which goes through all the rows remembered in Step 1, and deletes everything from the list that is no longer referred to by another row.

To put it simply, we do all this extra work just to receive a message that the object (row) has been deleted. However, this additional effort is important, because it’s the only way to get that necessary information.

Fig. 5: The created tool makes the graph of memory consumption visible throughout the entire operation of the game, and any time interval can be explored without the need to compile lists of objects at specific points in time.

This new tool will be used to combat memory leaks in our future games, and we hope that it will make it easier and faster to detect them.

If any of our readers are Unity developers, either professionals or enthusiasts, who might find this kind of tool useful, you are welcome to download its source code and compiled versions for Windows in our new section on Github!

Owlcat Games programmer Max Savenkov