NodeJS Architecture III: Memory management and Garbage Collection

In the previous post, we understood the main V8 components and their respective responsibilities. Now, we will explore a more distant part to understand how V8 manages memory and performs garbage collection in NodeJS.

Every variable, function, array or value that we generate in our application, needs to be stored in a temporary place until its execution ends. This place is called memory. Imagine this space like a high school locker, where each student has their own place to store books and materials. All spaces are addressed so that students can navigate and find their respective spaces. Superficially, we can say that the memory we use in our software works in the same way. We store a value in a specific space and we can retrieve it later through its address.

The memory mentioned before as a component that integrates our software is RAM, or Random Access Memory. We can divide this memory into two main parts: the Stack and the Heap. To understand why this division exists, first we need to understand the characteristics of each one and how they are used in the compilation process.

Before diving deeper into the memory Stack, let’s first understand what the data structure it implements represents. If you are not familiar with data structures, you might not know some common structures that can be observed in several tools and solutions under the hood in our applications.

Imagine that you are organizing your T-shirt drawer. You fold them and place them on top of each other, in other words, you stack your clothes. As you are a very lazy person, when you need to grab a T-shirt, you always choose the easiest one, in this case, always the one on top. Therefore, you will only wear the T-shirt at the bottom of the stack, which was also the first one you put there, when all the others are dirty. This is the main rule of a Stack: the first to enter will be the last one to leave, which we call LIFO (Last In, First Out).

On the other hand, when we talk about Heap, we can think in a more diverse structure without many rules. Imagine you are a shoe fan and you have many different models, although you need to wear yourself to match with your stacked T-shirts. To store this mountain of shoes you bought a giant cabinet for your house. This cabinet has several horizontal and vertical divisions to place your shoes, and the divisions are so huge that you had to enumerate each one to know where each shoe was stored. You can store a shoe in any division, since you write down the position where it was placed, and also, can use any shoe any moment, just knowing where it is, or seeking one by one. This is the Heap structure.

Supposing a distribution like the one shown in the above image, with our shoes being the green squares, we can say that if today we want our sneakers for a long walk, which are at I2, or our slippers for the cold weather, which are at F8. We can pick up any shoe, as well as store any of them.

This memory separation allows each region to store the data type it can work with most efficiently. In addition, the Stack is also responsible for building the software’s Call Stack, where for each new scope or layer we enter in the code, it stacks a new package of variables and data in memory. Let’s walk through an example with the following code snippet below:

In this code, we calculate the average of two numbers, using 10 and 20 as examples. At the beginning, the Call Stack is empty. When we call the function average, its execution is added to the stack along with the declaration of two internal variables a and b. Here, the sum function is called to find the numerator of the division, so a new item is added to the stack. However, this item is short-lived, as it only performs the sum of the two numbers and returns the result to the average function, removing the item again from the top of the stack. Now, with this result, the division function is ready to be called, inserting a new item into the stack once more. With the division result returning to average and removing its execution from the Call Stack, the average calculation function is also ready to return, removing the last item from the structure and ending program execution. Observe this flow in the following image with each step of the execution:

Observing the structure, it is possible to see that there is a logical sequence of stacking throughout execution. Furthermore, there are several performance benefits when storing data in the Stack. For example, the fact that the Stack size is fixed allows it to be pre-allocated during the software initialization, eliminating the need to make system calls to the operating system to allocate memory at runtime. The Stack also has a well-defined structure for data flow, where we only manipulate the items at the top of the stack. This provides us with the exact address for allocation and lookup, making writes and reads faster compared to the Heap. Another benefit of the Stack is storing data in a sequential manner, ensuring full utilization of memory space, which does not happen in the Heap, as we will see later. But the question that remains regarding all these points is: if the Stack is so amazing, why don’t we store everything there?

The Stack has some limitations that are tied to its own advantages, such as its fixed size, which guarantees pre-allocation but also makes it inflexible. Imagine again the T-shirt drawer example, and suppose you bought a drawer sized to hold exactly 10 T-shirts. However, you received a new T-shirt for your birthday and no longer have space for it. Your drawer is now overloaded. When your Stack is overloaded, trying to hold more data than its capacity allows, it stops working due to a very famous and characteristic error, the Stack Overflow. This can happen easily, as the space allocated for the Stack is relatively small. Moreover, in the software we build, we use data structures that are much more complex than just primitive types. Often, we also need variables, functions and data that must be used accessible across the entire application. In contrast, the Stack provides scope-lovel isolation. Therefore, it would be necessary to duplicate data across different items in the Call Stack to access it, and even then, write operations would be extremely difficult, as any change would need to be replicated everywhere that variable exists in the Call Stack.

Thus, we introduce the use of the Heap to handle writing to free spaces, address-based lookup, storing complex structures, and using data in the global scope, etc. Considering that Heap allocation is slower, since as mentioned earlier, it depends on system calls to the operating system to reserve memory space, and in addition, requires identifying free spaces because the structure is not sequential like the Stack, we can consolidate a clearer view of the division of what each memory region stores: the Stack is responsible for the Call Stack with primitive types and pointers, while the Heap manages complex structures, like Array, Objects, etc.

We also have the scenario where multiple NodeJS Worker Threads and parallel operations are executed. When this happens, each Worker runs its own isolated V8 instance, meaning each one has its own Heap and its own Stack. Furthermore, we can mention a phenomenon that likely every developer has had to face at least once in their life: memory leaks. This occurs when we allocate space in the Heap, but before our software terminates, we don’t free it, leaving an occupied memory space that is no longer used by anything. In low-level languages, like C, this would be equivalent to allocating space with malloc and overwriting the pointer before freeing the memory with the free function.

You likely noticed the presence of a new term in the paragraphs above: pointers. To understand the need for pointers, let’s first take a quick look at Arrays.

An Array is a data structure that allows storing multiple values sequentially in memory. The word “sequential” here is of extreme importance. Looking at our shoe cabinet with several compartments, imagine now that you want to store shoes from the same brand next to each other. You try to place them all on the same shelf, as shown in the image below:

When you declare a new Array in your code, you are making this separation in the Heap as seen above, however, at the same time, we are creating a way to reference this allocated address on the Stack. In our analogy, you stored your brand-sorted shoes in your giant cabinet and wrote down on a small piece of paper where each brand starts, for example: brand A starts at A1 and brand B starts at A4. Technically, what is happening is that you are storing the Array values in the Heap, and a pointer on the Stack that shows where this Array begins. You don’t need to write down on your paper that items A1, A2 and A3 belong to brand A, since one premise of Arrays is that they are stored sequentially in memory. Therefore, if you know that there are 3 shoes from brand A and that the first one is at A1, you can easily predict where all the others are.

This same organization happens for Objects, however, V8 manages this with different naming conventions. Each Array key is called index and must be a numerical value, just like the Arrays that we deal with in programming languages, where each key points to an element. On the other hand, Object keys are called properties, they can be alphanumeric and point to values. Thus, we have the following layout for these elements in memory:

However, you are a very lucky person, and as if receiving a T-shirt for your birthday were not enough, you won a raffle from brand A and got 2 new pairs of shoes. Now, you need to reorganize your shoes in the cabinet. This reveals a problem with our current structure: it would be necessary to move Objects in memory, either by positioning the item from brand B in another space, or by moving the entire Array from brand A, and this would result in updating all pointers for these variables to point to the new address. To add a layer of abstraction, ensure high performance and allow easy manipulation of this data, the structure stored in the Heap is not directly an Array (or Object), but rather structures implemented by V8 that use a slightly different mechanism.

The Stack pointer actually points to a JSArray, an intermediate structure that V8 uses to manage pointers and values in memory for Arrays. It works like a header that holds information such as length (the array size), elements (previously mentioned as the values stored by the Array), properties (values bound to non-numeric keys), and Map (or hidden class), a structure that maintains the shape of that Array or Object. The elements property, instead of storing values directly, has a pointer that points to another data structure called FixedArray, which in turn holds the actual stored values.

Objects, in turn, have their equivalent structure, the JSObject. Both are considered HeapObjects. Analyzing the JSObject is a good opportunity to understand the Map structure and how it is used. This structure helps us group Objects with the same shape, that is, imagine you create 3 shoe instances and all of them have the same keys (properties): brand, size and color. All three will share the same Map. For this management process, Maps have several properties, such as instance_size, which represents the size an instance of that Object occupies, elements_kind, which we will see next, and descriptors, which map each property to its offset relative to the starting position of the entity’s pointer.

The elements_kind varies according to the data type and the way the Array or Object is populated. We have various types, and while we won’t dive deep into each one, we can highlight a few examples, such as kinds with the PACKED prefix, which represents Arrays and Objects that are fully populated, while the HOLEY prefix represents those with missing elements. In addition, we have divisions by data type, such as PACKED_SMI_ELEMENTS for Small Integers, PACKED_DOUBLE_ELEMENTS for floating-point numbers, and PACKED_ELEMENTS for other data types. 

Notice that this entire structure now allows us, when we push an item to an Array or modify an object in a way that forces them to move in memory, to avoid changing any variable pointers, thanks to this abstraction, we only need to update the internal pointers of the JSObject or JSArray pointing to their elements.

However, even with this level of abstraction, the Heap still presents some problems, such as memory space fragmentation and the retaining of unused variables.

As mentioned earlier, what would be done in our cabinet to add 2 more shoes to the 3 we had there previously would be to move all the shoes from brand A to a new space where all of them would fit side by side, allocating them now at A5.

Imagine this happening more frequently and with larger volumes of data. We could say that the Heap memory layout would look something like the image below:

All the empty spaces represent unoccupied spaces in the Heap. Supposing that each element of an Array occupies one of these spaces, looking at the image, we would no longer be able to store a new Array of 5 items, even though there are 5 empty spaces in the Heap, because they are not sequential. Fragmentation is a representation of what memory can look like after manipulation, whether by adding, updating, or removing values from it.

Previously, we talked about memory leaks, where we don’t free the memory allocated to variables that are no longer needed. Today, most modern programming languages already have a built-in mechanism responsible for finding these items that are no longer used and discarding them. This mechanism is called the Garbage Collector.

In V8, the garbage collection tool is named Orinoco. It follows a series of steps to ensure that memory continues to be reused, reorganized and periodically cleaned: identifying active and inactive items in memory, recycling the memory occupied by inactive items, and compacting or defragmenting memory.

To understand how it works more deeply, we first need to understand that it separates the Heap into two virtual parts. One is called the Young Generation and the other is the Old Generation.

The Young Generation is based on the generational hypothesis, which assumes that most objects stored in memory are short-lived, in other words, they die young, whether it is an internal function variable or an arbitrary string. Therefore, V8 reserves only a small space for this management. This space is subdivided into two additional parts: from-space and to-space. When a new Heap item is declared, it enters the Young Generation in the from-space. After some execution time, if the from-space becomes full, the algorithm identifies active objects and moves them to the to-space, then discards the entire from-space. After this step, both spaces are reorganized, where the former from-space becomes to-space, and to-space becomes from-space. When this object movement operation from one space to another happens, it runs in a mode called stop-the-world, where the JavaScript stops completely while moving the objects. However, this is still a fast process despite the pause, since, considering the generational hypothesis, only a small portion of these objects are copied, so the volume is very low. This process is executed by an algorithm called Scavenger, and the garbage collector in the Young Generation is called Minor GC.

If an object survives two rounds of Scavenger, it is moved to the Old Generation, where the garbage collection process is called Major GC, and runs a Mark-Compact algorithm. This algorithm’s process is executed in three steps, starting with mark, where the object graph is completely analyzed and active items are marked. In the next step, sweep occurs, where all previously unmarked objects are discarded and memory space is freed. Finally, compact performs defragmentation, moving objects to fill the empty memory spaces.

All of this process ensures memory reusability throughout the execution of our software. To handle all of this efficiently, Orinoco applies optimization techniques to each process, for instance, the marking step is performed incrementally, that is, it marks some items, returns control to JavaScript, and then resumes, repeating this cycle. Additionally, part of marking and sweeping work runs in background threads, while the main JavaScript process continues running on the main thread. Finally, when stop-the-world is required, it runs multiple parallel threads to shorten the pause. 

V8 is an extremely complex tool that requires a lot of study time to understand its inner workings. The goal of this initial series of posts was to bring a broad overview with a touch of depth in some concepts, which should always be further studied and explored. In the future, we might resume with an exclusive series about V8, but for now, we will move on to the next post in the series, where we will start to explore Libuv and the Event Loop.