On friday, may 5th 2023 Yugabyte released version 2.17.3.0 of the database. Version 2.17.3.0 is a pre-release version.
However, there is an exciting feature that will be extremely handy for understanding, tuning and troubleshooting the YSQL (PostgreSQL compatible) API: The pg_stat_activity PostgreSQL standard catalog table has gotten two new fields:
- allocated_mem_bytes
- rss_mem_bytes
allocated_mem_bytes
The allocated_mem_bytes field shows the memory allocated by the memory allocator. PostgreSQL is setup in an extensible way, which includes the ability to choose a memory allocator, which for PostgreSQL is ptmalloc, and for YSQL is tcmalloc. PostgreSQL has the ability to change the memory allocator, but by default uses the operating system memory allocator.
The memory shown in the allocated_mem_bytes field is the amount of memory the allocator has allocated for the postgres backend as well as our pggate addition to communicate with DocDB.
It is important to realise that the allocator gets involved after the process has forked from the postgres server process and has paged in all the required memory to run the process, so once the postgres "application" gets run and allocates memory for its processing.
The allocated_mem_bytes field can show memory that was previously allocated and in reality is freed, but still is part of the virtual memory allocation. One specific change we made in YugabyteDB for the backend tcmalloc allocator is to free aggressively to prevent memory oversubscription. If this specific case is happening, the rss_mem_bytes field will show a smaller amount of memory than the allocated_mem_bytes field.
rss_mem_bytes
The rss_mem_bytes field shows the operating system level view of the memory usage, for which it is showing the RSS: the resident set size.
The resident set size are all the pages that are paged into the process' memory. Some of these pages can be shared with other processes, such as memory mapped pages.
The memory that the memory allocator (tcmalloc with YugabyteDB, ptmalloc for PostgreSQL) allocates is administered as RSS once the allocation gets read or written.
In general, the rss_mem_bytes/resident set size will be larger than allocated_mem_bytes, because the linux level dealing with the executable and execution requires some memory too. But this not always be true.
shared memory / buffer cache
Please mind that with YSQL there still is shared memory visible that is defined as shared memory for the database. This memory is not allocated using the a memory allocator.
This shared memory is used by some global database views, and with PostgreSQL is used to allocate the buffer cache. In YSQL, we do not use the postgres level buffer cache. Therefore you will find that almost nothing of this memory is paged in and therefore actually taking memory.
The amount of memory that is paged in will be visible in rss_mem_bytes, shared memory is totally outside of allocated_mem_bytes.
Top comments (0)