There is a way to query GPU memory usage in Vulkan - use DXGI

In my GDC 2018 talk “Memory management in Vulkan and DX12” (slides freely available, video behind GDC Vault paywall) I said that in Direct3D 12 you can query for the exact amount of GPU memory used and available, while in Vulkan there is no way to do that, so I recommend to just query for memory capacity (VkMemoryHeap::size) and limit your usage to around 80% of it. It turns out that I wasn’t quite right. If you code for Windows, there is a way to do this. I assumed that the mentioned function IDXGIAdapter3::QueryVideoMemoryInfo is part of Direct3D 12 interface, while it is actually part of DirectX Graphics Infrastructure (DXGI). This is a more generic, higher level Windows API that allows you to enumerate adapters (graphics cards) installed in the system, query for their parameters and outputs (monitors) connected to them. Direct3D refers to this API, but it’s not the same.

Key question is: Can you use DXGI to query for GPU memory usage while doing graphics using Vulkan, not D3D11 or D3D12? Would it return some reasonable data and not all zeros? Short answer is: YES! I’ve made an experiment - I wrote a simple app that creates various Vulkan objects and queries DXGI for memory usage. Results look very promising. But before I move on to the details, here is a short primer of how to use this DXGI interface, for all non-DirectX developers:

1: Use C++ in Visual Studio. You may also use some other compiler for Windows or other programming language, but it will be probably harder to setup.

2: Install relatively new Windows SDK.

3: #include <dxgi1_4.h> and <atlbase.h>

4: Link with “dxgi.lib”.

5: Create Factory object:

IDXGIFactory4* dxgiFactory = nullptr;
CreateDXGIFactory1(IID_PPV_ARGS(&dxgiFactory));

Don’t forget to release it at the end:

dxgiFactory->Release();

6: Write a loop to enumerate available adapters. Choose and remember suitable one.

IDXGIAdapter3* dxgiAdapter = nullptr;
IDXGIAdapter1* tmpDxgiAdapter = nullptr;
UINT adapterIndex = 0;
while(m_DxgiFactory->EnumAdapters1(adapterIndex, &tmpDxgiAdapter) != DXGI_ERROR_NOT_FOUND)
{
    DXGI_ADAPTER_DESC1 desc;
    tmpDxgiAdapter>GetDesc1(&desc);
    if(!dxgiAdapter && desc.Flags == 0)
    {
        tmpDxgiAdapter->QueryInterface(IID_PPV_ARGS(&dxgiAdapter));
    }
    tmpDxgiAdapter->Release();
    ++adapterIndex;
}

At the end, don’t forget to release it:

dxgiAdapter->Release();

Please note that using new version of DXGI interfaces like DXGIFactory4 and DXGIAdapter3 requires some relatively new version (I’m not sure which one) of both Windows SDK on developer’s side (otherwise it won’t compile) and updated Windows system on user’s side (otherwise function calls with fail with appropriate returned HRESULT).

7: To query for GPU memory usage at the moment, use this code:

DXGI_QUERY_VIDEO_MEMORY_INFO info = {};
dxgiAdapter->QueryVideoMemoryInfo(0, DXGI_MEMORY_SEGMENT_GROUP_LOCAL, &info);

There are two possible options:

• DXGI_MEMORY_SEGMENT_GROUP_LOCAL is the memory local to the GPU, so basically video RAM.
• DXGI_MEMORY_SEGMENT_GROUP_NON_LOCAL is the system RAM.

Among members of the returned structure, the most interesting is CurrentUsage. It seems to precisely reflect the use of GPU memory - it increases when I allocate a new VkDeviceMemory object, as well as when I use some implicit memory by creating other Vulkan resources, like a swap chain, descriptor pools and descriptor sets, command pools and command buffers, query pools etc.

Other DXGI features for video memory - callback for budget change notification (IDXGIAdapter3::RegisterVideoMemoryBudgetChangeNotificationEvent) and reservation (IDXGIAdapter3::SetVideoMemoryReservation) may also work with Vulkan, but I didn’t check them.

As an example, on my system with GPU = AMD Radeon RX 580 8 GB and 16 GB of system RAM, on program startup and before any Vulkan or D3D initialization, DXGI reports following data:

Local:
    Budget=7252479180 CurrentUsage=0
    AvailableForReservation=3839547801 CurrentReservation=0
Nonlocal:
    Budget=7699177267 CurrentUsage=0
    AvailableForReservation=4063454668 CurrentReservation=0

8: You may want to choose correct DXGI adapter to match the physical device used in Vulkan. Even on the system with just one discrete GPU there are two adapters reported, one of them being software renderer. I exclude it by comparing desc.Flags == 0, which means this is a real, hardware-accelerated GPU, not DXGI_ADAPTER_FLAG_REMOTE or DXGI_ADAPTER_FLAG_SOFTWARE.

Good news is that even when there are more such adapters in the system, there is a way to match them between DXGI and Vulkan. Both APIs return something called Locally Unique Identifier (LUID). In DXGI it’s in DXGI_ADAPTER_DESC1::AdapterLuid. In Vulkan it’s in VkPhysicalDeviceIDProperties::deviceLUID. They are of different types - two 32-bit numbers versus array of bytes, but it seems that simple, raw memory compare works correctly. So the way to find DXGI adapter matching Vulkan physical device is:

// After obtaining VkPhysicalDevice of your choice:
VkPhysicalDeviceIDProperties physDeviceIDProps = {
    VK_STRUCTURE_TYPE_PHYSICAL_DEVICE_ID_PROPERTIES };
VkPhysicalDeviceProperties2 physDeviceProps = {
    VK_STRUCTURE_TYPE_PHYSICAL_DEVICE_PROPERTIES_2 };
physDeviceProps.pNext = &physDeviceIDProps;
vkGetPhysicalDeviceProperties2(physicalDevice, &physDeviceProps);

// While enumerating DXGI adapters, replace condition:
// if(!dxgiAdapter && desc.Flags == 0)
// With this:
if(memcmp(&desc.AdapterLuid, physDeviceIDProps.deviceLUID, VK_LUID_SIZE) == 0)

Please note that function vkGetPhysicalDeviceProperties2 requires Vulkan 1.1, so set VkApplicationInfo::apiVersion = VK_API_VERSION_1_1. Otherwise the call results in “Access Violation” error.

In my next blog post, I will present detailed results of my experiment with DXGI used with Vulkan application, tested on 2 different GPUs. Please come back in few days...