roman/OptiScaler2

Watch

Fork 0

Fix the ImGui overlay and DLSS-G interlock under vkd3d-proton #1138

Open

y4my4my4m wants to merge 1 commit from y4my4my4m/proton-overlay-fixes into master

y4my4my4m commented

2026-09-04 15:00:05 +00:00

(Migrated from github.com)

Opening the OptiScaler menu on a D3D12 game under Proton removes the GPU device. This fixes that, and four related problems found while tracking it down.

Tested on an RTX 4090 / Cyberpunk 2077 / vkd3d-proton. Builds clean.

The overlay draws through Vulkan on a D3D12 game

IdentifyGpu::getPrimaryGpu().usesDxvk is true for a vkd3d-proton D3D12 title as well as a DXVK D3D11 one, so LocalPresent takes the early return at wrapped_swapchain.cpp:327 and MenuOverlayDx::Present is never reached. MenuOverlayVk becomes the live ImGui backend.

It then submits on a VkQueue it selected itself, unserialised, into the present path Streamline owns. _vkPresentMutex is declared twice -- menu_overlay_vk.cpp:20 and Vulkan_Hooks.cpp:33 -- and locked nowhere; the lock_guard at :514 is commented out. With DLSS-G active, the first menu open removes the device: VK_ERROR_DEVICE_LOST reported by vkd3d's own hk_vkQueueSubmit2 about half a second after the overlay's submit.

Measured across two runs: menu open with DLSS-G off survived 1257 overlay submits; with DLSS-G on it died after 4. A separate 82-second run with DLSS-G on and the menu never opened had zero device losses.

The early return is now D3D11 only, so these titles reach MenuOverlayDx::Present and draw on the game's own queue.

ImGui holds one renderer backend in io.BackendRendererUserData and vkCreateSwapchainKHR is hooked either way, so both overlays would claim it. MenuOverlayVk stands down for a vkd3d-proton D3D12 swapchain, and MenuOverlayDx::Present carries the reverse guard in case a different DXVK build creates the VkSwapchain first. Worst case becomes no overlay rather than a wild pointer.

The DLSS-G interlock never armed

Streamline_Hooks.cpp already forces DLSSGMode::eOff with eRetainResourcesWhenOff while the menu is visible, and charges delayMenuRenderBy -- exactly the protection needed here. It is keyed on state.swapchainApi == API::Vulkan. Under vkd3d-proton that value is DX12 while the Vulkan overlay is the live backend, so the guard silently never applied.

It is now keyed on the overlay, via a new State::menuOverlayIsVulkan set when MenuOverlayVk's objects are built and cleared on every teardown path. swapchainApi is deliberately left alone: it also drives menu_common.cpp:3197-3275 and :4643-4690, and flipping it would grey out DLSS-G and FSR-FG in the menu.

The gate is factored into StreamlineHooks::applyMenuDlssgInterlock and called from DLSSG_Dx12::Dispatch too, which pushed eOn through the raw StreamlineProxy::DLSSGSetOptions export and bypassed the hook entirely.

The frozen check trips under frame generation

TickFrozenCheck runs per present; _frameCount advances per evaluate. Two problems: these titles reach both LocalPresent and hkvkQueuePresentKHR, so it ticked twice per present and halved the threshold; and the flat > 10 assumes one present per evaluate, which frame generation breaks by design. At 4X and above the feature reads as frozen while the game is running, and the menu flashes "is active, but not currently used by the game" every few frames.

One tick per present, and the threshold scales with dlssgDetectedInterpolationCount.

DisableFlipMetering stopped working with Streamline 2.14

dllmain.cpp only loads and hooks NVAPI when nvapi64.dll is already present, which at init it is not. Streamline 2.14 resolves its NVAPI entry points as it initialises -- before the QueryInterface detour is installed -- so the module is hooked but the caller holds pre-hook addresses and the deny list is never consulted.

Confirmed by log ordering: nvapi64.dll loads at 12.690, the hook lands at 12.819. With the window closed, FlipMetering is disabled! fires; without it, never.

NVAPI is now loaded and hooked up front when the option asks for it, guarded on Nvidia and on the option being set, so no process gains NVAPI it would not otherwise have had.

Three latent defects in menu_overlay_vk

Found while diagnosing the above. None caused the device loss; the second cost real diagnosis time.

_ImVulkan_Info.CheckVkResultFn was never assigned, so ImGui's Vulkan backend discarded every VkResult including DEVICE_LOST (imgui_impl_vulkan.cpp:429 is a no-op without it).
The framebuffer destroy guard tested fd->BackbufferView after the preceding block had nulled it, so framebuffers were never destroyed.
QueuePresent and DestroyVulkanObjects were unserialised. Now _vkCleanMutex, released across MenuOverlayBase::RenderMenu and re-validated on re-acquisition -- RenderMenu can re-enter DestroyVulkanObjects through IdentifyGpu::getPrimaryGpu() and the swapchain hook, so a non-recursive mutex held across it would deadlock. The dead duplicate _vkPresentMutex declarations are removed.

Scope

Windows and native-Vulkan behaviour is unchanged: menuOverlayIsVulkan is false there and every gate evaluates as before. The routing change affects Proton D3D12 titles, which is the population currently losing the device.

Not runtime-tested on Windows or on a native Vulkan title; I have neither.

Opening the OptiScaler menu on a D3D12 game under Proton removes the GPU device. This fixes that, and four related problems found while tracking it down. Tested on an RTX 4090 / Cyberpunk 2077 / vkd3d-proton. Builds clean. ## The overlay draws through Vulkan on a D3D12 game `IdentifyGpu::getPrimaryGpu().usesDxvk` is true for a vkd3d-proton D3D12 title as well as a DXVK D3D11 one, so `LocalPresent` takes the early return at `wrapped_swapchain.cpp:327` and `MenuOverlayDx::Present` is never reached. `MenuOverlayVk` becomes the live ImGui backend. It then submits on a `VkQueue` it selected itself, unserialised, into the present path Streamline owns. `_vkPresentMutex` is declared twice -- `menu_overlay_vk.cpp:20` and `Vulkan_Hooks.cpp:33` -- and locked nowhere; the `lock_guard` at `:514` is commented out. With DLSS-G active, the first menu open removes the device: `VK_ERROR_DEVICE_LOST` reported by vkd3d's own `hk_vkQueueSubmit2` about half a second after the overlay's submit. Measured across two runs: menu open with DLSS-G off survived 1257 overlay submits; with DLSS-G on it died after 4. A separate 82-second run with DLSS-G on and the menu never opened had zero device losses. The early return is now D3D11 only, so these titles reach `MenuOverlayDx::Present` and draw on the game's own queue. ImGui holds one renderer backend in `io.BackendRendererUserData` and `vkCreateSwapchainKHR` is hooked either way, so both overlays would claim it. `MenuOverlayVk` stands down for a vkd3d-proton D3D12 swapchain, and `MenuOverlayDx::Present` carries the reverse guard in case a different DXVK build creates the VkSwapchain first. Worst case becomes no overlay rather than a wild pointer. ## The DLSS-G interlock never armed `Streamline_Hooks.cpp` already forces `DLSSGMode::eOff` with `eRetainResourcesWhenOff` while the menu is visible, and charges `delayMenuRenderBy` -- exactly the protection needed here. It is keyed on `state.swapchainApi == API::Vulkan`. Under vkd3d-proton that value is `DX12` while the Vulkan overlay is the live backend, so the guard silently never applied. It is now keyed on the overlay, via a new `State::menuOverlayIsVulkan` set when `MenuOverlayVk`'s objects are built and cleared on every teardown path. `swapchainApi` is deliberately left alone: it also drives `menu_common.cpp:3197-3275` and `:4643-4690`, and flipping it would grey out DLSS-G and FSR-FG in the menu. The gate is factored into `StreamlineHooks::applyMenuDlssgInterlock` and called from `DLSSG_Dx12::Dispatch` too, which pushed `eOn` through the raw `StreamlineProxy::DLSSGSetOptions` export and bypassed the hook entirely. ## The frozen check trips under frame generation `TickFrozenCheck` runs per present; `_frameCount` advances per evaluate. Two problems: these titles reach both `LocalPresent` and `hkvkQueuePresentKHR`, so it ticked twice per present and halved the threshold; and the flat `> 10` assumes one present per evaluate, which frame generation breaks by design. At 4X and above the feature reads as frozen while the game is running, and the menu flashes "is active, but not currently used by the game" every few frames. One tick per present, and the threshold scales with `dlssgDetectedInterpolationCount`. ## DisableFlipMetering stopped working with Streamline 2.14 `dllmain.cpp` only loads and hooks NVAPI when `nvapi64.dll` is already present, which at init it is not. Streamline 2.14 resolves its NVAPI entry points as it initialises -- before the `QueryInterface` detour is installed -- so the module is hooked but the caller holds pre-hook addresses and the deny list is never consulted. Confirmed by log ordering: `nvapi64.dll` loads at `12.690`, the hook lands at `12.819`. With the window closed, `FlipMetering is disabled!` fires; without it, never. NVAPI is now loaded and hooked up front when the option asks for it, guarded on Nvidia and on the option being set, so no process gains NVAPI it would not otherwise have had. ## Three latent defects in menu_overlay_vk Found while diagnosing the above. None caused the device loss; the second cost real diagnosis time. - `_ImVulkan_Info.CheckVkResultFn` was never assigned, so ImGui's Vulkan backend discarded every `VkResult` including `DEVICE_LOST` (`imgui_impl_vulkan.cpp:429` is a no-op without it). - The framebuffer destroy guard tested `fd->BackbufferView` after the preceding block had nulled it, so framebuffers were never destroyed. - `QueuePresent` and `DestroyVulkanObjects` were unserialised. Now `_vkCleanMutex`, released across `MenuOverlayBase::RenderMenu` and re-validated on re-acquisition -- `RenderMenu` can re-enter `DestroyVulkanObjects` through `IdentifyGpu::getPrimaryGpu()` and the swapchain hook, so a non-recursive mutex held across it would deadlock. The dead duplicate `_vkPresentMutex` declarations are removed. ## Scope Windows and native-Vulkan behaviour is unchanged: `menuOverlayIsVulkan` is false there and every gate evaluates as before. The routing change affects Proton D3D12 titles, which is the population currently losing the device. Not runtime-tested on Windows or on a native Vulkan title; I have neither.

FakeMichau (Migrated from github.com) reviewed

2026-09-04 15:00:05 +00:00

This pull request has changes conflicting with the target branch.

OptiScaler/State.h
OptiScaler/framegen/dlssg/DLSSG_Dx12.cpp
OptiScaler/hooks/Streamline_Hooks.cpp
OptiScaler/hooks/Vulkan_Hooks.cpp
OptiScaler/menu/menu_overlay_vk.cpp
OptiScaler/upscalers/IFeature.cpp
OptiScaler/wrapped/wrapped_swapchain.cpp

View command line instructions

Manual merge helper

Use this merge commit message when completing the merge manually.

Merge commit title

Merge pull request 'Fix the ImGui overlay and DLSS-G interlock under vkd3d-proton' (#1138) from y4my4my4m/proton-overlay-fixes into master

Merge commit body

Reviewed-on: https://git.khaliavchik.com/roman/OptiScaler2/pulls/1138

Merge pull request 'Fix the ImGui overlay and DLSS-G interlock under vkd3d-proton' (#1138) from y4my4my4m/proton-overlay-fixes into master

Reviewed-on: https://git.khaliavchik.com/roman/OptiScaler2/pulls/1138

Checkout

From your project repository, check out a new branch and test the changes.

git fetch -u origin y4my4my4m/proton-overlay-fixes:y4my4my4m/proton-overlay-fixes

git switch y4my4my4m/proton-overlay-fixes

Merge

Merge the changes and update on Forgejo.

Warning: The "Autodetect manual merge" setting is not enabled for this repository, you will have to mark this pull request as manually merged afterwards.

git switch master

git merge --no-ff y4my4my4m/proton-overlay-fixes

git switch y4my4my4m/proton-overlay-fixes

git rebase master

git switch master

git merge --ff-only y4my4my4m/proton-overlay-fixes

git switch y4my4my4m/proton-overlay-fixes

git rebase master

git switch master

git merge --no-ff y4my4my4m/proton-overlay-fixes

git switch master

git merge --squash y4my4my4m/proton-overlay-fixes