roman/OptiScaler2
Watch
1
0
Fork
You've already forked OptiScaler2
 0
Code Issues 12 Pull requests 7 Projects Releases 26 Packages Wiki Activity Actions

Add optional Neural Rendering with pipeline controls and residual upscaling #1158

Open
wilsjo2 wants to merge 17 commits from wilsjo2/codex/upstream-neural-rendering into master
pull from: wilsjo2/codex/upstream-neural-rendering
merge into: roman:master
Conversation 0 Commits 17 Files changed 202 +31796 -364
wilsjo2 commented 2026-09-12 19:57:10 +00:00 (Migrated from github.com)
Copy link

This adds optional Neural Rendering before/after the game's upscaler or on the finished picture. It defaults off and runs inside OptiScaler; users supply NVIDIA's NR runtime separately.

Features

  • A pipeline menu for placement, model passes/resolution, strength, skin/environment and HDR/exposure controls.
  • Early generation with later application: reconstruct the clean game image and upscale NR's edit separately. Source RR adds motion-vector accumulation; placement stays manual.
  • Classic, Matched residual and Matched residual + DLSS enlargement. The last enlarges the edit over the original full-resolution image.
  • Frame hold, comparisons, capture and GPU timing.
  • Shared D3D12 processing for native games and bridges; a separate native Vulkan pipeline. Private enlargement requires D3D12. Private RR needs compatible native guides; post/finished enlargement uses SR.

Fork MFG unlocks, external-FG mode, residual interpolation, NVFP4 selection and helper DLLs are removed. Official FG/MFG remains.

Review

Based on master 5ee53e38, including compatibility prerequisite #1157 until it lands. Other bridge/swapchain/lifetime changes share NR integration points; the file inventory explains their purpose. Attribution to Dagherbou, hhkbble, y4my4my4m, cmh1448 and RenoDX remains in the docs.

Commit-by-commit review guide — 17 NR commits

Each commit builds Release x64. The two prerequisite commits from #1157 are unchanged. Counts below exclude generated shader headers and include tests/docs.

Commit Focus Added lines
960ca493 Add shared shader routing and private texture scaling helpers 684
2743e413 Add NR settings, driver adapter and GPU lifetime tracking 2,170
4af597eb Add NR colour codecs, shader contracts and GPU regressions 3,308
fbf104d9 Add exposure discovery, readback and calibration anchors 1,261
0de77ac4 Add independent NR upscaler adapters and runtime smoke tests 876
c16a98e7 Implement DX12 NR resource ownership and command completion 3,883
6c52a033 Connect DX12 pre/post NR with model passes and guide restoration 1,409
30b0471e Route D3D11 DLSS and NR through the shared DX12 pipeline 184
98e7a395 Carry early NR edits through a private upscaler 551
470e8c6c Accumulate RR residuals with motion and select compatible private RR 140
bc02e0d7 Apply NR to finished DX12 and D3D11 pictures before presentation 1,202
62c528b4 Enlarge matched post-upscale NR residuals with private DLSS SR 151
9ac11d4e Add held NR inputs and gameplay/photo pipeline capture checks 244
82ff9b0f Integrate native Vulkan NR and finished-picture presentation 3,460
944c55e2 Expose NR controls through the render pipeline menu 1,541
a3363e05 Package NR and document review scope, setup and game results 737

| dc5611d6 | Serialize NR recording/reset/cleanup to fix the Starfield tracking crash | 72 |

Generated shader headers remain with their HLSL sources and are marked for GitHub to collapse. The original 16-commit NR series preserves the previous PR's runtime source, shaders and tests. The final follow-up fixes the Starfield tracking race and adds its regression.

Games checked

Windows, RTX 5090, driver 616.64. These are short runs across integration builds, with existing saves/settings preserved:

Game Observations
BG3, DX11 -> D3D12 DLSS Main placements, passes, reduced resolution, enlargement and deferred hold; clean follow-up exercised post/finished private DLSS. No crash observed.
KCD2, D3D12 SR Placements, post/finished enlargement and hold. Precision fix removed dense speckles; faint grid and deferred brightness/banding remain.
Jedi: Survivor, D3D12 SR Fixed invalid depth SRV/device removal. Main placements and reduced finished-picture modes ran without that crash.
Hogwarts Legacy, D3D12 RR Placements, enlargement, passes, hold/comparison across runs. Clean follow-up completed without a crash; possible grid unresolved.
Cyberpunk 2077 User reported the submission-fixed build working well. Clean DLL installed; no independent all-mode check this round.

All five received the clean integration DLL. This review branch additionally includes newer upstream DX11 swapchain adaptation, checked by build/regressions. Exact builds, settings and limits distinguish that coverage.

Every NR commit built Release x64; the final head also passed a clean rebuild, alongside production lifetime/capture/proxy, WARP swapchain and WARP/Vulkan precision checks. Shader binaries match; packaging and documentation links pass. The rewritten 18-commit range passed a secret scan.

Known limitations: HDR/temporal/grid quality needs work. BG3's follow-up reported FG OFF/Reflex inactive; FG/MFG was not comprehensively tested. Long-session VRAM stability, native Vulkan gameplay and broader GPU/driver/mode coverage remain unverified.

Starfield crash fix

The v0.8.0 report points to a null recording entry during command-list reset. The ordinary buffer-transition path could modify that tracker without the lock held by reset notifications, including after NR initialization failed. Tracker operations now serialize internally; the surrounding resource entry points follow the same lock order. No GPU wait or helper DLL was added.

A four-worker WARP test crashes with the old tracker and passes with the fix, with all 8,000 retirement callbacks delivered exactly once. Release x64 and the existing proxy/lifetime/capture regressions passed. Starfield gameplay still needs tester confirmation; this does not resolve the separate 0xBAD0000B runtime-initialization error. No fork MFG unlock or build-flag changes are included in this PR.

This adds optional Neural Rendering before/after the game's upscaler or on the finished picture. It defaults off and runs inside OptiScaler; users supply NVIDIA's NR runtime separately. ## Features - A pipeline menu for placement, model passes/resolution, strength, skin/environment and HDR/exposure controls. - Early generation with later application: reconstruct the clean game image and upscale NR's edit separately. Source RR adds motion-vector accumulation; placement stays manual. - Classic, Matched residual and Matched residual + DLSS enlargement. The last enlarges the edit over the original full-resolution image. - Frame hold, comparisons, capture and GPU timing. - Shared D3D12 processing for native games and bridges; a separate native Vulkan pipeline. Private enlargement requires D3D12. Private RR needs compatible native guides; post/finished enlargement uses SR. Fork MFG unlocks, external-FG mode, residual interpolation, NVFP4 selection and helper DLLs are removed. Official FG/MFG remains. ## Review Based on master `5ee53e38`, including compatibility prerequisite #1157 until it lands. Other bridge/swapchain/lifetime changes share NR integration points; the [file inventory](https://github.com/wilsjo2/OptiScaler_DLSSNR/blob/codex/upstream-neural-rendering/docs/NR-UPSTREAM-DIFF-INVENTORY.md) explains their purpose. Attribution to Dagherbou, hhkbble, y4my4my4m, cmh1448 and RenoDX remains in the docs. <details> <summary>Commit-by-commit review guide — 17 NR commits</summary> Each commit builds Release x64. The two prerequisite commits from #1157 are unchanged. Counts below exclude generated shader headers and include tests/docs. | Commit | Focus | Added lines | | --- | --- | ---: | | [960ca493](https://github.com/wilsjo2/OptiScaler_DLSSNR/commit/960ca49385d115fb32168e22c44df3a4b1de1ec8) | Add shared shader routing and private texture scaling helpers | 684 | | [2743e413](https://github.com/wilsjo2/OptiScaler_DLSSNR/commit/2743e413765fb8188cf63377046e6ea13dc526d7) | Add NR settings, driver adapter and GPU lifetime tracking | 2,170 | | [4af597eb](https://github.com/wilsjo2/OptiScaler_DLSSNR/commit/4af597ebfe8cf73f9526377413150c4624796d16) | Add NR colour codecs, shader contracts and GPU regressions | 3,308 | | [fbf104d9](https://github.com/wilsjo2/OptiScaler_DLSSNR/commit/fbf104d949a3c55ae08c99f488ed288a211e5fae) | Add exposure discovery, readback and calibration anchors | 1,261 | | [0de77ac4](https://github.com/wilsjo2/OptiScaler_DLSSNR/commit/0de77ac462789901ebee69b562a1a57fed35a04b) | Add independent NR upscaler adapters and runtime smoke tests | 876 | | [c16a98e7](https://github.com/wilsjo2/OptiScaler_DLSSNR/commit/c16a98e7720b84cafb8a935b4d1e952bd19cfb81) | Implement DX12 NR resource ownership and command completion | 3,883 | | [6c52a033](https://github.com/wilsjo2/OptiScaler_DLSSNR/commit/6c52a0333a62c28603f7a1bf872277d265586028) | Connect DX12 pre/post NR with model passes and guide restoration | 1,409 | | [30b0471e](https://github.com/wilsjo2/OptiScaler_DLSSNR/commit/30b0471e6581a04b1816d8af19332adf98818810) | Route D3D11 DLSS and NR through the shared DX12 pipeline | 184 | | [98e7a395](https://github.com/wilsjo2/OptiScaler_DLSSNR/commit/98e7a395d4724474b7a320587d517f596b988545) | Carry early NR edits through a private upscaler | 551 | | [470e8c6c](https://github.com/wilsjo2/OptiScaler_DLSSNR/commit/470e8c6c38db56972ef64bb40ae046129a0e2801) | Accumulate RR residuals with motion and select compatible private RR | 140 | | [bc02e0d7](https://github.com/wilsjo2/OptiScaler_DLSSNR/commit/bc02e0d7e6a66c669c05568d1c23e42fdc2ad432) | Apply NR to finished DX12 and D3D11 pictures before presentation | 1,202 | | [62c528b4](https://github.com/wilsjo2/OptiScaler_DLSSNR/commit/62c528b4b30e35ae10e2e725124775d7cde41574) | Enlarge matched post-upscale NR residuals with private DLSS SR | 151 | | [9ac11d4e](https://github.com/wilsjo2/OptiScaler_DLSSNR/commit/9ac11d4e89c572439c9f6f70c84fce317b26102f) | Add held NR inputs and gameplay/photo pipeline capture checks | 244 | | [82ff9b0f](https://github.com/wilsjo2/OptiScaler_DLSSNR/commit/82ff9b0f72cb6bf03775fc5555579fe900539963) | Integrate native Vulkan NR and finished-picture presentation | 3,460 | | [944c55e2](https://github.com/wilsjo2/OptiScaler_DLSSNR/commit/944c55e21dea1c437e8aa3c20367126a593271bf) | Expose NR controls through the render pipeline menu | 1,541 | | [a3363e05](https://github.com/wilsjo2/OptiScaler_DLSSNR/commit/a3363e0504a28ca9167764e7a9da0f92eb182fa2) | Package NR and document review scope, setup and game results | 737 | | [dc5611d6](https://github.com/wilsjo2/OptiScaler_DLSSNR/commit/dc5611d62dac84dc657198d2508a2c7385df651d) | Serialize NR recording/reset/cleanup to fix the Starfield tracking crash | 72 | Generated shader headers remain with their HLSL sources and are marked for GitHub to collapse. The original 16-commit NR series preserves the previous PR's runtime source, shaders and tests. The final follow-up fixes the Starfield tracking race and adds its regression. </details> ## Games checked Windows, RTX 5090, driver 616.64. These are short runs across integration builds, with existing saves/settings preserved: | Game | Observations | | --- | --- | | BG3, DX11 -> D3D12 DLSS | Main placements, passes, reduced resolution, enlargement and deferred hold; clean follow-up exercised post/finished private DLSS. No crash observed. | | KCD2, D3D12 SR | Placements, post/finished enlargement and hold. Precision fix removed dense speckles; faint grid and deferred brightness/banding remain. | | Jedi: Survivor, D3D12 SR | Fixed invalid depth SRV/device removal. Main placements and reduced finished-picture modes ran without that crash. | | Hogwarts Legacy, D3D12 RR | Placements, enlargement, passes, hold/comparison across runs. Clean follow-up completed without a crash; possible grid unresolved. | | Cyberpunk 2077 | User reported the submission-fixed build working well. Clean DLL installed; no independent all-mode check this round. | All five received the clean integration DLL. This review branch additionally includes newer upstream DX11 swapchain adaptation, checked by build/regressions. [Exact builds, settings and limits](https://github.com/wilsjo2/OptiScaler_DLSSNR/blob/codex/upstream-neural-rendering/docs/NR-UPSTREAM-REVIEW.md) distinguish that coverage. Every NR commit built Release x64; the final head also passed a clean rebuild, alongside production lifetime/capture/proxy, WARP swapchain and WARP/Vulkan precision checks. Shader binaries match; packaging and documentation links pass. The rewritten 18-commit range passed a secret scan. Known limitations: HDR/temporal/grid quality needs work. BG3's follow-up reported FG OFF/Reflex inactive; FG/MFG was not comprehensively tested. Long-session VRAM stability, native Vulkan gameplay and broader GPU/driver/mode coverage remain unverified. ## Starfield crash fix The v0.8.0 report points to a null recording entry during command-list reset. The ordinary buffer-transition path could modify that tracker without the lock held by reset notifications, including after NR initialization failed. Tracker operations now serialize internally; the surrounding resource entry points follow the same lock order. No GPU wait or helper DLL was added. A four-worker WARP test crashes with the old tracker and passes with the fix, with all 8,000 retirement callbacks delivered exactly once. Release x64 and the existing proxy/lifetime/capture regressions passed. Starfield gameplay still needs tester confirmation; this does not resolve the separate 0xBAD0000B runtime-initialization error. No fork MFG unlock or build-flag changes are included in this PR.
❤️ 5
This pull request has changes conflicting with the target branch.
  • OptiScaler/OptiScaler.vcxproj
  • OptiScaler/OptiTypes.cpp
  • OptiScaler/OptiTypes.h
  • OptiScaler/State.h
  • OptiScaler/framegen/dlssg/DLSSG_Dx12.cpp
  • OptiScaler/hooks/D3D12_Hooks.cpp
  • OptiScaler/hooks/D3D12_Hooks.h
  • OptiScaler/hooks/DxgiFactory_Hooks.cpp
  • OptiScaler/hooks/Streamline_Hooks.cpp
  • OptiScaler/hooks/Vulkan_Hooks.cpp
  • OptiScaler/inputs/NVNGX_DLSS_Dx12.cpp
  • OptiScaler/menu/menu_common.cpp
  • OptiScaler/resource_tracking/ResTrack_dx12.cpp
  • OptiScaler/upscalers/FeatureProvider_Dx11.cpp
  • OptiScaler/upscalers/IFeature_Dx12.cpp
  • OptiScaler/upscalers/IFeature_Vk.cpp
  • OptiScaler/upscalers/IFeature_Vk.h
  • OptiScaler/upscalers/IFeature_VkwDx12.cpp
  • OptiScaler/upscalers/dlss/DLSSFeature.h
  • OptiScaler/upscalers/dlss/DLSSFeature_Dx11On12.cpp
  • OptiScaler/upscalers/dlss/DLSSFeature_Dx11On12.h
  • OptiScaler/with_dx12/dx11_with_dx12_sc.cpp
  • OptiScaler/wrapped/wrapped_swapchain.cpp
View command line instructions

Manual merge helper

Use this merge commit message when completing the merge manually.

Checkout

From your project repository, check out a new branch and test the changes.
git fetch -u origin wilsjo2/codex/upstream-neural-rendering:wilsjo2/codex/upstream-neural-rendering
git switch wilsjo2/codex/upstream-neural-rendering

Merge

Merge the changes and update on Forgejo.

Warning: The "Autodetect manual merge" setting is not enabled for this repository, you will have to mark this pull request as manually merged afterwards.

git switch master
git merge --no-ff wilsjo2/codex/upstream-neural-rendering
git switch wilsjo2/codex/upstream-neural-rendering
git rebase master
git switch master
git merge --ff-only wilsjo2/codex/upstream-neural-rendering
git switch wilsjo2/codex/upstream-neural-rendering
git rebase master
git switch master
git merge --no-ff wilsjo2/codex/upstream-neural-rendering
git switch master
git merge --squash wilsjo2/codex/upstream-neural-rendering
git switch master
git merge --ff-only wilsjo2/codex/upstream-neural-rendering
git switch master
git merge wilsjo2/codex/upstream-neural-rendering
git push origin master
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels   
0.9
release/0.9

  
3rd party
For other mod combos

  
bug
Something isn't working

  
documentation
Improvements or additions to documentation

  
duplicate
This issue or pull request already exists

  
enhancement
New feature or request

  
invalid
This doesn't seem right

  
linux
Linux issue

  
master
master branch

  
question
Further information is requested

  
undetermined
cause unknown

  
wontfix
This will not be worked on

No labels
0.9
3rd party
bug
documentation
duplicate
enhancement
invalid
linux
master
question
undetermined
wontfix
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
roman/OptiScaler2!1158
No description provided.