feat(capture/windows): hook APIs to avoid output reparenting that breaks DDA

[cgutman]

Description

This is a new approach for handling multi-GPU systems that could eliminate the need for ddprobe.exe and all the issues that come with locking the process to a single primary GPU. Rather than using registry values, we are hooking the underlying NtGdiDdDDIGetCachedHybridQueryValue() function that DXGI uses internally to track the GPU preference of the process and always returning D3DKMT_GPU_PREFERENCE_STATE_UNSPECIFIED. In my testing, this appears to completely eliminate the DXGI output reparenting (dxgi!ReparentOutputs()) that causes DDA to break.

This PR contains commits that completely remove ddprobe.exe and all options related to it. If we find a case that this new approach doesn't cover, we can drop those commits and preserve the options to use the legacy ddprobe.exe method.

In my testing on a multi-GPU desktop (AMD+Intel) and laptop (Intel+Nvidia) both running Win11 24H2, capture works properly regardless of which GPU the monitors are connected to or which GPU is set as default. The only restriction is that you can't use the monitor swap hotkeys to swap between monitors attached to different GPUs while streaming due to lack of support for switching encoders while streaming (unless you force software encoding).

I'd appreciate if @RebelliousX and @Nonary can test to ensure this doesn't break their streaming scenarios with virtual displays.

Screenshot

Issues Fixed or Closed

Type of Change

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• Dependency update (updates to dependencies)
• Documentation update (changes to documentation)
• Repository update (changes to repository files, e.g. .github/...)

Checklist

• My code follows the style guidelines of this project
• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas
• I have added or updated the in code docstring/documentation-blocks for new or existing methods/components

[codecov]

Codecov Report

Attention: Patch coverage is 85.71429% with 2 lines in your changes missing coverage. Please review.

Project coverage is 10.30%. Comparing base (c369e8e) to head (42daad2).
Report is 1 commits behind head on master.

Files with missing lines	Patch %	Lines
src/platform/windows/display_base.cpp	85.71%	0 Missing and 2 partials ⚠️

Additional details and impacted files

@@           Coverage Diff           @@
##           master    #3530   +/-   ##
=======================================
  Coverage   10.30%   10.30%           
=======================================
  Files          90       90           
  Lines       16171    16130   -41     
  Branches     7679     7655   -24     
=======================================
- Hits         1666     1662    -4     
+ Misses      13938    13911   -27     
+ Partials      567      557   -10     

Flag	Coverage Δ
Linux	9.68% <ø> (+<0.01%)	⬆️
Windows	10.77% <85.71%> (+<0.01%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
src/config.cpp	0.00% <ø> (ø)
src/config.h	0.00% <ø> (ø)
src/platform/windows/display_base.cpp	35.92% <85.71%> (+2.00%)	⬆️

[Nonary]

Does that mean we rely entirely on hooks now, since we’re no longer using the standard DXGI outputs? If so, won’t anti-cheat systems potentially block the hook? And if the hook fails, does that mean we’re unable to capture anything at all?

I had figured the only way to realistically fix the multi-GPU problem would be to refactor sunshine to capture in a separate process and then used shared memory, which is way out of my skill level.

[RebelliousX]

@cgutman I would love to test. But Windows build failed, I can't download artifact. 🤔

[FrogTheFrog]

Code-wise LGTM apart from a few minor requests.

[FrogTheFrog]

@cgutman I would love to test. But Windows build failed, I can't download artifact. 🤔

Please get the build from FrogTheFrog#4, where I hope I have fixed the build issue.

[RebelliousX]

@cgutman I would love to test. But Windows build failed, I can't download artifact. 🤔

Please get the build from FrogTheFrog#4, where I hope I have fixed the build issue.

Yours actually worked! That doesn't make any sense.. I am starting to lose faith in computer science, it only took a few "LOL" comments to make the build succeed 🤔

@cgutman I just tested this and it works great for the iGPU (leaving device id for monitor empty) and dGPU attached to VDD (when putting UUID device id for VDD).

Also at this point, the "Adapter Name" has no use now in my opinion, everything worked fine leaving it empty. So I think it is safe to remove.

I got to say, the log is a lot cleaner now.

Edit:

The only restriction is that you can't use the monitor swap hotkeys to swap between monitors attached to different GPUs while streaming due to lack of support for switching encoders while streaming (unless you force software encoding).

Maybe because I am use VDD, but WIN+P works fine while streaming. I was able to switch from only monitor 2 (VDD primary for dGPU, physical disconnected) to "PC Screen Only" (physical connected to iGPU) and streaming is still okay.

[ReenigneArcher]

Doc failures are working as intended. FYI:
https://github.com/LizardByte/doxyconfig/blob/4c9452482bd01cb36764dc914d4537b278ad4218/doxyconfig-Doxyfile#L136-L142

[cgutman]

Does that mean we rely entirely on hooks now, since we’re no longer using the standard DXGI outputs? If so, won’t anti-cheat systems potentially block the hook? And if the hook fails, does that mean we’re unable to capture anything at all?

We've basically just moved from one undocumented GPU preference mechanism (writing UserGpuPreferences registry values manually) to another (hooking the underlying API for preference selection). Both rely on OS implementation details that could break in an OS update, but there's not an official API to request this behavior so we're stuck with it.

We're only hooking APIs in our own process, so anti-cheat won't care about it. GPU preferences on the rest of the system will still behave normally.

[FrogTheFrog]

LGTM

[cgutman]

I've tested this in Intel+AMD and Intel+Nvidia scenarios on Win10 22H2 and it seems to work fine there too.

Update: Also tested with Easy-GPU-PV and that worked fine with the virtual display driver.

[sonarqubecloud]

Quality Gate failed

Failed conditions
2 New issues
2 New Code Smells (required ≤ 0)

See analysis details on SonarQube Cloud

Catch issues before they fail your Quality Gate with our IDE extension SonarQube for IDE