Cxbx-Reloaded Progress Report (October 2020)

NOTE: This was written and ready to go by November 2nd, I apologise for the delay posting, life got in the way, anyway, here goes:

Welcome everyone, to our spooky scary October report! What is so scary about it? The size of this thing to begin with. The amount of games that improved this month alone is enough to send chills down my spine (seriously, I hope you have some time to spare reading the results I got…), and a lot of improvements to various parts of the code in general that fixed many long-standing issues. Sound, graphics and performance, there’s something to everyone, but let’s take a look at our numbers before going deep into the changes:

General Improvements

Allow users to disable aspect ratio correction [PR #1970]

As an improvement to the feature implemented last month, this new option allows users to ignore the aspect ratio correction for whatever reason (want your graphics stretched to cover the screen or has some widescreen hack applied? We got you covered)

PsCreateSystemThreadEx: Fill dwThreadId immediately after creating the thread [PR #1972]

In xbox::PsCreateSystemThreadEx, ThreadHandle is filled immediately after the host thread creation. However, ThreadId is filled after the thread has started, and that creates a window for a race if ThreadId is used by the newly spawned thread. This is exactly what NASCAR Heat 2002 was doing, and the newly spawned guest thread ended up looking for the value ThreadId should have filled. Filling the output value immediately after the guest thread is threaded alleviates the issue, and also makes more sense from a logical perspective.

Fix Symbol Scan's Kernel Thunk Bug [PR #1974]

Some improvements for HLE's initialization usage:

●  Don't treat EmuOutputMessage's debug message as trace, let it handle from EmuLog's functionality.

●  Fix a crash if move EmuHLEIntercept before MapThunkTable call. Now it will be easy to relocate if necessary.

●  Fix hidden xbeType bug for kernel thunk and entry point look up. It should be checking against xbe than system type.

●  Finally, resolve symbol scan unable to register kernel thunk addresses properly due to MapThunkTable was called first.

Add an rdtsc false positive check for Group S Challenge [1.05] [PR #1975]

Group S Challenge [1.03] was going ingame for years, but [1.05] crashed on startup. Turns out, it's due to a very funny rdtsc false positive which detected the instruction in the middle of a call instruction: E8 0F 31 01 00

This PR adds a check for this false positive for a 0x01 rdtsc pattern (triggered also by the offset, as evidenced). Group S Challenge [1.05] can now reach the menus

Add several D3D8LTCG function variations for xdk-3911 [PR #1976]

This variation puts Stage on the stack and pTexture in eax. The existing implementation was shared with xdk-4721 with a wrong assumption that both functions are the same, but they were not. With this change, NASCAR Heat 2002 progressed from being completely broken to reaching in-game for the first time.

MSAA improvements basing on xdk-3911 behaviour [PR #1978]

Introduces those changes to MSAA scaling:

●  Scaling should be performed on Clear calls too

●  Scaling factors should be recalculated when binding a render target, and should be derived from multisampling type only when binding the back buffer.

The code contains a code path checking for X_D3DMULTISAMPLE_SAMPLING_MULTI sampling type and divides scaling by 2 if that is the case. This is how xdk-3911 runtime does it, but it seems to result in half-width rendering at the moment (possibly due to something else assuming that multisampling and supersampling are the same). After this code was merged, NASCAR Heat 2002 rendering in-game became almost perfect:

Actually zero g_InlineVertexBuffer_Table[0] [PR #1981]

operator= invoked default constructors where available, and a default constructor for D3DXVECTOR4 does nothing. memset/memcpy zeroes and copies the entire structure for real. Fixes a festival of colours in Star Wars: Knights of the Old Republic depending on random factors such as unrelated code changes, Debug/Release configuration and optimization options (in practice, relying on stale stack

Bonus: a very nice article from the author of the PR himself, going into details about how he reached said solution https://cookieplmonster.github.io/2020/10/20/cxbx-r-kotor-random-bug/

Improvements to old style (mask based) mixbin [PR #1983]

This PR implements several mixbin functionalities by converting the "Revision 1" mixbin (mask + array of volumes) to "Revision 2" mixbin (an array of (mixbin, volume) pairs). SetMixBinVolumes_12 and the initial mixbin setting have both been implemented.

Additionally, this PR fixes a few mistakes I spotted with volume control in mixbins, which caused the sounds to go quieter every time mixbin volumes were updated. Broken code flow looked as follows:

1. Voice volume is initially set to X. This sets effectively set it to X - headroom.
   Then, for every SetMixBinVolumes call:
2. Effective voice volume (X - headroom) was obtained.
3. Voice volume was set to the just obtained value (X - headroom). Because of headroom correction, this effectively sets it to X - 2*headroom!

The fix is to compensate for that by adding voice headroom to the volume in step 1.

IDirectSoundBuffer_SetBufferData: Stop the sound before waiting for finish [PR #1984]

CDirectSoundBuffer_SetBufferData implementations in games using dsound 3925 and 5455 and they both send a stop request before waiting for the buffer to stop playing. This resolves a sound related deadlock in NASCAR Heat 2002, since it was waiting for a looped sound to finish (which, naturally, never happened).

Make RDTSC and ACPI timers stateful to fix overflows [PR #1985]

Stateless RDTSC and ACPI timers were ticking relative to the host QPC and multiplied to nanoseconds. This resulted in values so huge they would overflow within 20-30 minutes.

Introducing state allows to multiply to nanoseconds only over a delta value, which should be reasonably small in almost all cases.

These calculations were also verified against data races by putting them under a mutex, but I only lock it if the "Run Xbox threads on all cores" hack is enabled. Without this hack it should not be required. Games that were confirmed as improved from this change: NASCAR Heat 2002 and GTA III.

Update cxbx-debugger to support switching XBEs during emulation [PR #1986]

Many Xbox games don’t have a single binary from where they can be launched, in many cases they switch between XBEs dedicated to some specific parts of the game (menus being one XBE and the game itself being other it’s common, in some cases even a character edition tool being an individual XBE is possible). This PR updates the debugger project to use MDI forms to allow multiple windows inside the main app, covering these specific cases.

Memory manager fixes [PR #1987]

This fixes three problems recently discovered in the VMManager introduced during the loader transition:

1.  The UnknownType type was never used for the pfn database pages, which caused incorrect statistics for this page type. Shouldn't affect anything since this type is not reported by MmQueryStatistics.

2.  Upon xbe reboot, the page statistics were not restored at all. Potentially affects titles since MmQueryStatistics reports some of these page types.

3.  When an xbe reboot occurs while using the debug memory layout, the pfn database was restored from the wrong address, thus restoring inconsistent values. Affects Sega Chihiro titles because they are signed with the debug key and thus they will (erroneously) use the debug memory layout.

Fix crash in Stranger's Wrath caused by false rdtsc positives [PR #1989]

This brings the game Oddworld: Stranger's Wrath from instantly crashing at startup to actually booting and rendering the title screens.

Avoid false rdtsc negative in 25 to life [PR #1990]

This issue was raised by LukeUsher, which noticed that an rdtsc instruction in 25 to life has the same pattern of Stranger's Wrath, thus resulting in a false negative. This changes the pattern used in CxBx-Reloaded to still recognize the false positive in Stranger's Wrath but also avoid the false negative in 25 to Life.

Typo fix for IDirectSoundBuffer_SetMinDistance patch [PR #1992]

Fixes the player car's engine sound in Burnout being inaudible in certain situations (e.g. looking back).

DSound improvements (alternative volume heuristics) [PR #1993]

This PR picks the maximum channel volume from all speakers (except for the LFE speaker).

Since the current code picks a single volume level for all channels, it will never be perfect - but those two PRs (#1991 and #1993) exist so we can test which of the two approaches provides better volume levels. Verified improvement: Burnout in-game engine slightly higher and menu sounds audible for the first time.

Treat identical X_D3DResource objects with a different address as one [PR #1994]

Fixes issues where resource objects got relocated in memory or reused without having been released properly first. Fixes black cars in NASCAR Heat 2002 caused by the game bug - texture resources were not refcounted properly before being released (thus this is not a memory leak), and later "reappeared" as reused under a different memory address.

SetViewport & Clear improvements [PR #1996]

1.  Scales viewport X and Y and not only Width and Height. Cherry picked from #1894

2.  Pre-emptively fixes possible issues when the game passes multiple rectangles to Clear.

Fixes a misplaced and broken rear view mirror in NASCAR Heat 2002 and potentially fixes misplaced skaters in Tony Hawk's Pro Skater 2X.

emux86: fix bad bios rom mapping [PR #1997]

Bios ROM region reads were incorrectly based at 0xFFF00000 instead of 0xFF000000. Prevents a crash in Shenmue II.

Simplify thread creation logic, remove hardcoded delays and tighten affinity changes [PR #1998]

This PR simplifies the thread creation logic and removes hardcoded "magic" sleeps in favour of tighter affinity changes. Those "magic" sleeps were introduced to work around games hanging due to timing issues, but it appears those hangs would happen, because child threads would initially execute on any core before changing their own affinity.

Changes introduced:

●  Changed ThreadId parameter type to PWORD.

●  Removed hStartedEvent since it doesn't seem to protect anything meaningful anymore, and only messed with thread scheduling.

●  Starts child threads suspended and finalize their initialization before resuming - this closes a possible race condition where a child thread uses ThreadHandle or dwThreadId and starts before _beginthreadex even returns. It also allowed me to remove "magic" sleeps by ensuring that the child thread really never starts on a wrong CPU core.

Gets The Warriors in-game. Before this PR, a race condition in the original game code was hit - the game spawns a thread which waits on the event. However, the event in question is created after spawning that thread. Previous thread creation timings always caused the child thread to try and wait on the event before it was created.

Pixel combiner fixes [PR #1999]

1.  Fixes Inserted to avoid LRP parameters referencing the output register pass relying on a wrong register. It changed the destination register address, but not the type.

2.  Implemented better Blue-to-Alpha handling by fixing the output mask (previously erroreously only updated the B channel) and inserting an additional mov rX.a, rX.b opcode.

Fixes GPU decoding of BINK videos - test case being The Warriors, but it might fix more games using a pixel combiner to decode BINK.

Add a fallback to CopyRects for cases which StretchRect can't handle [PR #2000]

Fixes menus in World Racing 2 because CopyRects tries to copy a texture to texture.

Pixel shader fix XFC SUM [PR #2001]

Fix how our current pixel shader conversion calculates the final combiner special purpose register 'sum' : it was accidentally multiplying instead of adding it's arguments! Pixel shaders that use this 'sum' register will show more accurate output because of this change.

Simplify pixel shader constant handling [PR #2002]

Since we've ported over to Direct3D 9, and we're using pixel shader version 2.0, we've got more than enough constants available to remove the need for constant packing.

Also, there was a left-over patch on SetPixelShaderConstant which must no longer be applied, since nowadays we read constant values straight from their corresponding render state slots. This also implies we no longer need to declare the final combiner constants as part of the shader assembly, because these 2 are also read from their corresponding xbox render state slots, and thus can be transferred to host on each update.

This will likely improve the output of pixel shaders which stay otherwise unchanged but rely on changing constant values.

Fix APU timer ticking at wrong frequency [PR #2003]

No known improvements, but might improve the accuracy of whatever is relying on the APU timer for some sound timings.

Improve reverse screenspace transformation precision [PR #2005]

When a D24 depth buffer was used, its viewport Z scale was 16777215.0f. This number is high enough to cause precision issues in the reverse screenspace transformation calculation for some elements. In my case, it shifted the Z to slightly above 1.0f, causing it to get clipped. Based on some tests, pre-dividing the divisor increases the numerical stability of those calculations and allows to retain better precision. WARN: pre-multiplying instead of pre-dividing might seem like a better idea, but I found it did not resolve the precision issues!

Also adds an "unrelated" change of better batching for viewport scale/offset constant submission calls. Verified improvement: makes Burnout 3 display menus properly.

Changes to D3D patches [PR #2007]

●  Patches D3D_BlockOnTime_4 just like its non-LTCG equivalent.

●  Unpatches D3DDevice_GetTransform and trampolines SetTransform/MultiplyTransform to the guest code. This fixes cases where games tried to use a potentially unpatched or inlined GetTransform and/or read the D3D state directly.

Fixes Burnout 3 freezing past the car selection screen and makes it render geometry properly. Currently this is not yet visible because of an unhandled "binding surface as a texture" case, but rendering now does happen. Verified improvements: Burnout no longer freezing past car selection screen and Kingdom Under Fire characters now visible in-game.

Guard against nested SetTransform/MultiplyTransform calls [PR #2011]

MultiplyTransform can be implemented in means of doing the multiplication internally and SetTransform which corrupted the host's internal state (host SetTransform called with a multiplied state, followed by a host MultiplyTransform resulting in a double multiplication).

This PR introduces a guard variable to ensure we call to host only once per the patch chain and keep the internal state pristine. I facilitated this by introducing a NestedPatchCounter class which acts as a scoped counter - when used on a variable shared between potentially conflicting/nested patches, this class allows functions to skip parts of the host set-up if it is either redundant or harmful (like in this case).

Fixes a regression in 25 to Life introduced by #2007

Interlocked lockcounts [PR #2012]

Makes LockCount for RW locks and critical sections atomic, as previously those two primitives were (ironically) thread unsafe. No known change in behaviour, but on paper it might improve the stability with the All cores hack.

Notable Game Improvements

This is probably the biggest and spookiest (it’s October after all) list of improvements we had in a single month, so take a seat, grab something to eat and drink, and enjoy the ride (as it NEVER ENDS…)

Shrek Super Party [PLAYABLE]

Verified as playable. Some parts of the game can be slow depending on your hardware, but the most important parts (the actual mini-games you play to progress) should run with no performance issues.

Colin McRae Rally 2003 [PLAYABLE]

Verified as playable. The game was in-game for a long time, but for the first time we had some gamebreaking issues fixed (crashing after some time in-game or hanging while changing the camera angle), making it rock solid stability wise (8 hours on the clock playing it nonstop to be sure). Even that annoying “rainbow sun” rendering issue was fixed, so you can enjoy those beautiful sun rays without going blind.

Spikeout: Battle Street [PLAYABLE]

Already playable for some time, but it’s even better now. Some of the visual issues with some textures rendering the wrong color (or not being visible at all) were fixed, and as a bonus even the performance improved. You can watch this side-by-side comparison for more details https://youtu.be/GbMK50IdGiE

Manhunt [IN-GAME]

The 3D models of the enemies used to be invisible at some camera angles and some specific distances (specially at the execution animations), now they’re always visible (still missing some textures). You can see the improvements in action here https://youtu.be/1bA_pBxmGfE

Beatdown - Fists of Vengeance [IN-GAME]

The game still has some visual issues, but the models are slightly less flickering now.

The Sims 2 [INTRO-MENUS]

Compatibility improved. Game reached the intros for the first time.

Blood Wake [PLAYABLE]

The rainbow colors on the water and some other wrong colors (e.g the sand) are now fixed. Here’s a side-by-side comparison for the improvements https://youtu.be/p4zxhm8en74

Dennou Taisen DroneZ [IN-GAME]

Some more geometry is now visible while in-game (still a mess, but better).

F1 2001 [IN-GAME]

Small performance improvement.

Tony Hawk's Underground [IN-GAME]

Verified as in-game.

Malice [INTRO-MENUS]

Verified as able to reach the menus.

Midway Arcade Treasures 2 [IN-GAME]

Verified as able to reach in-game in all games.

Midway Arcade Treasures 3 [IN-GAME]

Verified as able to reach in-game in most games.

Dave Mirra Freestyle BMX 2 [IN-GAME]

Many textures are now visible and only far away parts of the scenarios show major issues (still not confirmed as playable due to the lack of testing).

Hunter The Reckoning Redeemer [IN-GAME]

Previously the game had a hang around 20 seconds into the intro animation, now it’s able to show the entire animation and hangs right before the actual game. Entire intro can be watched here https://youtu.be/BuO9wRVE1Ic

Mortal Kombat Deadly Alliance [INTRO-MENUS]

Verified as able to reach the intros.

Advent Rising [INTRO-MENUS]

Some parts of the intro and menu are now visible.

Intellivision Lives [IN-GAME]

Verified as able to reach in-game in all games.

Dreamfall [IN-GAME]

Always able to reach in-game now (but not much other than the HUD is visible).

Crime LIfe - Gang Wars [IN-GAME]

Menus are visible for the first time and you can see the models being rendered under a black screen (using the wireframe rendering mode).

SSX 3 [IN-GAME]

Verified as able to reach in-game.

Thief - Deadly Shadows [INTRO-MENUS]

Able to reach the “Press Start” screen for the first time.

Max Payne 2 [IN-GAME]

Some of the issues while going between rooms improved (not 100% fixed yet) and some issues with character details also improved. Recent footage of the state of the game here https://youtu.be/_OCeo1cYkSU

Trivial Pursuit Unhinged [x] [IN-GAME]

Verified as able to reach in-game.

Chase - Hollywood Stunt Driver [INTRO-MENUS]

Now 100% of the menus are rendered properly (before we only had most of the screen being black and some minor items being visible at all).

Juiced [INTRO-MENUS]

Verified as able to reach the intros.

Teenage Mutant Ninja Turtles 2 BattleNexus [IN-GAME]

Game is now able to reach in-game for the first time (rendering in-game under a black screen, so not much to see for now).

Teenage Mutant Ninja Turtles 3 Mutant Nightmare [IN-GAME]

Same improvement as TMNT 2, rendering under a black screen.

Spartan - Total Warrior [IN-GAME]

Stability improved. Game used to crash some seconds after reaching in-game, now you’re able to “play” as much as you want without crashing, but rendering alone will prevent you from progressing much (watch this video to understand the issues and as a spoiler I’ll already warn you that’s painful to watch https://youtu.be/8LIJKnATPbo ).

Godzilla Destroy All Monsters Melee [IN-GAME]

The rainbow colors are gone in the game, but we still have some major issues preventing it from being playable.

Godzilla Save the Earth [IN-GAME]

Same as Destroy All Monsters Melee, improved but still not playable.

Burnout 3 - Takedown [IN-GAME]

Game is now able to show the menus for the first time… and we’re able to reach in-game (there’s a major issue preventing most things other than the HUD to be rendered properly for now)

Chicken Little [INTRO-MENUS]

Intro and first loading screen are visible for the first time.

Dragonball Z Sagas [IN-GAME]

Menus and graphics in-game are visible for the first time.

The Nightmare Before Christmas [IN-GAME]

Menus and intros are now visible, but flickering.

Disneys Extreme Skate Adventure [IN-GAME]

Can reach in-game without any hack now (previously needed the “All cores” hack enabled for some machines), and rendering improved https://youtu.be/huL2o9aChco

Kingdom Under Fire: The Crusaders [IN-GAME]

Models of the characters and enemies are visible for the first time. Here’s a 30+ minutes test showcasing the graphics and stability https://youtu.be/okKK2gyMRKE

Outlaw Volleyball [IN-GAME]

Game no longer crashes after a few minutes in-game and some visual issues were fixed (still not confirmed as playable due to the lack of testing).

Largo Winch: Empire Under Threat [IN-GAME]

Verified as able to reach in-game.

IHRA Drag Racing 2004 [IN-GAME]

Rendering improved and performance is slightly better.

Crimson Skies [IN-GAME]

Both rendering and performance improved a lot.

Chicago Enforcer [IN-GAME]

Some more geometry is now visible in-game (still missing major parts of most characters).

Frogger Ancient Shadow  [IN-GAME]

Major gamebreaking issue from the past is now fixed (not rendering part of the scenario in some levels, making it impossible to progress at some point), but there’s still a random crash preventing it from being fully playable.

Sensible Soccer 2006 [IN-GAME]

Verified as able to reach in-game for the first time.

Conflict Desert Storm 2  [IN-GAME]

Rainbow colors in some light effects and textures are now fixed.

Warpath [IN-GAME]

Some more geometry visible in-game.

Crash Twinsanity [IN-GAME]

Most of the geometry rendering improved, so you can progress much more into the game.

Still not playable, but you can check how it looks here https://youtu.be/WWrrU-LWxc4

Dai Senryaku VII [IN-GAME]

In-game now, but it was playable for some days until it regressed. The gamebreaking issue was that the graphics didn’t scaled properly to the screen size at any given resolution (even at native), so you had a “zoomed in” game that you couldn’t see most of it. With this fixed, at some point you were able to finish the entire game, but a regression sneaked in and while it’s still possible to finish the game, you need to wait for a much longer time to get into any level (the emulator will hang for some time and then resume into the mission). Here’s how the game runs after the hang https://youtu.be/Zu6d670fM0M

Nascar Heat 2002 [PLAYABLE]

With a nice combo of improvements, now the game is fully playable (and it’s looking amazing at higher resolutions https://youtu.be/HcaqENWLQ4o ).

Whiteout [IN-GAME]

The rendering issue with the screen being zoomed in while using turbo (the post processing effect used to break the entire screen scaling) is now fixed https://youtu.be/a_CBvpYPe88

Blowout - Military Fighting Unity [IN-GAME]

Mini-map (and the map on the menus) now render at the correct scale and the performance improved. Some footage here with the current status https://youtu.be/IJcJsv1y5Fs

Robotech - Battlecry [IN-GAME]

Graphics regression was fixed (pretty sure no one even noticed this happened), and as a bonus some other visual issues in some scenarios also were fixed https://youtu.be/B47ernAyAcQ

Batman - Rise of Sin Tzu [IN-GAME]

Another gamebreaking issue fixed: when some enemies used the gas grenade, the post processing effect that covers the screen used to break the entire screen scaling, making it zoomed in and impossible to see more than 1/4 of anything.

Tony Hawk's Pro Skater 2X [PLAYABLE]

Ghosting visual issue fixed.

Drihoo [IN-GAME]

Compatibility improved a lot (going from not even booting to in-game).

Curse - The Eye of Isis [IN-GAME]

Verified as able to reach in-game.

The Warriors [IN-GAME]

After a nice combo of improvements, this game progressed from broken to “almost playable” (a single issue with the sound looping is the only thing left to reach our beloved status). Here’s some footage of ass being kicked https://youtu.be/vEM6igFnqmU

Final Words

After all this, what can we expect for the next month? Even more improvements, of course! You can take a look at our PRs to have some clues about what’s going to be the next big things here. As always, you can also check a detailed list with all the up to date results for all the games tested so far in the pdf file below. That’s all folks, see you again in the next report ;)