Sins of the PS2: True Crime Streets of L.A.

What happened to [this game] was a true crime...

The most enjoyable thing about PS2 emulation is how broken PS2 games are. It always seems like there's something new to uncover, and each game is different from the last. Some games are so broken, they deserve to have an article written about their sins. True Crime: Streets of L.A. is one of those games, and last year I wanted to see why it never worked on PCSX2. No matter what settings were used, the game would always crash on menus with the dreaded "TLB miss" message. Prafull of the PCSX2 team had prepared a patch which allowed True Crime to go in-game, but unfortunately the game crashed in one of the missions after severe graphical glitches. Thus, True Crime remained unplayable.

Of note was that this game didn't work on any emulator, so this wasn't just a PCSX2-specific issue. This meant that True Crime was abusing some unknown edge case in the hardware. Of course, it caught my attention. After acquiring a copy of the game, I decided to reverse-engineer it, hoping to unravel its secrets. It was last year that I dove into the abyss, with no idea what was going wrong...

A TLB Primer

Those of you who have used PCSX2 for a while may have come across the "TLB miss" error. In some games this is harmless, but most of the time, any game that hits this error will crash. True Crime hits this error too, and here it definitely stops working. What's the deal?

The TLB is a fast way to handle memory management. During startup, the PS2 BIOS initializes the TLB by defining where all valid memory exists. Although it is possible for games to change the TLB, most games, including True Crime, use the BIOS's default mapping. A TLB miss occurs when a game tries to access memory that isn't defined in the TLB. This triggers an exception on the CPU, which can be handled in a variety of ways. On a typical operating system, a TLB miss could either terminate the process causing it or add new memory for the process to access. The PS2 does not have a true OS, and only one process (the game) is running at all times. Because of this, a TLB miss indicates a fatal error, so the kernel tries to communicate with debugging software to report the error. As retail games do not have debuggers, the console hangs.

PCSX2 is not a real console, and so it takes some liberties with TLB misses. It chooses to simply log and otherwise ignore those errors, under the assumption that a game that TLB misses can still work normally. TLB misses are not necessarily emulation errors either: they can be caused by bad dumps or faulty patches. Here the TLB miss is very much an emulation error, something so insidious that no one could have predicted it.

The Great Unknown

Aside from the TLB miss, I had no clue what was going wrong. Several megabytes of game code awaited me, and analyzing the entire executable would have driven me even more mad than I already am. I needed to play it smart and think about why the game could be crashing.

Looking at the ISO's contents, I saw that True Crime had a custom module called STMMAN.IRX. My previous article on Dead or Alive 2 discussed how most games have custom modules running on the PS2's Input/Output Processor. To my great delight, I found that STMMAN had full debug symbols, meaning that I could discern all function names and gain a better understanding of the code more quickly!

Even without looking at the code, I knew just by looking at the function names that this custom module is responsible for streaming data from the DVD. I hypothesized that a problem was occurring with STMMAN, and the TLB miss was caused by the game being unable to load some asset from the disc. The game proper had no symbols, but after several days, I was able to piece together how the game talked with the streaming module. As it turned out, I was on the right track, but the actual issue still eluded me...

Bugspiracy

What's truly beautiful about this game is that it has multiple issues all working in tandem, and if only one bug were fixed, the game would work just fine on an emulator.

Let's work backwards. As I correctly guessed, the TLB miss was caused by a game asset not loading. True Crime tries to read from a CDVD stream, but this fails. The game assumes that the read was successful anyway, so it tries to access a NULL pointer and dies.

Why did the CDVD stream read fail? At some point, the stream gets erased so that it no longer points to valid data. Normally, when the game reads assets, it will open a CDVD stream, read from it, and close the stream. However, during initialization, the game opens three streams and assumes they will always remain open. It was from one of these streams that the game read from, but all three streams had been erased.

The reason the streams were erased was because, for whatever reason, the game re-initializes the streaming library multiple times. When the STMMAN init function is called, all opened streams will be erased. There was no reason for the game to call this function multiple times, and patching out all redundant calls was sufficient for the game to work.

But that does not explain why the game works on real hardware. Those multiple calls will happen regardless, and the streaming library will not re-initialize itself if a certain "is_initialized" flag is set. At the end of initialization, the game sets this variable... so then why was it getting set back to zero by the time the initialization function was called again?

Undocumentation

No one could have anticipated the answer. The code in PCSX2 that triggered the game's bug was written many, many years ago, and the game (likely unintentionally) relies on such an obscure edge case that any reasonable person would forgive PCSX2 for not having handled it.

Certain streaming functions use a 4-byte receive buffer to get a reply from the IOP that tells certain information. The is_initialized variable, mentioned in the previous section, is stored right next to this buffer. This is the game's sin, for this relies on undefined behavior in the PS2's DMA engine. The DMAC, which handles DMA on the EE side, can only transfer in units of quadwords, or 16 bytes. This means that whenever the IOP sends its reply, the is_initialized variable gets overwritten. Unlike the EE DMAC, IOP DMA works in units of words, or 4 bytes. STMMAN only specifies a transfer length of 1 word, which means that only the data that goes into the receive buffer is defined. What is is_initialized being overwritten with?

It is actually a common case for IOP DMA data lengths to not be aligned on a quadword boundary, as the IOP has no obligation to do so. PCSX2 handles this by reading additional data from IOP memory. However, STMMAN never initializes the additional data being read, which means that it will always be zeroed out. Thus, is_initialized will always be set to zero, causing the bugged chain reaction mentioned above.

As an experiment, on DobieStation I modified IOP DMA to append ones to unaligned DMA transfers, and this made the game work. However, this is likely not what real hardware does. SIF, the interface used for the EE and IOP to communicate over DMA, has a FIFO where all intermediate data is stored. Hardware tests had previously confirmed that if a FIFO were to overflow - more data is stored in it than it can fit - old data is erased with the new data. My guess was that unaligned transfers would take old data from previous transfers within the FIFO and append it to the current transfer to align the length to a quadword boundary. Implementing this also fixes True Crime on DobieStation. As for PCSX2, as I didn't feel comfortable tinkering with it's DMA code, I submitted a patch for the NTSC version of the game. On the newest dev versions, True Crime is now able to go in-game. All in all, a happy ending for a game that was broken for years!

Though the game is no jaw-dropper, you can check out some footage of it on PCSX2: https://www.youtube.com/watch?v=pmnHy-hSZi4