What is ReadyToRun (R2R)?

The deployment methods and execution environments of .NET are diverse, and I plan to write a separate post to summarize them later.

Typically, .NET code is compiled natively at runtime by the Just-in-Time (JIT) compiler. However, when deploying with the ReadyToRun option, some parts are pre-compiled natively, while the rest is compiled at runtime.

There are no significant limitations other than an increase in the size of the assembly, making it easy to apply.

Unable to Debug R2R Mini Dumps

When a crash occurs on a game server, a full dump that includes the entire memory is usually created to identify the cause.

Since capturing the entire memory can take a long time and may encounter issues along the way, a mini dump containing only minimal information, such as the call stack, is often created first, followed by a full dump.

In our project, we were storing not only PDB files but also binaries like DLLs and EXEs on the symbol server. However, there was an issue where the binaries corresponding to the mini dump with the ReadyToRun deployment option could not be retrieved from the symbol server.

Identifying the Cause

The full dump could be debugged without any issues. Although I didn't investigate in detail, it seemed that the R2R binaries were loaded into the full dump memory, which was not problematic.

Upon checking, I found that the larger R2R binaries were not being uploaded to the symbol server. The symstore failed to upload the file due to type recognition issues.

Error message:

skip (not a known file type. ErrorLevel is 13)

I tried uploading the binaries before they increased in size to the symbol server, but since the keys used by symstore and the debugger include size information, this binary was unusable.

TimeDataStamp (4 bytes) + SizeOfImage (3 bytes)

Portable Executable - Wikipedia

When R2R is applied, the binary size increases, causing the latter part to change:

• Before: CF33DFB3388000
• After: CF33DFB37d2000

While it was possible to manually find the R2R binary corresponding to the mini dump and debug it alongside the dump file, or to use only the full dump for debugging, it was inconvenient.

Attempt 1 - Failure

You can specify which MINIDUMP_TYPE to include in the mini dump.

MINIDUMP_TYPE (minidumpapiset.h) - Win32 apps | Microsoft Learn

I thought including the R2R binary loaded in the mini dump might work.

Using MiniDumpWithModuleHeaders would include the binary, but the debugger could not recognize it. Even if it were possible, the size would increase, negating the purpose of the mini dump.

Attempt 2 - Solution

No matter what options were provided, symstore did not upload the R2R enlarged binaries. Since the files I wanted to upload were not symbol files but simple binaries, I thought directly uploading them to the symbol server would not cause any issues for the debugger.

The build script was written in Python, so I used Python to retrieve the symbol key from the PE header information and uploaded it directly to the symbol server:

pip install pefile

def get_symbol_key(filename):
    pe = pefile.PE(filename)
    return format(pe.FILE_HEADER.TimeDateStamp, 'x').upper() + format(pe.OPTIONAL_HEADER.SizeOfImage, 'x').upper()

Now, the binaries can be correctly found in the mini dump.

Issues When Deploying with SingleFile

By default, .NET executables are merely bootstrap files, with the actual code loaded and executed from DLLs.

To make deployment easier, there is an option called SingleFile that bundles everything into a single EXE. However, the problem arises when using SingleFile with ReadyToRun, as I could not find the enlarged DLL files in the output folder.

Example output folder:

• Output/Release
  • Program.exe (bundled file)
• Output/Release/Win-x64
  • Program.dll (original size before R2R was applied)

Since C# has deterministic compilation, I could resolve this by building twice with SingleFile on/off, deploying the bundled file, and manually uploading the R2R-applied DLL to the symbol server.

To save build time, I added a process to extract the sub-binaries from the SingleFile deployed file and upload them to the symbol server.

Install ilspycmd:

dotnet tool install --global ilspycmd

Usage:

ilspycmd -d -o output_folder Program.exe (bundled file)