What Is ReadyToRun (R2R)?
.NET has a variety of deployment models and runtime environments, so I plan to cover them separately in a future post.
Normally, .NET code is compiled to native code at runtime by the JIT (Just-in-Time) compiler.
With ReadyToRun deployment, however, some code is precompiled to native code, while the rest is compiled at runtime.
Apart from increasing the size of assemblies, it has few major limitations, making it easy to adopt.
Unable to Debug R2R Minidumps
When a game server crashes, we typically capture a full dump containing all of its memory to investigate the cause.
Writing all that memory takes time and can go wrong along the way, so we often capture a minidump with minimal information, such as the call stack, before capturing the full dump.
In our project, we stored not only PDB files but also binaries such as DLLs and EXEs on a symbol server.
However, we ran into an issue where the binaries needed to debug a minidump from a ReadyToRun deployment could not be retrieved from the symbol server.
Finding the Cause
I was able to debug the full dump without any issues.
I didn't investigate closely, but I suspected it worked because the R2R binaries were loaded in the memory captured by the full dump.
On inspection, I found that the larger R2R binaries had not been stored on the symbol server.
Symstore had skipped them because it couldn't recognize their file type.
Error message:
skip (not a known file type. ErrorLevel is 13)
I tried uploading the original binaries to the symbol server, but they couldn't be used because the key used by Symstore and the debugger includes size information.
TimeDataStamp(4bytes) + SizeOfImage(3bytes)
Portable Executable - Wikipedia
R2R increased the binary size, so the latter part of the key changed.
- Before: CF33DFB3388000
- After: CF33DFB37d2000
I could debug the minidump by manually finding the corresponding R2R binaries and placing them next to the dump file, or by using only full dumps, but neither option was convenient.
Attempt 1 - Failed
You can use MINIDUMP_TYPE to specify what to include in a minidump.
MINIDUMP_TYPE (minidumpapiset.h) - Win32 apps | Microsoft Learn
I wondered whether I could include the loaded R2R binaries in the minidump.
Using MiniDumpWithModuleHeaders did include the binaries, but the debugger didn't recognize them.
Even if it had worked, the increased size would have defeated the purpose of a minidump.
Attempt 2 - Solution
No matter which options I gave Symstore, it wouldn't upload the larger R2R binaries.
Since the files I wanted to upload were ordinary binaries rather than symbol files, I figured the debugger should be able to find them if I uploaded them directly to the symbol server.
The build script was written in Python, so I used Python to calculate the symbol key from the PE header information and upload the binaries directly to the symbol server.
pip install pefile
def get_symbol_key(filename):
pe = pefile.PE(filename)
return format(pe.FILE_HEADER.TimeDateStamp, 'x').upper() + format(pe.OPTIONAL_HEADER.SizeOfImage, 'x').upper()
The debugger could now find the binaries when debugging minidumps, too.
Issues with SingleFile Deployment
By default, the .exe in a .NET application is just a bootstrapper; the actual code is loaded and executed from .dll files.
To simplify deployment, .NET offers a SingleFile option that bundles everything into one exe.
The problem was that when using SingleFile and ReadyToRun together, I couldn't find the larger .dll file in the output folder.
Example output folder:
- Output/Release
- Program.exe (bundled file)
- Output/Release/Win-x64
- Program.dll (original size before R2R)
Because C# compilation is deterministic, I could build twice—once with SingleFile on and once with it off—then deploy the bundled file and manually upload the R2R DLL to the symbol server.
To save build time, I instead added a step to extract the binaries from the SingleFile bundle and upload them to the symbol server.
Install ilspycmd
dotnet tool install --global ilspycmd
Usage
ilspycmd -d -o 출력폴더 Program.exe(통합파일)