Visual Studio 6.0 crash fix (buffer overflow)

Discussion in 'Mixed Languages' started by kebabstorm, May 3, 2022.

  1. kebabstorm

    kebabstorm MDL Junior Member

    Aug 3, 2016
    94
    121
    0
    #1 kebabstorm, May 3, 2022
    Last edited: May 3, 2022
    Code:
    // Microsoft Visual Studio 6.0
    // Crash Fix (Buffer Overflow)
    //
    // Module: Resource Editor
    // DEVRES.PKG v6.0.8168.0 - 17.6.1998 0:00
    //
    // Original SHA1 : 59afd55f13310dcdbfff777fe6f4c7d0a8191a82
    // Fixed SHA1    : 00bb8497adca2467eaba022a34bf4fdafd3d7c6c
    //
    
    --------
    
    
    0x00004518 / 0x50403518:
    
    FF 25 74 1A 40 50    ; jmp     ds:__imp_??2@YAPAXI@Z ; operator new(uint)
    ->
    E9 8F 0F 10 00 90    ; jmp     0001054AC ; nop
    
    --------
    
    
    0x001054AC / 0x505044AC:
    
    00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
    ->
    55 89 E5 8B 45 08 B9 02 00 00 00 F7 E1 50 FF 15 74 1A 40 50 59 89 EC 5D C3
    
    55                      push   ebp
    89 E5                   mov    ebp,esp
    8B 45 08                mov    eax,DWORD PTR [ebp+0x8]
    B9 02 00 00 00          mov    ecx,0x2
    F7 E1                   mul    ecx
    50                      push   eax
    FF 15 74 1A 40 50       call   DWORD PTR ds:0x50401a74
    59                      pop    ecx
    89 EC                   mov    esp,ebp
    5D                      pop    ebp
    C3                      ret
    
    --------
    
    
    0x000001E8:
    
    AC 34 10 00
    ->
    DC 34 10 00          ; increase .text section virtual size by 30 bytes
    
    
    --------
    
    
    0x00000140:
    
    29 8D 19 00
    ->
    77 FD 17 00          ; fix PE checksum
    
    
    --------
    
    Explanation

    The real reason why Visual Studio 6 crashes on new hardware has actually nothing to do with your OS, it works just fine in later Windows versions too but there are multiple buffer overflows in the resource editor module (DEVRES.PKG) which have seemingly gone unnoticed all these years.

    The resource editor module has multiple calls to the C++ new[] operator using values that are too small without doing bound checking. But because of the way memory allocation and alignment works, it just always got "lucky" and did not crash, as the pointer memory block had more "extra" space than requested and then it wasn't overwritten or accessed by anything else before the operation was completed.

    What has changed since then, is every part of your computer has become a lot faster; processor, memory, I/O, the works. There were no 16 core 32 thread processors in the early 2000s, nor 5000MB/s SSDs or fast memory. So now when running on a modern processor, what happens is that the memory which resource editor didn't properly allocate, gets overwritten by another thread while it's still being used (yes, VS6 is multithreaded, even if very lightly so).

    That's also the reason why all sorts of different bags of tricks work to "fix" it, what they have in common is that they change the execution path (by slowing processing down and/or changing how memory is allocated/aligned) slightly so by luck you pass the buffer overflow without crashing. The compatibility layer slows it down, as does running it in a VM. Running it from a HDD instead of SSD can make the difference, changing folders, restarting the computer, or even something like running a 4K video stream in the background.

    I know it's 2022 but I don't think Visual Studio 6 is any less useful today than it ever was. The resource editor is still the best comparing to all the new editions, and even if I code something in VS2019 I still use the editor in VC6 to design the dialogs. Also if you want to just code a quick tool which will work in any Windows version without any redistributables, VC6 is still the pick.

    In any case, there are two ways to fix the crash for good.

    The Fix

    1. Delete the "DEVRES.PKG" file, but you will lose the resource editor.

    or

    2. Patch the buffer overflow in "DEVRES.PKG" with a hex editor.