patching Kernel-WindowsMaxMemAllowedx64 in WHS2011

Discussion in 'Mixed Languages' started by Transient, Mar 20, 2012.

  1. Transient

    Transient MDL Novice

    Dec 21, 2007
    15
    0
    0
    #1 Transient, Mar 20, 2012
    Last edited: Mar 26, 2012
    I've been trying to patch ntoskrnl.exe on Windows Home Server 2011 to increase the memory limit from 8GB to 16GB (or simply remove it altogether). I've been operating on the presumption that I can patch it in the same way Vista 32-bit was patched to remove the 4GB limit (ie. Geoff Chappell's work).

    After spending the past 3 days poking through disassembly, I'm at a bit of a loss. Even with symbols, there doesn't seem to be a Unicode string for Kernel-WindowsMaxMemAllowedx64 anywhere.

    I'm hoping someone here is more informed on how the limit is applied in 64-bit Windows and can nudge me in the right direction. I'm not really looking for someone to give the answer outright as I'm hoping to learn something new in the process. :)

    Also, in case it helps, WHS2011 SP1 uses the same ntoskrnl.exe as Windows 7 x64 SP1 and Windows 2008 R2 SP1, so the limit would be loaded in the same way in those systems.

    Edit: It was actually a problem with my disassembly. For whatever reason, it was incomplete. There indeed is a reference to Kernel-WindowsMaxMemAllowedx64 in ntoskrnl.exe. Patching that, I was able to lift the lame 8GB limit.
     
  2. wtoriano

    wtoriano MDL Novice

    Mar 27, 2012
    1
    0
    0
    How do I do this?
     
  3. RottenMutt

    RottenMutt MDL Novice

    Apr 23, 2010
    1
    0
    0
    I would be interested in this as well, my server has 16GB and can only use 8GB as you noted...
     
  4. AJolly

    AJolly MDL Novice

    Jun 3, 2011
    1
    0
    0
    I'm running into the same problem as you - IDA doesn't want to show the full disassembly and I don't see any Xrefs to Kernel-WindowsMaxMemAllowedx64. Any ideas?
     
  5. correcto

    correcto MDL Novice

    Jun 25, 2010
    12
    2
    0
    #5 correcto, Dec 18, 2015
    Last edited: Dec 18, 2015
    Transient:
    Were you able to patch ntoskernl as your last comment? Please share if doable, thanks
     
  6. QuantumBug

    QuantumBug MDL Developer

    Mar 7, 2012
    1,485
    1,321
    60
    So you're looking for a PAE?
     
  7. PAYMYRENT

    PAYMYRENT MDL Developer

    Jul 28, 2009
    1,462
    419
    60
    Time for replacing strings again...