I've been trying to patch ntoskrnl.exe on Windows Home Server 2011 to increase the memory limit from 8GB to 16GB (or simply remove it altogether). I've been operating on the presumption that I can patch it in the same way Vista 32-bit was patched to remove the 4GB limit (ie. Geoff Chappell's work). After spending the past 3 days poking through disassembly, I'm at a bit of a loss. Even with symbols, there doesn't seem to be a Unicode string for Kernel-WindowsMaxMemAllowedx64 anywhere. I'm hoping someone here is more informed on how the limit is applied in 64-bit Windows and can nudge me in the right direction. I'm not really looking for someone to give the answer outright as I'm hoping to learn something new in the process. Also, in case it helps, WHS2011 SP1 uses the same ntoskrnl.exe as Windows 7 x64 SP1 and Windows 2008 R2 SP1, so the limit would be loaded in the same way in those systems. Edit: It was actually a problem with my disassembly. For whatever reason, it was incomplete. There indeed is a reference to Kernel-WindowsMaxMemAllowedx64 in ntoskrnl.exe. Patching that, I was able to lift the lame 8GB limit.
I'm running into the same problem as you - IDA doesn't want to show the full disassembly and I don't see any Xrefs to Kernel-WindowsMaxMemAllowedx64. Any ideas?