A little trick I invented to hook syscall made by gamemon.des sometime ago. It only works on Windows XP and above, due to the design of syscall mechanism itself. So how it works? Lets get into some research about syscall mechanism.
As you know NT-based Windows (Windows NT/2k/XP/2k3/Vista) have ring0 and ring3 layer, each of them holds the kernelmode and usermode stuff respectively. Usermode is isolated from kernelmode, means you cannot access kernelmode in anyway from usermode. So in order to communicate with kernelmode, you make a syscall. Processor will transfer control to kernelmode, kernelmode processes your syscall and transfer the control back to usermode again.
Each version of Windows have different syscall mechanism. For Windows 2000 and older, the syscall mechanism is as shown below.
MOV EAX, SyscallNumber ; requested syscall number
LEA EDX, [ESP+4] ; EDX = params...
INT 2Eh ; throw the execution to the KM handler
RET 4*NUMBER_OF_PARAMS ; return
Each syscall number is moved into EAX and invoked through INT 2Eh. In Windows XP and later, the syscall mechanism had changed.
MOV EAX, 101h ; syscall number: NtTerminateProcess
MOV EDX, 7FFE0300h ; EDX = 7FFE0300h
CALL EDX ; call 7FFE0300h
Notice the difference. Instead of INT 2Eh, now it is replaced by CALL EDX which leads us to ntdll.KiFastSystemCall, a tiny stub containing the SYSENTER instruction.
MOV EDX, ESP
This is where KiFastSystemCall hook came in. We can install hook on the stub to catch all syscalls made by the process, including undocumented NtUser*, NtGdi* syscalls.