x64 calling convention

Many of you are probably already familiar with the x64 calling convention^{1 2} in 64-bit Windows – where generally speaking first four parameters³ are passed in registers RCX, RDX, R8 and R9 with 32 bytes of spill area reserved on stack just in case callee has to store the parameters on stack in order to free up the registers. Rest of the parameters (if any i.e. fifth parameter onwards) end up being on stack just like in x86 32bit. That is the essence of the calling convention and because of four dedicated registers, you will see more movs and less pushes and pops in 64-bit assembly when function calls are made.

Also since stack needs to be aligned to 16-bit boundaries before entering a function, you may see stack adjustments that do not exactly correspond to stack usage requirements. For example a function that does not take any parameters may have to eat 8 bytes off the stack before calling a child function, since return address on stack claimed 8 bytes and made RSP not 16 byte aligned any more. In general you will see functions adjusting stack for 0x20 or 0x28 bytes because of the 32-byte shadow area and stack alignment requirements.

One other thing to remember about this convention, is that the callee never cleans up the stack. While this allows variable argument functions to work in 64-bit, it makes it little bit harder to find out just how many parameters a function takes in 64-bit. Also the artifacts of previous calls on the same stack make it a bit trickier to decipher and understand program state. On the other hand, the stack pointer (RSP) does not jump around as much as it does in 32-bit because functions typically reserve upfront the maximum stack space they would need for calling other functions. RBX, RBP, RDI, RSI, R12-R15 are preserved across function calls.

If a function is returning more than 64-bit of data⁴, RCX ends up having a reference to returned structure rather than the first parameter. The actual first parameter (if any) is passed in RDX instead. This could be seen as a hidden first argument, with the caveat that the compiler arranges to copy to this argument before returning from the function.

If you use gcc to compile 64-bit binaries on Linux however you will see the generated code follows a different convention. It uses RDI, RSI, RDX, RCX,  R8 and R9 in that order for first six integer/pointer arguments instead. Linux⁵ follows what is known as AMD64 Application Binary Interface (ABI) specification. Since RDI and RSI are used as arguments, obviously they are not preserved across function calls like in Windows.