Many of you are probably already familiar with the x64 calling convention1 2 in 64-bit Windows - where generally speaking first four parameters3 are passed in registers RCX, RDX, R8 and R9 with 32 bytes of spill area reserved on stack just in case callee has to store the parameters on stack in order to free up the registers. Rest of the parameters (if any i.e. fifth parameter onwards) end up being on stack just like in x86 32bit. That is the essence of the calling convention and because of four dedicated registers, you will see more movs and less pushes and pops in 64-bit assembly when function calls are made.

Also since stack needs to be aligned to 16-bit boundaries before entering a function, you may see stack adjustments that do not exactly correspond to stack usage requirements. For example a function that does not take any parameters may have to eat 8 bytes off the stack before calling a child function, since return address on stack claimed 8 bytes and made RSP not 16 byte aligned any more. In general you will see functions adjusting stack for 0x20 or 0x28 bytes because of the 32-byte shadow area and stack alignment requirements.

One other thing to remember about this convention, is that the callee never cleans up the stack. While this allows variable argument functions to work in 64-bit, it makes it little bit harder to find out just how many parameters a function takes in 64-bit. Also the artifacts of previous calls on the same stack make it a bit trickier to decipher and understand program state. On the other hand, the stack pointer (RSP) does not jump around as much as it does in 32-bit because functions typically reserve upfront the maximum stack space they would need for calling other functions. RBX, RBP, RDI, RSI, R12-R15 are preserved across function calls.

If a function is returning more than 64-bit of data4, RCX ends up having a reference to returned structure rather than the first parameter. The actual first parameter (if any) is passed in RDX instead. This could be seen as a hidden first argument, with the caveat that the compiler arranges to copy to this argument before returning from the function.

If you use gcc to compile 64-bit binaries on Linux however you will see the generated code follows a different convention. It uses RDI, RSI, RDX, RCX,  R8 and R9 in that order for first six integer/pointer arguments instead. Linux5 follows what is known as AMD64 Application Binary Interface (ABI) specification. Since RDI and RSI are used as arguments, obviously they are not preserved across function calls like in Windows.

Tagged with →  
Share →

2 Responses to x64 calling convention

  1. Eldad says:

    I am trying to use CL 16.0 for x64 (VS 2010) to produce some readable ASM code for an example, and CL insists on preallocating a ton of stack space (28h bytes), with the following line:

    sub rsp, 40 ; 00000028H

    Question is, how can I disable this behavior? It is difficult to explain to the class and I like to show them clean, explicable code…

    Surely it doesn’t need that space. On x86, this seems to be controlled by the edit-and-continue switches (/Zi vs. /ZI), but these don’t have any effect in the x64 case. Any idea how to make CL only allocate as much stack as it actually needs?

  2. Satya Das says:

    If you are making any direct or indirect call to another function, that is normal. If you write a leaf function you may have better luck in minimizing stack requirements.

Leave a Reply

Your email address will not be published. Required fields are marked *

*

Looking for something?

Use the form below to search the site:


Still not finding what you're looking for? Drop us a note so we can take care of it!

Visit our friends!

A few highly recommended friends...

Set your Twitter account name in your settings to use the TwitterBar Section.