Thread overview
Strange AV in asm mode (code only for amd64)
Nov 05, 2017
user1234
Nov 05, 2017
user1234
Nov 06, 2017
user1234
Nov 05, 2017
Eugene Wissner
Nov 05, 2017
user1234
November 05, 2017
Hello, try this:

---
import std.stdio;

alias Proc = size_t function();

size_t allInnOne()
{
    asm pure nothrow
    {
        mov RAX, 1;
        ret;
        nop;nop;nop;nop;nop;nop;nop;
        mov RAX, 2;
        ret;
    }
}

void main()
{
    Proc proc1 = &allInnOne;
    Proc proc2 = cast(Proc) (cast(void*)proc1 + 16);
    writeln(proc1(), " ", proc2());
}
---

The call to proc1() gens a SEGFAULT after the first RET.
Remove the call to proc1() and it works.

Why that ?
November 05, 2017
On Sunday, 5 November 2017 at 13:43:15 UTC, user1234 wrote:
> [...]

Hmmm it was just the amount of nops.

---
import std.stdio;

alias Proc = size_t function();

size_t allInnOne()
{
    asm pure nothrow
    {
        naked;
        mov RAX, 1;
        ret;
        nop;nop;
        mov RAX, 2;
        ret;
    }
}

void main()
{
    Proc proc1 = &allInnOne;
    Proc proc2 = cast(Proc) (cast(void*)&allInnOne + 8);
    writeln(proc1(), " ",proc2());
}
---

November 05, 2017
On Sunday, 5 November 2017 at 13:43:15 UTC, user1234 wrote:
> Hello, try this:
>
> ---
> import std.stdio;
>
> alias Proc = size_t function();
>
> size_t allInnOne()
> {
>     asm pure nothrow
>     {
>         mov RAX, 1;
>         ret;
>         nop;nop;nop;nop;nop;nop;nop;
>         mov RAX, 2;
>         ret;
>     }
> }
>
> void main()
> {
>     Proc proc1 = &allInnOne;
>     Proc proc2 = cast(Proc) (cast(void*)proc1 + 16);
>     writeln(proc1(), " ", proc2());
> }
> ---
>
> The call to proc1() gens a SEGFAULT after the first RET.
> Remove the call to proc1() and it works.
>
> Why that ?

One of the problems is that "naked" is missing in your assembly. If you write

asm pure nothrow
{
     naked;
     mov RAX, 1;
     ret;
     nop;nop;nop;nop;nop;nop;nop;
     mov RAX, 2;
     ret;
}

writeln(proc1()) works. Without "naked" dmd generates the prologue and epilogue for your function. Inside the assembly you return from the function without restoring the stack. It causes the segfault. So you have to write the prologue before returning or use nacked assembly.
With "naked" and "Proc proc2 = cast(Proc) (cast(void*)proc1 + 8);" the example works.
November 05, 2017
On Sunday, 5 November 2017 at 14:27:18 UTC, Eugene Wissner wrote:
> On Sunday, 5 November 2017 at 13:43:15 UTC, user1234 wrote:
>> [...]
>
> One of the problems is that "naked" is missing in your assembly. If you write
>
> asm pure nothrow
> {
>      naked;
>      mov RAX, 1;
>      ret;
>      nop;nop;nop;nop;nop;nop;nop;
>      mov RAX, 2;
>      ret;
> }
>
> writeln(proc1()) works. Without "naked" dmd generates the prologue and epilogue for your function. Inside the assembly you return from the function without restoring the stack. It causes the segfault. So you have to write the prologue before returning or use nacked assembly.
> With "naked" and "Proc proc2 = cast(Proc) (cast(void*)proc1 + 8);" the example works.

Yeah thanks, i figured it out too. Also there was too much nops.
November 06, 2017
On Sunday, 5 November 2017 at 14:25:24 UTC, user1234 wrote:
> On Sunday, 5 November 2017 at 13:43:15 UTC, user1234 wrote:
>> [...]
>
> Hmmm it was just the amount of nops.
>
> ---
> import std.stdio;
>
> alias Proc = size_t function();
>
> size_t allInnOne()
> {
>     asm pure nothrow
>     {
>         naked;
>         mov RAX, 1;
>         ret;
>         nop;nop;
>         mov RAX, 2;
>         ret;
>     }
> }
>
> void main()
> {
>     Proc proc1 = &allInnOne;
>     Proc proc2 = cast(Proc) (cast(void*)&allInnOne + 8);
>     writeln(proc1(), " ",proc2());
> }
> ---

That's a nice trick against static analysis. I imagine well an attacker trying to fight against the first part, even if never executed, assuming he found that this function got executed and then he assumes that the code get executed from the start.