Agfdhyk

I'm thinking of writing a JIT library that would compile and execute untrusted code on Windows, Mac, and Linux. The language that the untrusted code is written in is "safe", in the sense that: arrays are bound-checked, it is impossible to have pointers to inaccessible memory, and there is no way to make system calls. Furthermore, the stack space used by each function of the untrusted code is fixed during compilation (i.e. no alloca() or C-style variable length arrays).

However, this language can make recursive function calls, which means that a stack overflow can potentially occur.

Usually, stack overflows will cause a segmentation fault because the OS will add an inaccessible guard page at the end of the stack (at least, all three OSes that I care about will do this). Assuming I don't handle SIGSEGV or otherwise try to catch the segmentation fault, this will cause a crash, bringing down my JIT runtime and the rest of the application with it. That's good - the untrusted code didn't manage to wrest control over the application or access data from it.

But apparently, if a function allocates a large amount of stack memory (e.g. by declaring a large array on the stack), it is possible to "jump" over the guard page. So carefully-crafted code can be made to overflow the stack without causing a segmentation fault. That means that it would be possible to craft the untrusted code to allocate enough stack memory such that my JIT library would compile it to machine code that jumps the guard page. Such malicious code would be able to do bad things, such as: overwriting data from the rest of the application, so that when the JITted code returns to the application, the application would read the modified data and do things that it was not intended to do. This means that the malicious code has effectively gained control over the whole application (and would be able to do anything the application had the permission to do).

What is the proper way to detect a stack overflow (either crashing the whole program, or returning control to the JIT runtime so that I can handle it)?

I could conceivably think of two possible ways:

• Query from the OS the remaining stack space, and check it at the prologue of every function. If there is not enough stack space that it can be made to call abort(). It can be queried for Linux like this answer, and there's a way for the other OSes as well.


• Probe the required stack space at the prologue of every function. This will access the guard page if there is not enough stack space. This is the case for WAVM.