· 9 years ago · Nov 29, 2016, 05:30 PM
1Hello there! You are currently visiting gopherspace through a
2proxy. To learn more about gopher and how to browse it, read this.
3______________________________________________________________________
4
5###################################################################
6 Writing C software without the standard library
7 Linux Edition
8###################################################################
9There are many tutorials on the web that explain how to build a
10simple hello world in C without the libc on AMD64, but most of them
11stop there.
12
13I will provide a more complete explanation that will allow you to
14build yourself a little framework to write more complex programs.
15The code will support both AMD64 and i386.
16
17Major credits to http://betteros.org/ which got me into researching
18libc-free programming.
19
20Why would you want to avoid libc?
21- Your code will have no dependencies other than the compiler.
22- Not including the massive header files and not linking the
23 standard library makes compilation faster. It will be nearly
24 instantaneous even for thousands of lines of code.
25- Executables are incredibly small (the http mirror server for my
26 gopherspace is powered by a 10kb executable).
27- Easy to optimize for embedded computers that have very limited
28 resources.
29- Easy to port to other architectures as long as they are
30 documented, without having to worry whether the libs you use
31 support it or not.
32- Above all, it exposes the inner workings of the OS, architecture
33 and libc, which teaches you a lot and makes you more aware of
34 what you're doing even when using high level libraries.
35- It's a fun challenge!
36
37I might not be an expert yet, but I will share my methods with you.
38
39For now this guide is linux-only, but I will be writing a windows
40version when I feel like firing up a virtual machine.
41
42###################################################################
43 Basic AMD64 Setup
44###################################################################
45When we learn C, we are taught that main is the first function
46called in a C program. In reality, main is simply a convention of
47the standard library.
48
49Let's write a simple hello world and debug it.
50We will compile with debug information (flag -g) as well as no
51optimization (-O0) to be able to see as much as possible in the
52debugger.
53-------------------------------------------------------------------
54$ cat > hello.c << "EOF"
55#include <stdio.h>
56
57int main(int argc, char* argv[])
58{
59 printf("hello\n");
60
61 return 0;
62}
63EOF
64
65$ gcc -O0 -g hello.c
66$ ./a.out
67hello
68
69$ gdb a.out
70(gdb) break main
71(gdb) run
72(gdb) backtrace
73#0 main (argc=1, argv=0x7fffffffd7f8) at hello.c:6
74-------------------------------------------------------------------
75
76Hmm... seems like gdb is hiding stuff from us. Let's tell it that
77we actually care about seeing libc functions:
78-------------------------------------------------------------------
79(gdb) set backtrace past-main on
80(gdb) set backtrace past-entry on
81(gdb) bt
82#0 main (argc=1, argv=0x7fffffffd7f8) at hello.c:6
83#1 0x00007ffff7a5f630 in __libc_start_main (main=0x400556 <main>,
84 argc=1, argv=0x7fffffffd7f8, init=<optimized out>,
85 fini=<optimized out>, rtld_fini=<optimized out>,
86 stack_end=0x7fffffffd7e8)
87 at libc-start.c:289
88#2 0x0000000000400489 in _start ()
89-------------------------------------------------------------------
90
91That's much better! As we can see, the first function that's really
92called is _start, which then calls __libc_start_main which is
93clearly a standard library initialization function which then calls
94main.
95
96You can go take a look at _start __libc_start_main in glibc source
97if you want, but it's not very interesting for us as it sets up a
98bunch of stuff for dynamic linking and such that we will never use
99since we want a static executable.
100
101Let's recompile our hello world with optimization (-O2), without
102debug information and with stripping (-s) to see how large it is:
103-------------------------------------------------------------------
104$ gcc -s -O2 hello.c
105$ wc -c a.out
1066208 a.out
107-------------------------------------------------------------------
1086kb for a simple hello world? that's a lot!
109
110Even if I add other size optimization flags such as
111-Wl,--gc-sections -fno-unwind-tables
112-fno-asynchronous-unwind-tables -Os it just won't go below 6kb.
113
114We will now progressively strip this program down by first getting
115rid of the standard library and then learning how to invoke
116syscalls without having to include any headers.
117
118So how do we get rid of the standard library? If we try to compile
119our current code with -nostdlib we will run into linker errors:
120-------------------------------------------------------------------
121$ gcc -s -O2 -nostdlib hello.c
122/usr/lib/gcc/x86_64-pc-linux-gnu/4.9.3/../../../../x86_64-pc-linux-
123gnu/bin/ld: warning: cannot find entry symbol _start; defaulting to
1240000000000400120
125/tmp/ccTn8ClC.o: In function `main':
126hello.c:(.text.startup+0xa): undefined reference to `puts'
127collect2: error: ld returned 1 exit status
128-------------------------------------------------------------------
129
130The linker is complaining about _start missing, which is what we
131would expect from our previous debugging.
132
133We also have a linker error on puts, which is to be expected since
134it's a libc function. But how do we print "hello" without puts?
135
136The linux kernel exposes a bunch of syscalls, which are functions
137that user-space programs can enter to interact with the OS.
138You can see a list of syscalls by running "man syscalls" or
139visiting this site:
140http://man7.org/linux/man-pages/man2/syscalls.2.html
141
142How do we find out which syscall puts uses? We can either look
143through the syscall list, or simply install strace to trace
144syscalls and write a simple program that uses puts.
145
146The strace method is extremely useful. If you don't know how to
147do something with syscalls, do it with libc and then strace it to
148see which syscalls it uses on the target architecture.
149-------------------------------------------------------------------
150$ cat > puts.c << "EOF"
151#include <stdio.h>
152
153int main(int argc, char* argv[])
154{
155 puts("hello");
156
157 return 0;
158}
159EOF
160
161$ gcc puts.c
162$ strace ./a.out > /dev/null
163- stuff we don't care about -
164write(1, "hello\n", 6) = 6
165exit_group(0) = ?
166+++ exited with 0 +++
167-------------------------------------------------------------------
168
169So it's using the write syscall.
170
171Note how I pipe stdout to /dev/null in strace? That's because
172strace output is in stderr and we don't want to have it mixed with
173a.out's output.
174
175Let's check the manpage for write:
176-------------------------------------------------------------------
177$ man 2 write
178SYNOPSIS
179 #include <unistd.h>
180
181 ssize_t write(int fd, const void *buf, size_t count);
182
183DESCRIPTION
184 write() writes up to count bytes from the buffer pointed
185 buf to the file referred to by the file descriptor fd.
186-------------------------------------------------------------------
187
188In linux, there are 3 standard file descriptors:
189- stdin: used to pipe data into the program or to read user input.
190- stdout: output
191- stderr: alternate output for error messages
192
193If we read "man stdout", we will see that they are simply defined
194as 0, 1 and 2.
195
196So all we have to do is replace our puts with a write to stream 1
197(stdout).
198-------------------------------------------------------------------
199#include <unistd.h>
200
201int main(int argc, char* argv[])
202{
203 write(1, "hello\n", 6);
204
205 return 0;
206}
207-------------------------------------------------------------------
208
209Let's try to compile it again:
210-------------------------------------------------------------------
211$ gcc -s -O2 -nostdlib hello.c
212hello.c: In function ?main?:
213hello.c:6:5: warning: ignoring return value of ?write?, declared
214with attribute warn_unused_result [-Wunused-result]
215 write(1, "hello\n", 6);
216 ^
217/usr/lib/gcc/x86_64-pc-linux-gnu/4.9.3/../../../../x86_64-pc-linux-
218gnu/bin/ld: warning: cannot find entry symbol _start; defaulting to
2190000000000400120
220/tmp/ccJXwSsr.o: In function `main':
221hello.c:(.text.startup+0x14): undefined reference to `write'
222collect2: error: ld returned 1 exit status
223-------------------------------------------------------------------
224
225Oh no! The "write" function is part of the standard library!
226How do we invoke syscalls without having to link the standard lib?
227
228Let's take a look at section "A.2.1 Calling Conventions" of the
229AMD64 ABI specification. If you are on i386 (32-bit), just follow
230along, we will port this to i386 soon in a moment.
231
232If you're completely clueless about asm, you should still be
233able to understand once you see the example. I'm not that good
234at asm myself.
235
236https://software.intel.com/sites/default/files/article/402129/
237mpx-linux64-abi.pdf
238
239-------------------------------------------------------------------
2401. User-level applications use as integer registers for passing the
241sequence %rdi, %rsi, %rdx, %rcx, %r8 and %r9. The kernel interface
242uses %rdi, %rsi, %rdx, %r10, %r8 and %r9.
243
2442. A system-call is done via the syscall instruction. The kernel
245destroys registers %rcx and %r11.
246
2473. The number of the syscall has to be passed in register %rax.
248
2494. System-calls are limited to six arguments, no argument is passed
250directly on the stack.
251
2525. Returning from the syscall, register %rax contains the result of
253the system-call. A value in the range between -4095 and -1
254indicates an error, it is -errno.
255
2566. Only values of class INTEGER or class MEMORY are passed to the
257kernel.
258-------------------------------------------------------------------
259
260In poor words, all we need to do is write an asm wrapper that:
261
262- takes the syscall number followed by either pointers or integers
263 as parameters
264- sets rax to the syscall number
265- sets rdi, rsi, rdx, r10, r8 and r9 to the parameters. calls that
266 take less than 6 parameters will ignore the excess ones.
267- executes "syscall"
268- returns the contents of rax
269
270Now if we read section 3.4 of the specification or the quick
271cheatsheet at http://wiki.osdev.org/Calling_Conventions , we will
272see that on AMD64 the registers used to pass parameters to regular
273functions are almost the same as the syscalls, except for r10 which
274is replaced with rcx. The return register is also the same (rax).
275
276This means that our syscall wrapper will only be able to accept and
277forward a maximum of 5 parameters (because the first parameter is
278already being used to pass the syscall number).
279
280We could use the stack to take more than 6 parameters, but let's
281not make our lives more complicated when we don't even need to call
282syscalls with 6 parameters yet.
283
284The abi also states that:
285-------------------------------------------------------------------
286Registers %rbp, %rbx and %r12 through %r15 ?belong? to the calling
287function and the called function is required to preserve their
288values. In other words, a called function must preserve these
289registers? values for its caller. Remaining registers ?belong? to
290the called function. If a calling function wants to preserve such a
291register value across a function call, it must save the value in
292its local stack frame.
293-------------------------------------------------------------------
294
295Which means that we don't have to worry about saving and restoring
296the values of rdi, rsi, rdx, r10, r8 and r9 inside of our syscall
297wrapper, because it's up to the caller to save them and gcc will
298take care of that (since we will be calling it from C code).
299
300Putting it all together, this will be our syscall wrapper (in intel
301syntax):
302-------------------------------------------------------------------
303mov rax,rdi /* rax (syscall number) = func param 1 (rdi) */
304mov rdi,rsi /* rdi (syscall param 1) = func param 2 (rsi) */
305mov rsi,rdx /* rsi (syscall param 2) = func param 3 (rdx) */
306mov rdx,rcx /* rdx (syscall param 3) = func param 4 (rcx) */
307mov r10,r8 /* r10 (syscall param 4) = func param 5 (r8) */
308mov r8,r9 /* r8 (syscall param 5) = func param 6 (r9) */
309syscall /* enter the syscall (return value will be in rax */
310ret /* return value is already in rax, we can return */
311-------------------------------------------------------------------
312
313How do we embed arbitrary asm into our program though? One way is
314gcc inline assembler, but I personally find the syntax ugly.
315
316We're going to write a .S file in GAS (GNU Assembler) syntax and
317let gcc compile and link it with your hello.c .
318-------------------------------------------------------------------
319cat > hello.S << "EOF"
320/* enable intel asm syntax without the % prefix for registers */
321.intel_syntax noprefix
322
323/* this marks the .text section of a PE executable, which contains
324 program code */
325.text
326 /* exports syscall5 to other compilation units (files) */
327 .globl syscall5
328
329 syscall5:
330 mov rax,rdi
331 mov rdi,rsi
332 mov rsi,rdx
333 mov rdx,rcx
334 mov r10,r8
335 mov r8,r9
336 syscall
337 ret
338EOF
339-------------------------------------------------------------------
340
341You can find syscalls numbers here:
342http://betteros.org/ref/syscall.php
343https://filippo.io/linux-syscall-table/
344
345Or by simply letting the C preprocessor print it for you:
346-------------------------------------------------------------------
347$ printf "#include <sys/syscall.h>\nblah SYS_write" | \
348 gcc -E - | grep blah
349blah 1
350-------------------------------------------------------------------
351-E runs the preprocessor on the file, expanding all macros and
352therefore replacing #define consts with their value, while - means
353that we use stdin as input (which we pipe in from printf).
354Then we just mark a line with blah so we can grep it, followed by
355the constant we want to know.
356Syscall numbers are usually named SYS_ followed by the syscall name
357You can also add the -m32 flags to check values for 32-bit (i386).
358
359Remember the prototype for write from earlier?
360-------------------------------------------------------------------
361ssize_t write(int fd, const void *buf, size_t count);
362-------------------------------------------------------------------
363
364ssize_t and size_t are types defined by unistd. A quick inspection
365reveals that they are 64-bit integers and that the extra s in
366ssize means signed:
367-------------------------------------------------------------------
368$ printf "#include <unistd.h>" | gcc -E - | grep size_t
369typedef long int __blksize_t;
370typedef long int __ssize_t;
371typedef __ssize_t ssize_t;
372typedef long unsigned int size_t;
373-------------------------------------------------------------------
374
375If we try -m32 we will also see that this will be a 32-bit integer
376on 32-bit, which means that it's the same size as the
377architecture's pointers. I like to call this kind of integer
378intptr.
379
380Now we can import syscall5 in hello.c and make a write function
381that calls it:
382-------------------------------------------------------------------
383void* syscall5(
384 void* number,
385 void* arg1,
386 void* arg2,
387 void* arg3,
388 void* arg4,
389 void* arg5
390);
391
392typedef unsigned long int uintptr; /* size_t */
393typedef long int intptr; /* ssize_t */
394
395static
396intptr write(int fd, void const* data, uintptr nbytes)
397{
398 return (intptr)
399 syscall5(
400 (void*)1, /* SYS_write */
401 (void*)(intptr)fd,
402 (void*)data,
403 (void*)nbytes,
404 0, /* ignored */
405 0 /* ignored */
406 );
407}
408
409int main(int argc, char* argv[])
410{
411 write(1, "hello\n", 6);
412
413 return 0;
414}
415-------------------------------------------------------------------
416
417See that (void*)(intptr) double cast on fd? If fd is 32-bit and
418void* is 64-bit, we would get a warning that we are implicitly
419casting it to a different size, so we need to explicitly specify
420that we want that conversion by adding the intptr cast.
421
422This should be done every time you cast to and from pointers when
423the destination type is not guaranteed to be the same size as
424pointers. Especially when targeting multiple architectures.
425
426Also note how we cast the const qualifier away from data to avoid
427a warning.
428
429If we compile now, we are finally only missing _start!
430-------------------------------------------------------------------
431$ gcc -s -O2 -nostdlib hello.S hello.c
432/usr/lib/gcc/x86_64-pc-linux-gnu/4.9.3/../../../../
433x86_64-pc-linux-gnu/bin/ld: warning: cannot find entry symbol
434_start; defaulting to 0000000000400120
435-------------------------------------------------------------------
436
437So, how do we define _start? Where do we get argc and argv from?
438We need to know the initial state of registers and the stack.
439
440Back to the AMD64 ABI document. In figure 3.9, we can see the
441initial state of the stack:
442
4430 to rsp: undefined
444rsp : argc <- top of the stack (last pushed value)
445rsp+8 : argv[0]
446rsp+16 : argv[1]
447rsp+24 : argv[2]
448... : ...
449rsp+8*argc : argv[argc - 1]
450rsp+8+8*argc : 0
451* more stuff we don't care about *
452
453And right below it we have the initial state of the registers:
454-------------------------------------------------------------------
455%rbp: The content of this register is unspecified at process
456 initialization time, but the user code should mark the
457 deepest stack frame by setting the frame pointer to zero.
458
459%rsp: The stack pointer holds the address of the byte with lowest
460 address which is part of the stack. It is guaranteed to be
461 16-byte aligned at process entry.
462
463%rdx: a function pointer that the application should register with
464 atexit (BA_OS).
465-------------------------------------------------------------------
466
467So we know that rbp must be zeroed and that rsp points to the top
468of the stack. We don't care about rdx.
469
470If you don't understand how the stack works, it's basically a
471chunk of memory where data is appended (pushed) or retrieved (pop)
472at the end.
473
474In AMD64's convention we're actually prepending and removing data
475at the beginning of the block of memory since the stack is said to
476"grow downwards", which means that when we push something on the
477stack, the stack pointer gets lower.
478
479Since the ABI states that the stack pointer is 16-byte aligned, we
480must remember always push data whose size is a multiple of 16. For
481example, 2 64-bit integers are 16 bytes. It's often necessary to
482either push useless data or simply align the stack pointer when
483the pushed values don't happen to be aligned.
484
485Putting it all together, our _start function needs to:
486- zero rbp
487- put argc into rdi (1st parameter for main)
488- put the stack address of argv[0] into rsi (2nd param for main),
489 which will be interpreted as an array of char pointers.
490- align stack to 16-bytes
491- call main
492
493Here's our new hello.S:
494-------------------------------------------------------------------
495.intel_syntax noprefix
496.text
497 .globl _start, syscall5
498
499 _start:
500 xor rbp,rbp /* xoring a value with itself = 0 */
501 pop rdi /* rdi = argc */
502 /* the pop instruction already added 8 to rsp */
503 mov rsi,rsp /* rest of the stack as an array of char ptr */
504
505 /* zero the las 4 bits of rsp, aligning it to 16 bytes
506 same as "and rsp,0xfffffffffffffff0" because negative
507 numbers are represented as
508 max_unsigned_value + 1 - abs(negative_num) */
509 and rsp,-16
510 call main
511 ret
512
513 syscall5:
514 mov rax,rdi
515 mov rdi,rsi
516 mov rsi,rdx
517 mov rdx,rcx
518 mov r10,r8
519 mov r8,r9
520 syscall
521 ret
522-------------------------------------------------------------------
523
524It finally compiles! It runs correctly, but we get a segmentation
525fault when we exit:
526-------------------------------------------------------------------
527$ gcc -s -O2 -nostdlib hello.S hello.c
528$ ./a.out
529hello
530Segmentation fault
531-------------------------------------------------------------------
532
533But why?
534
535When we execute a call instruction, the return address (address of
536the intruction to jump to after the function returns) is pushed
537onto the stack implicitly and the ret instruction implicitly pops
538it and jumps to it.
539
540The _start function is very special, as it has no return address,
541so our ret instruction in _start is trying to jump back to an
542invalid memory location, executing garbage data as code or
543triggering access violations.
544
545We need to tell the OS to kill our process and never reach the ret
546in _start. The syscall _EXIT(2) is just what we need:
547-------------------------------------------------------------------
548$ man 2 _EXIT
549NAME
550 _exit, _Exit - terminate the calling process
551
552SYNOPSIS
553 #include <unistd.h>
554
555 void _exit(int status);
556
557 #include <stdlib.h>
558
559 void _Exit(int status);
560
561$ printf "#include <sys/syscall.h>\nblah SYS_exit" | \
562 gcc -E - | grep blah
563blah 60
564-------------------------------------------------------------------
565
566The status code will simply be the return value of main, which is
567stored in rax as we know.
568
569New hello.S:
570-------------------------------------------------------------------
571.intel_syntax noprefix
572.text
573 .globl _start, syscall5
574
575 _start:
576 xor rbp,rbp
577 pop rdi
578 mov rsi,rsp
579 and rsp,-16
580 call main
581
582 mov rdi,rax /* syscall param 1 = rax (ret value of main) */
583 mov rax,60 /* SYS_exit */
584 syscall
585
586 ret /* should never be reached, but if the OS somehow fails
587 to kill us, it will cause a segmentation fault */
588
589 syscall5:
590 mov rax,rdi
591 mov rdi,rsi
592 mov rsi,rdx
593 mov rdx,rcx
594 mov r10,r8
595 mov r8,r9
596 syscall
597 ret
598-------------------------------------------------------------------
599
600Our program finally runs and terminates correctly! Let's give
601ourselves a good pat on the back.
602-------------------------------------------------------------------
603$ gcc -s -O2 -nostdlib hello.S hello.c
604$ ./a.out
605hello
606-------------------------------------------------------------------
607
608Let's check the executable size now:
609-------------------------------------------------------------------
610$ wc -c a.out
6111008 a.out
612-------------------------------------------------------------------
613
614We're almost below 1kb and it's 6 times smaller than before, but we
615can shrink it further.
616
617First of all, gcc generates unwind tables by default, which are
618used for exception handling and other stuff we don't care about.
619
620Let's turn those off:
621-------------------------------------------------------------------
622$ gcc -s -O2 \
623 -nostdlib \
624 -fno-unwind-tables \
625 -fno-asynchronous-unwind-tables \
626 hello.S hello.c
627
628$ wc -c a.out
629736 a.out
630-------------------------------------------------------------------
631
632Woah, we shaved almost 300 bytes off!
633
634As a last step, we can check the executable for useless sections:
635-------------------------------------------------------------------
636$ objdump -x a.out
637
638a.out: file format elf64-x86-64
639a.out
640architecture: i386:x86-64, flags 0x00000102:
641EXEC_P, D_PAGED
642start address 0x000000000040011a
643
644Program Header:
645 LOAD off 0x0000000000000000 vaddr 0x0000000000400000
646 paddr 0x0000000000400000 align 2**21
647 filesz 0x0000000000000153 memsz 0x0000000000000153
648 flags r-x
649 STACK off 0x0000000000000000 vaddr 0x0000000000000000
650 paddr 0x0000000000000000 align 2**4
651 filesz 0x0000000000000000 memsz 0x0000000000000000
652 flags rwx
653PAX_FLAGS off 0x0000000000000000 vaddr 0x0000000000000000
654 paddr 0x0000000000000000 align 2**3
655 filesz 0x0000000000000000 memsz 0x0000000000000000
656 flags --- 2800
657
658Sections:
659Idx Name Size VMA LMA ...
660 0 .text 0000005c 00000000004000f0 00000000004000f0 ...
661 CONTENTS, ALLOC, LOAD, READONLY, CODE
662 1 .rodata 00000007 000000000040014c 000000000040014c ...
663 CONTENTS, ALLOC, LOAD, READONLY, DATA
664 2 .comment 0000002a 0000000000000000 0000000000000000 ...
665 CONTENTS, READONLY
666SYMBOL TABLE:
667no symbols
668-------------------------------------------------------------------
669
670.text is the code
671.rodata is Read Only data (such as the string "hello" in our case)
672
673So we need both of these.
674
675But what's that .comment section?
676-------------------------------------------------------------------
677$ objdump -s -j .comment a.out
678
679a.out: file format elf64-x86-64
680
681Contents of section .comment:
682 0000 4743433a 20284765 6e746f6f 20342e39 GCC: (Gentoo 4.9
683 0010 2e332070 312e352c 20706965 2d302e36 .3 p1.5, pie-0.6
684 0020 2e342920 342e392e 3300 .4) 4.9.3.
685-------------------------------------------------------------------
686
687Just information about the compiler, it seems. That's 1 byte for
688every character of that string, let's get rid of it!
689-------------------------------------------------------------------
690$ strip -R .comment a.out
691$ wc -c a.out
692624 a.out
693-------------------------------------------------------------------
694
695There we go, we have achieved a nearly ten-fold size improvement
696on our little hello world.
697
698Let's set up a build script with all those compiler flags and let's
699also make it output the executable with a proper name.
700
701Also, I'm going to add the following useful flags:
702
703-Wl,--gc-sections: get rid of any unused code sections
704
705-fdata-sections: separate each function into its own code section.
706 this lets gc-sections do its job. these two
707 options combined will get rid of any dead code you
708 might accidentally leave in your program. it also
709 gets rid of unused functions in statically linked
710 libraries.
711
712-fno-stack-protector: doesn't generate extra code to guard against
713 overflows overwriting the return address.
714
715-Wa,--noexecstack: mark the stack memory as non-executable. this is
716 just extra security since we don't need to be
717 executing code off the stack's memory.
718
719-fno-builtin: disable all builtin gcc functions (such as math
720 routines and other stuff). we will implement them
721 ourselves as needed.
722
723-std=c89 -pedantic: follow the old c89 standard strictly. this
724 should force us to write code more compatible
725 with old compilers.
726
727-Wall: enable all warnings.
728-Werror: treat all warnings as error. can't let our code build with
729 unchecked warnings.
730
731-------------------------------------------------------------------
732$ cat > build.sh << "EOF"
733#!/bin/sh
734
735exename="hello"
736
737gcc -std=c89 -pedantic -s -O2 -Wall -Werror \
738 -nostdlib \
739 -fno-unwind-tables \
740 -fno-asynchronous-unwind-tables \
741 -fdata-sections \
742 -Wl,--gc-sections \
743 -Wa,--noexecstack \
744 -fno-builtin \
745 -fno-stack-protector \
746 hello.S hello.c \
747 -o $exename \
748\
749&& strip -R .comment $exename
750EOF
751
752$ chmod +x ./build.sh
753$ ./build.sh
754$ wc -c hello
755624 hello
756$ ./hello
757hello
758-------------------------------------------------------------------
759
760As you might have noticed, we are doing a lot of useless mov's in
761that syscall5 wrapper on syscalls that take less than 5 parameters.
762
763Let's make one wrapper for each parameter count. This will increase
764performance slightly at the cost of a slightly bigger executable.
765
766You are free to remove the ones you don't use once you finish
767prototyping your program.
768
769New hello.S
770-------------------------------------------------------------------
771.intel_syntax noprefix
772.text
773 .globl _start, syscall,
774 .globl syscall1, syscall2, syscall3, syscall4, syscall5
775
776 _start:
777 xor rbp,rbp
778 pop rdi
779 mov rsi,rsp
780 and rsp,-16
781 call main
782 mov rdi,rax
783 mov rax,60 /* SYS_exit */
784 syscall
785 ret
786
787 syscall:
788 mov rax,rdi
789 syscall
790 ret
791
792 syscall1:
793 mov rax,rdi
794 mov rdi,rsi
795 syscall
796 ret
797
798 syscall2:
799 mov rax,rdi
800 mov rdi,rsi
801 mov rsi,rdx
802 syscall
803 ret
804
805 syscall3:
806 mov rax,rdi
807 mov rdi,rsi
808 mov rsi,rdx
809 mov rdx,rcx
810 syscall
811 ret
812
813 syscall4:
814 mov rax,rdi
815 mov rdi,rsi
816 mov rsi,rdx
817 mov rdx,rcx
818 mov r10,r8
819 syscall
820 ret
821
822 syscall5:
823 mov rax,rdi
824 mov rdi,rsi
825 mov rsi,rdx
826 mov rdx,rcx
827 mov r10,r8
828 mov r8,r9
829 syscall
830 ret
831-------------------------------------------------------------------
832
833Now we can change our write function to use syscall3 instead.
834
835We will also change argv in our main to be char const* since we
836probably won't be modifying it. This is normally not allowed on the
837standard C library, but we aren't using it :^).
838
839Using the syscall numbers directly is a bit hard to read so let's
840also make a header with all the syscall numbers we use:
841-------------------------------------------------------------------
842$ cat > syscalls.h << "EOF"
843#define SYS_write 1
844#define SYS_exit 60
845EOF
846-------------------------------------------------------------------
847
848We will also define the syscall number as uintptr so that we don't
849need to cast to void*.
850
851new hello.c
852-------------------------------------------------------------------
853#include "syscalls.h"
854
855typedef unsigned long int uintptr;
856typedef long int intptr;
857
858void* syscall3(
859 uintptr number,
860 void* arg1,
861 void* arg2,
862 void* arg3
863);
864
865static
866intptr write(int fd, void const* data, uintptr nbytes)
867{
868 return (uintptr)
869 syscall3(
870 SYS_write,
871 (void*)(intptr)fd,
872 (void*)data,
873 (void*)nbytes
874 );
875}
876
877int main(int argc, char const* argv[])
878{
879 write(1, "hello\n", 6);
880
881 return 0;
882}
883-------------------------------------------------------------------
884
885We can include headers in .S files, so let's also include it in
886hello.S
887-------------------------------------------------------------------
888#include "syscalls.h"
889
890.intel_syntax noprefix
891.text
892 .globl _start, syscall,
893 .globl syscall1, syscall2, syscall3, syscall4, syscall5
894
895 _start:
896 xor rbp,rbp
897 pop rdi
898 mov rsi,rsp
899 and rsp,-16
900 call main
901 mov rdi,rax
902 mov rax,SYS_exit
903 syscall
904 ret
905
906...
907-------------------------------------------------------------------
908
909Having to pass the string length every time is annoying, so let's
910implement our own strlen and puts.
911
912I'm also going to make a "internal" alias for static, which makes
913it easier to search for static functions, rather than static
914variables, in a large codebase. I got this idea from Casey Muratori
915from handmade hero.
916
917-------------------------------------------------------------------
918#include "syscalls.h"
919
920typedef unsigned long int uintptr;
921typedef long int intptr;
922
923#define internal static
924
925void* syscall3(
926 uintptr number,
927 void* arg1,
928 void* arg2,
929 void* arg3
930);
931
932/* ------------------------------------------------------------- */
933
934#define stdout 1
935
936internal
937intptr write(int fd, void const* data, uintptr nbytes)
938{
939 return (uintptr)
940 syscall3(
941 SYS_write,
942 (void*)(intptr)fd,
943 (void*)data,
944 (void*)nbytes
945 );
946}
947
948/* ------------------------------------------------------------- */
949
950internal
951uintptr strlen(char const* str)
952{
953 char const* p;
954 for (p = str; *p; ++p);
955 return p - str;
956}
957
958internal
959uintptr puts(char const* str) {
960 return write(stdout, str, strlen(str));
961}
962
963/* ------------------------------------------------------------- */
964
965int main(int argc, char const* argv[])
966{
967 puts("hello\n");
968
969 return 0;
970}
971-------------------------------------------------------------------
972
973If you don't understand my strlen function, it's pretty simple: C
974strings are null-terminated (the byte after the last character is
975zero), so I just iterate the characters through a pointer until
976I find a zero byte, and then I subtract the current position from
977the beginning of the string.
978
979libc does all kinds of crazy tricks to optimize this for large
980strings, which I haven't looked into.
981
982As you can see, I've also separated the code into sections with
983those spacer comments for readability. I grouped all the syscall
984wrappers together, followed by utility functions, followed by
985the program's code.
986
987Now we have a nice framework for AMD64 programs, but we're not
988going to stop here. We're going to set this up to also cross
989compile for i386, which is a very common architecture in low-end
990servers (such as the one I host my gopher mirror on).
991
992###################################################################
993 Porting to i386
994###################################################################
995Let's move all the AMD64-specific code into a dedicated folder.
996-------------------------------------------------------------------
997$ mkdir amd64
998$ mv hello.S amd64/start.S
999$ mv syscalls.h amd64/
1000-------------------------------------------------------------------
1001
1002Now we can make a architecture-specific main.c where we define the
1003integer types and main, which just calls hello_run, or whatever you
1004want to name your program's entry point. This file includes hello.c
1005just before main.
1006
1007I also make it define AMD64 in case we need to do platform checking
1008in the code. Platform specific code should be kept separated
1009whenever possible, though.
1010
1011-------------------------------------------------------------------
1012$ cat > amd64/main.c << "EOF"
1013#define AMD64
1014#include "syscalls.h"
1015
1016typedef unsigned long int u64;
1017typedef unsigned int u32;
1018typedef unsigned short int u16;
1019typedef unsigned char u8;
1020
1021typedef long int i64;
1022typedef int i32;
1023typedef short int i16;
1024typedef signed char i8;
1025
1026typedef i64 intptr;
1027typedef u64 uintptr;
1028
1029#include "../hello.c"
1030
1031int main(int argc, char const* argv[]) {
1032 return hello_run(argc, argv);
1033}
1034EOF
1035-------------------------------------------------------------------
1036
1037Yes, you can include .c files, which just get pasted into the file.
1038This results in a single compilation unit even though we have
1039multiple files, which speeds up compilation (unless your project is
1040massive) and saves us the pain of typing every filename in our
1041build script. This is yet another tick I got from Casey.
1042
1043By the way, you can check integer types on any architecture with
1044the usual gcc preprocessor trick:
1045-------------------------------------------------------------------
1046$ printf "#include <stdint.h>" | gcc -E - | grep int64
1047typedef long int int64_t;
1048typedef unsigned long int uint64_t;
1049
1050$ printf "#include <stdint.h>" | gcc -E - | grep int32
1051typedef int int32_t;
1052typedef unsigned int uint32_t;
1053
1054$ printf "#include <stdint.h>" | gcc -E - | grep int16
1055typedef short int int16_t;
1056typedef unsigned short int uint16_t;
1057
1058$ printf "#include <stdint.h>" | gcc -E - | grep int8
1059typedef signed char int8_t;
1060typedef unsigned char uint8_t;
1061-------------------------------------------------------------------
1062
1063And for the size of pointers, you can write a simple program that
1064printfs sizeof(void*).
1065
1066hello.c will now look like this (remember, we moved the integer
1067definitions to main.c and renamed main to hello_run, and
1068syscalls.h is already included in main.c):
1069
1070-------------------------------------------------------------------
1071#define internal static
1072
1073void* syscall3(
1074 uintptr number,
1075 void* arg1,
1076 void* arg2,
1077 void* arg3
1078);
1079
1080/* ------------------------------------------------------------- */
1081
1082#define stdout 1
1083
1084internal
1085intptr write(int fd, void const* data, uintptr nbytes)
1086{
1087 return (uintptr)
1088 syscall3(
1089 SYS_write,
1090 (void*)(intptr)fd,
1091 (void*)data,
1092 (void*)nbytes
1093 );
1094}
1095
1096/* ------------------------------------------------------------- */
1097
1098internal
1099uintptr strlen(char const* str)
1100{
1101 char const* p;
1102 for (p = str; *p; ++p);
1103 return p - str;
1104}
1105
1106internal
1107uintptr puts(char const* str) {
1108 return write(stdout, str, strlen(str));
1109}
1110
1111/* ------------------------------------------------------------- */
1112
1113internal
1114int hello_run(int argc, char const* argv[])
1115{
1116 puts("hello\n");
1117
1118 return 0;
1119}
1120-------------------------------------------------------------------
1121
1122Modify the build script to follow the new structure:
1123-------------------------------------------------------------------
1124#!/bin/sh
1125
1126exename="hello"
1127
1128gcc -std=c89 -pedantic -s -O2 -Wall -Werror \
1129 -nostdlib \
1130 -fno-unwind-tables \
1131 -fno-asynchronous-unwind-tables \
1132 -fdata-sections \
1133 -Wl,--gc-sections \
1134 -Wa,--noexecstack \
1135 -fno-builtin \
1136 -fno-stack-protector \
1137 amd64/start.S amd64/main.c \
1138 -o $exename \
1139\
1140&& strip -R .comment $exename
1141-------------------------------------------------------------------
1142
1143Now we can create the main.c for i386:
1144-------------------------------------------------------------------
1145$ mkdir i386
1146$ cat > i386/main.c << "EOF"
1147#define I386
1148#include "syscalls.h"
1149
1150typedef unsigned long long int u64;
1151typedef unsigned int u32;
1152typedef unsigned short int u16;
1153typedef unsigned char u8;
1154
1155typedef long long int i64;
1156typedef int i32;
1157typedef short int i16;
1158typedef signed char i8;
1159
1160typedef i32 intptr;
1161typedef u32 uintptr;
1162
1163#include "../hello.c"
1164
1165int main(int argc, char const* argv[]) {
1166 return hello_run(argc, argv);
1167}
1168EOF
1169-------------------------------------------------------------------
1170
1171Note how intptr is defined as a 32-bit integer and u64 is long
1172long on 32-bits.
1173
1174Let's now grab syscall numbers for i386 and throw them into
1175syscalls.h:
1176-------------------------------------------------------------------
1177$ printf "#include <sys/syscall.h>\nblah SYS_write" \
1178 | gcc -m32 -E - | grep blah
1179blah 4
1180
1181$ printf "#include <sys/syscall.h>\nblah SYS_exit" \
1182 | gcc -m32 -E - | grep blah
1183blah 1
1184
1185$ cat > i386/syscalls.h << "EOF"
1186#define SYS_write 4
1187#define SYS_exit 1
1188EOF
1189-------------------------------------------------------------------
1190
1191We need to write a i386 start.S and you guessed it, it's time to
1192look at the ABI specification once again!
1193
1194http://www.sco.com/developers/devspecs/abi386-4.pdf
1195
1196This time I will just summarize the differences from amd64:
1197
1198- Registers are 32-bit so we push 4 bytes at a time.
1199
1200- The stack is aligned to 4 bytes, but we will still align it to
1201 16 bytes because it can improve performance by preventing
1202 misaligned SSE accesses (according to glibc).
1203
1204- ebp needs to be zeroed (32-bit version of rbp)
1205
1206- esp is the stack pointer (32-bit version of rsp)
1207
1208- Return values for functions and syscalls are in eax
1209
1210- The instruction to enter syscalls is "int 0x80"
1211
1212- Syscall parameters are passed in ebx, ecx, edx, esi, edi, ebp
1213
1214- Function parameters are passed entirely through the stack by
1215 pushing them in reverse order, which means that we will be able
1216 to access them sequentially every 4 bytes on the stack.
1217 VERY IMPORTANT DIFFERENCE. We won't be using registers to pass
1218 parameters to main anymore nor to pull parameters in syscall
1219 wrappers.
1220
1221- Functions are expected to preserve ebx, esi, edi, ebp, esp on
1222 their own VERY IMPORTANT! we will have to save and restore these
1223 registers manually in our syscall wrappers!
1224
1225- Function callers are expected to clean up the parameters off the
1226 stack after the call. VERY IMPORTANT
1227
1228- As explained earlier, the return address is implicitly pushed on
1229 the stack so the function parameters will start at esp+4.
1230
1231In short, our _start will look something like this:
1232-------------------------------------------------------------------
1233xor ebp,ebp
1234
1235pop esi /* argc */
1236mov ecx,esp /* argv */
1237
1238/* 16-byte stack alignment is not mandatory here but
1239 according to glibc it improves SSE performance */
1240and esp,-16
1241
1242/* push garbage to align to 16 bytes */
1243push 0xb16b00b5
1244push 0xb16b00b5
1245push ecx /* argv */
1246push esi /* argc */
1247call main
1248add esp,16
1249/* on i386 it's up to the caller to clean up the stack. we can
1250 either pop them into scratch registers or just add the total
1251 size of the parameters in bytes to the stack pointer */
1252
1253mov ebx,eax
1254mov eax,SYS_exit
1255int 0x80
1256ret
1257-------------------------------------------------------------------
1258
1259... and our syscall5 wrapper will look like this:
1260-------------------------------------------------------------------
1261push ebx
1262push esi
1263push edi
1264mov eax,[esp+4+12]
1265mov ebx,[esp+8+12]
1266mov ecx,[esp+12+12]
1267mov edx,[esp+16+12]
1268mov esi,[esp+20+12]
1269mov edi,[esp+24+12]
1270int 0x80
1271pop edi
1272pop esi
1273pop ebx
1274ret
1275-------------------------------------------------------------------
1276
1277See how I'm pushing registers on the stack to preserve them to then
1278pop them (in reverse order since it's LIFO)? That's very important
1279on i386.
1280
1281Also, you might be wondering what's going on with the esp offsets.
1282You have to keep in mind that every time I push a register on the
1283stack, esp is decremented by 4, so I have to skip the registers I
1284pushed on the stack (3 registers = 12 bytes) to get to the
1285parameters. Don't forget that the return address is also on the
1286stack, so parameters start at + 4.
1287
1288And here's our complete i386 start.S
1289-------------------------------------------------------------------
1290$ cat > i386/start.S << "EOF"
1291#include "syscalls.h"
1292
1293.intel_syntax noprefix
1294.text
1295 .globl _start, syscall
1296 .globl syscall1, syscall2, syscall3, syscall4, syscall5
1297
1298 _start:
1299 xor ebp,ebp
1300 pop esi
1301 mov ecx,esp
1302 and esp,-16
1303 push 0xb1gb00b5
1304 push 0xb1gb00b5
1305 push ecx
1306 push esi
1307 call main
1308 add esp,16
1309 mov ebx,eax
1310 mov eax,SYS_exit
1311 int 0x80
1312 ret
1313
1314 syscall:
1315 mov eax,[esp+4]
1316 int 0x80
1317 ret
1318
1319 syscall1:
1320 push ebx
1321 mov eax,[esp+4+4]
1322 mov ebx,[esp+8+4]
1323 int 0x80
1324 pop ebx
1325 ret
1326
1327 syscall2:
1328 push ebx
1329 mov eax,[esp+4+4]
1330 mov ebx,[esp+8+4]
1331 mov ecx,[esp+12+4]
1332 int 0x80
1333 pop ebx
1334 ret
1335
1336 syscall3:
1337 push ebx
1338 mov eax,[esp+4+4]
1339 mov ebx,[esp+8+4]
1340 mov ecx,[esp+12+4]
1341 mov edx,[esp+16+4]
1342 int 0x80
1343 pop ebx
1344 ret
1345
1346 syscall4:
1347 push ebx
1348 push esi
1349 mov eax,[esp+4+8]
1350 mov ebx,[esp+8+8]
1351 mov ecx,[esp+12+8]
1352 mov edx,[esp+16+8]
1353 mov esi,[esp+20+8]
1354 int 0x80
1355 pop esi
1356 pop ebx
1357 ret
1358
1359 syscall5:
1360 push ebx
1361 push esi
1362 push edi
1363 mov eax,[esp+4+12]
1364 mov ebx,[esp+8+12]
1365 mov ecx,[esp+12+12]
1366 mov edx,[esp+16+12]
1367 mov esi,[esp+20+12]
1368 mov edi,[esp+24+12]
1369 int 0x80
1370 pop edi
1371 pop esi
1372 pop ebx
1373 ret
1374EOF
1375-------------------------------------------------------------------
1376
1377Now we need to modify our build script to handle multiple
1378architectures.
1379
1380I will just make the script take the arch subfolder name as a
1381parameter.
1382
1383This is not enough though, because each architecture will have some
1384extra compiler flags. For example, on i386 we need -m32 to ensure a
138532-bit build even on amd64 dev machines, as well as -Wno-long-long
1386which suppresses a warning about 64 bit integers being a
1387nonstandard gcc extension on 32-bit.
1388
1389We will make our build script source a flags.sh script in the
1390architecture-specific folder which just exports COMPILER_FLAGS with
1391all the extra stuff it wants.
1392
1393-------------------------------------------------------------------
1394$ cat > build.sh << "EOF"
1395#!/bin/sh
1396
1397exename="hello"
1398archname=${1:-amd64} # if not specified, default to amd64
1399
1400# if flags.sh exists in the arch folder, source it
1401if [ -e $archname/flags.sh ]; then
1402 source $archname/flags.sh
1403fi
1404
1405gcc -std=c89 -pedantic -s -O2 -Wall -Werror \
1406 -nostdlib \
1407 -fno-unwind-tables \
1408 -fno-asynchronous-unwind-tables \
1409 -fdata-sections \
1410 -Wl,--gc-sections \
1411 -Wa,--noexecstack \
1412 -fno-builtin \
1413 -fno-stack-protector \
1414 $COMPILER_FLAGS \
1415 $archname/start.S $archname/main.c \
1416 -o $exename \
1417\
1418&& strip -R .comment $exename
1419EOF
1420
1421$ cat > i386/flags.sh << "EOF"
1422#!/bin/sh
1423
1424export COMPILER_FLAGS="-m32 -Wno-long-long"
1425EOF
1426-------------------------------------------------------------------
1427
1428Now we can compile both architectures easily with minimal code
1429redundancy:
1430-------------------------------------------------------------------
1431$ wc -c hello
1432720 hello
1433$ ./hello
1434hello
1435$ ./build.sh i386
1436$ wc -c hello
1437608 hello
1438$ ./hello
1439hello
1440-------------------------------------------------------------------
1441
1442And there you have it! You now have a nice framework to develop
1443libc-free programs.
1444
1445As you can see, the 32-bit executable is slightly smaller. This is
1446mostly because pointers are half as large compared to 64-bit.
1447
1448###################################################################
1449 Legacy syscalls on i386
1450###################################################################
1451There are a few things you should be extremely careful with when
1452dealing with syscalls, especially when targeting multiple
1453architectures.
1454
1455Some syscalls, such as stat, might return their stuff in a struct.
1456Be extremely careful to check the struct layout and size of the
1457types used, because it will often change drastically between
1458architectures.
1459
1460-------------------------------------------------------------------
1461$ man 2 stat
1462NAME
1463 stat, fstat, lstat, fstatat - get file status
1464
1465SYNOPSIS
1466 #include <sys/types.h>
1467 #include <sys/stat.h>
1468 #include <unistd.h>
1469
1470 int stat(const char *pathname, struct stat *buf);
1471 int fstat(int fd, struct stat *buf);
1472 int lstat(const char *pathname, struct stat *buf);
1473
1474$ printf "#include <sys/stat.h>" | gcc -E - | grep -A 1 "int stat"
1475extern int stat (const char *__restrict __file,
1476 struct stat *__restrict __buf) __attribute__ ((__nothrow__ ,
1477 __leaf__)) __attribute__ ((__nonnull__ (1, 2)));
1478-------------------------------------------------------------------
1479
1480-------------------------------------------------------------------
1481$ printf "#include <sys/stat.h>" \
1482 | gcc -E - | grep -A 60 "struct stat"
1483struct stat
1484{
1485 __dev_t st_dev;
1486 __ino_t st_ino;
1487 __nlink_t st_nlink;
1488 __mode_t st_mode;
1489 __uid_t st_uid;
1490 __gid_t st_gid;
1491 int __pad0;
1492 __dev_t st_rdev;
1493 __off_t st_size;
1494 __blksize_t st_blksize;
1495 __blkcnt_t st_blocks;
1496# 91 "/usr/include/bits/stat.h" 3 4
1497 struct timespec st_atim;
1498 struct timespec st_mtim;
1499 struct timespec st_ctim;
1500# 106 "/usr/include/bits/stat.h" 3 4
1501 __syscall_slong_t __glibc_reserved[3];
1502# 115 "/usr/include/bits/stat.h" 3 4
1503};
1504
1505$ printf "#include <sys/stat.h>" | gcc -E - \
1506 | grep '__dev_t\|__ino_t\|__nlink_t\|__mode_t\|__uid_t\|__gid_t'
1507typedef unsigned long int __dev_t;
1508typedef unsigned int __uid_t;
1509typedef unsigned int __gid_t;
1510typedef unsigned long int __ino_t;
1511typedef unsigned int __mode_t;
1512typedef unsigned long int __nlink_t;
1513
1514$ printf "#include <sys/stat.h>" | gcc -E - \
1515 | grep '__blksize_t\|__blkcnt_t\|__syscall_slong_t\|__off_t'
1516typedef long int __off_t;
1517typedef long int __blksize_t;
1518typedef long int __blkcnt_t;
1519typedef long int __syscall_slong_t;
1520
1521$ printf "#include <sys/stat.h>" | gcc -E - \
1522 | grep -A 10 "struct timespec"
1523struct timespec
1524{
1525 __time_t tv_sec;
1526 __syscall_slong_t tv_nsec;
1527};
1528
1529$ printf "#include <sys/stat.h>" | gcc -E - | grep "__time_t"
1530typedef long int __time_t;
1531-------------------------------------------------------------------
1532
1533-------------------------------------------------------------------
1534$ printf "#include <sys/stat.h>" \
1535 | gcc -m32 -E - | grep -A 60 "struct stat"
1536struct stat
1537{
1538 __dev_t st_dev;
1539 unsigned short int __pad1;
1540 __ino_t st_ino;
1541 __mode_t st_mode;
1542 __nlink_t st_nlink;
1543 __uid_t st_uid;
1544 __gid_t st_gid;
1545 __dev_t st_rdev;
1546 unsigned short int __pad2;
1547 __off_t st_size;
1548 __blksize_t st_blksize;
1549 __blkcnt_t st_blocks;
1550# 91 "/usr/include/bits/stat.h" 3 4
1551 struct timespec st_atim;
1552 struct timespec st_mtim;
1553 struct timespec st_ctim;
1554# 109 "/usr/include/bits/stat.h" 3 4
1555 unsigned long int __glibc_reserved4;
1556 unsigned long int __glibc_reserved5;
1557};
1558
1559$ printf "#include <sys/stat.h>" | gcc -m32 -E - \
1560 | grep '__dev_t\|__ino_t\|__nlink_t\|__mode_t\|__uid_t\|__gid_t'
1561__extension__ typedef __u_quad_t __dev_t;
1562__extension__ typedef unsigned int __uid_t;
1563__extension__ typedef unsigned int __gid_t;
1564__extension__ typedef unsigned long int __ino_t;
1565__extension__ typedef unsigned int __mode_t;
1566__extension__ typedef unsigned int __nlink_t;
1567
1568$ printf "#include <sys/stat.h>" \
1569 | gcc -m32 -E - | grep '__u_quad_t'
1570__extension__ typedef unsigned long long int __u_quad_t;
1571
1572$ printf "#include <sys/stat.h>" | gcc -m32 -E - \
1573 | grep '__blksize_t\|__blkcnt_t\|__syscall_slong_t'
1574__extension__ typedef long int __off_t;
1575__extension__ typedef long int __blksize_t;
1576__extension__ typedef long int __blkcnt_t;
1577__extension__ typedef long int __syscall_slong_t;
1578
1579$ printf "#include <sys/stat.h>" | gcc -m32 -E - \
1580 | grep -A 10 "struct timespec"
1581struct timespec
1582{
1583 __time_t tv_sec;
1584 __syscall_slong_t tv_nsec;
1585};
1586
1587$ printf "#include <sys/stat.h>" | gcc -m32 -E - | grep "__time_t"
1588__extension__ typedef long int __time_t;
1589-------------------------------------------------------------------
1590
1591As you can see, the stat struct is substantially different for
1592i386 and amd64 and the contained types are also different in size.
1593
1594This is not all there is to it though. Some syscalls have multiple
1595versions of them with different structs for historical reasons, and
1596gcc might wrap them in some weird way, using its own struct.
1597
1598stat is one of them. Suppose you use the above structs and assume
1599libc, stat struct is right.
1600Let's make a simple program that stats a file and dumps the stat
1601struct to stdout for us to inspect.
1602
1603These are the files:
1604-------------------------------------------------------------------
1605$ cat amd64/syscalls.h
1606#define SYS_write 1
1607#define SYS_stat 4
1608#define SYS_exit 60
1609
1610$ cat i386/syscalls.h
1611#define SYS_write 4
1612#define SYS_stat 106
1613#define SYS_exit 1
1614
1615$ cat stat.c
1616#define internal static
1617
1618void* syscall2(
1619 uintptr number,
1620 void* arg1,
1621 void* arg2
1622);
1623
1624void* syscall3(
1625 uintptr number,
1626 void* arg1,
1627 void* arg2,
1628 void* arg3
1629);
1630
1631/* ------------------------------------------------------------- */
1632
1633#define stdout 1
1634
1635internal
1636intptr write(int fd, void const* data, uintptr nbytes)
1637{
1638 return (uintptr)
1639 syscall3(
1640 SYS_write,
1641 (void*)(intptr)fd,
1642 (void*)data,
1643 (void*)nbytes
1644 );
1645}
1646
1647typedef u64 dev_t;
1648typedef intptr syscall_slong_t;
1649typedef intptr time_t;
1650
1651typedef struct
1652{
1653 time_t sec;
1654 syscall_slong_t nsec;
1655}
1656timespec;
1657
1658typedef struct
1659{
1660 dev_t dev;
1661#ifdef I386
1662 u16 __pad1;
1663#endif
1664 uintptr ino;
1665 uintptr nlink;
1666 u32 mode;
1667 u32 uid;
1668 u32 gid;
1669#ifdef AMD64
1670 int __pad0;
1671#endif
1672 dev_t rdev;
1673#ifdef I386
1674 u16 __pad2;
1675#endif
1676 intptr size;
1677 intptr blksize;
1678 intptr blocks;
1679 timespec atim;
1680 timespec mtim;
1681 timespec ctim;
1682#ifdef AMD64
1683 syscall_slong_t __glibc_reserved[3];
1684#else
1685 u32 __glibc_reserved4;
1686 u32 __glibc_reserved5;
1687#endif
1688}
1689stat_info;
1690
1691internal
1692int stat(char const* path, stat_info* s)
1693{
1694 return (int)(intptr)
1695 syscall2(
1696 SYS_stat,
1697 (void*)path,
1698 s
1699 );
1700}
1701
1702/* ------------------------------------------------------------- */
1703
1704internal
1705int stat_run(int argc, char const* argv[])
1706{
1707 stat_info si;
1708
1709 if (stat("/etc/hosts", &si) == 0) {
1710 write(stdout, &si, sizeof(stat_info));
1711 }
1712
1713 return 0;
1714}
1715-------------------------------------------------------------------
1716
1717Now if we hexdump output from amd64 and i386, we will see that
1718something is not quite right on i386:
1719-------------------------------------------------------------------
1720$ ./build.sh
1721$ ./stat | hexdump -C
172200000000 12 08 00 00 00 00 00 00 50 59 0a 00 00 00 00 00
172300000010 01 00 00 00 00 00 00 00 a4 81 00 00 00 00 00 00
172400000020 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
172500000030 bc 04 00 00 00 00 00 00 00 10 00 00 00 00 00 00
172600000040 08 00 00 00 00 00 00 00 24 b2 e9 57 00 00 00 00
172700000050 d1 f4 e1 2f 00 00 00 00 e8 d8 5e 57 00 00 00 00
172800000060 a0 3a b4 24 00 00 00 00 e8 d8 5e 57 00 00 00 00
172900000070 20 c8 0f 25 00 00 00 00 00 00 00 00 00 00 00 00
173000000080 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
173100000090
1732$ ./build.sh i386
1733$ ./stat | hexdump -C
173400000000 12 08 00 00 50 59 0a 00 a4 81 01 00 00 00 00 00
173500000010 00 00 00 00 bc 04 00 00 00 10 00 00 08 00 00 00
173600000020 24 b2 e9 57 d1 f4 e1 2f e8 d8 5e 57 a0 3a b4 24
173700000030 e8 d8 5e 57 20 c8 0f 25 00 00 00 00 00 00 00 00
173800000040 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
173900000050 00 00 00 00 00 00 00 00
174000000058
1741-------------------------------------------------------------------
1742
1743We know dev_t is a 64-bit integer from our previous investigations,
1744so why is other stuff being packed after the 4th byte? The first
17458 bytes of the structs should be the same as amd64!
1746
1747If you scroll through the stat manpage, you will find this:
1748-------------------------------------------------------------------
1749Over time, increases in the size of the stat structure have led to
1750three successive versions of stat(): sys_stat() (slot __NR_old?
1751stat), sys_newstat() (slot __NR_stat), and sys_stat64() (slot
1752__NR_stat64) on 32-bit platforms such as i386. The first two ver?
1753sions were already present in Linux 1.0 (albeit with different
1754names); the last was added in Linux 2.4. Similar remarks apply for
1755fstat() and lstat().
1756
1757The kernel-internal versions of the stat structure dealt with by
1758the different versions are, respectively:
1759
1760 __old_kernel_stat
1761 The original structure, with rather narrow fields,
1762 and no padding.
1763
1764 stat Larger st_ino field and padding added to various
1765 parts of the structure to allow for future expansion.
1766
1767 stat64 Even larger st_ino field, larger st_uid and st_gid
1768 fields to accommodate the Linux-2.4 expansion of UIDs
1769 and GIDs to 32 bits, and various other enlarged
1770 fields and further padding in the structure. (Vari?
1771 ous padding bytes were eventually consumed in Linux
1772 2.6, with the advent of 32-bit device IDs and
1773 nanosecond components for the timestamp fields.)
1774
1775The glibc stat() wrapper function hides these details from applica?
1776tions, invoking the most recent version of the system call provided
1777by the kernel, and repacking the returned information if required
1778for old binaries.
1779-------------------------------------------------------------------
1780
1781So it's likely that glibc is tampering with stat instead of just
1782forwarding the syscall.
1783
1784You can actually check this by writing a small libc stat test and
1785using strace to trace syscalls:
1786-------------------------------------------------------------------
1787$ cat > stattest.c << "EOF"
1788#include <sys/stat.h>
1789
1790int main()
1791{
1792 struct stat s;
1793 stat("/etc/hosts", &s);
1794 return 0;
1795}
1796EOF
1797
1798$ gcc -m32 stattest.c
1799$ strace ./a.out
1800execve("./a.out", ["./a.out"], [/* 83 vars */]) = 0
1801[ Process PID=22487 runs in 32 bit mode. ]
1802... stuff we don't care about ...
1803stat64("/etc/hosts", {st_mode=S_IFREG|0644, st_size=1212, ...}) = 0
1804exit_group(0) = ?
1805+++ exited with 0 +++
1806-------------------------------------------------------------------
1807
1808Yep, as expected, the stat call is getting translated to stat64!
1809
1810So how do we fix this? By not trusting libc headers and digging
1811into the kernel headers (which I found by googling the kernel
1812struct names):
1813-------------------------------------------------------------------
1814$ printf "#include <asm/stat.h>" \
1815 | gcc -m32 -E - | grep -A 30 "struct stat"
1816struct stat {
1817 unsigned long st_dev;
1818 unsigned long st_ino;
1819 unsigned short st_mode;
1820 unsigned short st_nlink;
1821 unsigned short st_uid;
1822 unsigned short st_gid;
1823 unsigned long st_rdev;
1824 unsigned long st_size;
1825 unsigned long st_blksize;
1826 unsigned long st_blocks;
1827 unsigned long st_atime;
1828 unsigned long st_atime_nsec;
1829 unsigned long st_mtime;
1830 unsigned long st_mtime_nsec;
1831 unsigned long st_ctime;
1832 unsigned long st_ctime_nsec;
1833 unsigned long __unused4;
1834 unsigned long __unused5;
1835};
1836-------------------------------------------------------------------
1837
1838That's a very different than what glibc headers were telling us!
1839There is no padding and st_dev is 4 bytes instead of 8, as well
1840as a lot of other fields having smaller sizes.
1841
1842What about the 64-bit version of it?
1843-------------------------------------------------------------------
1844$ printf "#include <asm/stat.h>" \
1845 | gcc -E - | grep -A 30 "struct stat"
1846struct stat {
1847 __kernel_ulong_t st_dev;
1848 __kernel_ulong_t st_ino;
1849 __kernel_ulong_t st_nlink;
1850
1851 unsigned int st_mode;
1852 unsigned int st_uid;
1853 unsigned int st_gid;
1854 unsigned int __pad0;
1855 __kernel_ulong_t st_rdev;
1856 __kernel_long_t st_size;
1857 __kernel_long_t st_blksize;
1858 __kernel_long_t st_blocks;
1859
1860 __kernel_ulong_t st_atime;
1861 __kernel_ulong_t st_atime_nsec;
1862 __kernel_ulong_t st_mtime;
1863 __kernel_ulong_t st_mtime_nsec;
1864 __kernel_ulong_t st_ctime;
1865 __kernel_ulong_t st_ctime_nsec;
1866 __kernel_long_t __unused[3];
1867};
1868-------------------------------------------------------------------
1869
1870This one seems to have the correct layout, except that some of the
1871values are unsigned rather than signed.
1872
1873Here's our fixed stat struct:
1874-------------------------------------------------------------------
1875typedef uintptr dev_t;
1876typedef intptr syscall_slong_t;
1877typedef uintptr syscall_ulong_t;
1878typedef uintptr time_t;
1879
1880typedef struct
1881{
1882 time_t sec;
1883 syscall_ulong_t nsec;
1884}
1885timespec;
1886
1887typedef struct
1888{
1889 dev_t dev;
1890 uintptr ino;
1891#ifdef AMD64
1892 uintptr nlink;
1893 u32 mode;
1894 u32 uid;
1895 u32 gid;
1896 u32 __pad0;
1897#else
1898 u16 mode;
1899 u16 nlink;
1900 u16 uid;
1901 u16 gid;
1902#endif
1903 dev_t rdev;
1904 uintptr size;
1905 uintptr blksize;
1906 uintptr blocks;
1907 timespec atim;
1908 timespec mtim;
1909 timespec ctim;
1910#ifdef AMD64
1911 syscall_slong_t __unused[3];
1912#else
1913 u32 __unused4;
1914 u32 __unused5;
1915#endif
1916}
1917stat_info;
1918-------------------------------------------------------------------
1919
1920Now we can run it again and verify that the struct is properly
1921populated in both architectures (I added comments to show where
1922fields are, those aren't actually part of hexdump)
1923-------------------------------------------------------------------
1924$ ./stat | hexdump -C
192500000000 12 08 00 00 00 00 00 00 50 59 0a 00 00 00 00 00
1926 | dev | ino |
1927
192800000010 01 00 00 00 00 00 00 00 a4 81 00 00 00 00 00 00
1929 | nlink | mode | uid |
1930
193100000020 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
1932 | gid | __pad0 | rdev |
1933
193400000030 bc 04 00 00 00 00 00 00 00 10 00 00 00 00 00 00
1935 | size | blksize |
1936
193700000040 08 00 00 00 00 00 00 00 24 b2 e9 57 00 00 00 00
1938 | blocks | atim.sec |
1939
194000000050 d1 f4 e1 2f 00 00 00 00 e8 d8 5e 57 00 00 00 00
1941 | atim.nsec | mtim.sec |
1942
194300000060 a0 3a b4 24 00 00 00 00 e8 d8 5e 57 00 00 00 00
1944 | mtim.nsec | ctim.sec |
1945
194600000070 20 c8 0f 25 00 00 00 00 00 00 00 00 00 00 00 00
1947 | ctim.nsec | __unused[0] |
1948
194900000080 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
1950 | __unused[1] | __unused[2] |
1951
195200000090
1953
1954$ ./build.sh i386
1955$ ./stat | hexdump -C
195600000000 12 08 00 00 50 59 0a 00 a4 81 01 00 00 00 00 00
1957 | dev | ino | mode |nlink| uid | gid |
1958
195900000010 00 00 00 00 bc 04 00 00 00 10 00 00 08 00 00 00
1960 | rdev | size | blksize | blocks |
1961
196200000020 24 b2 e9 57 d1 f4 e1 2f e8 d8 5e 57 a0 3a b4 24
1963 | atim.sec | atim.nsec | mtim.sec | mtim.nsec |
1964
196500000030 e8 d8 5e 57 20 c8 0f 25 00 00 00 00 00 00 00 00
1966 | ctim.sec | ctim.nsec | __unused4 | __unused5 |
1967
196800000040
1969-------------------------------------------------------------------
1970
1971In short, try getting structs from kernel headers instead of libc.
1972
1973###################################################################
1974 Legacy sockets on i386
1975###################################################################
1976
1977Another thing you should be aware of, is that some syscalls might
1978work entirely differently on i386 because of historical reasons.
1979
1980Socket syscalls are a perfect example. i386 doesn't have SYS_accept
1981and as far as I know the other socket syscalls are also not
1982guaranteed to exist.
1983
1984Instead, i386 multiplexes all socket syscalls through a single
1985syscall named "socketcall", which takes an additional param which
1986specifies which socket operation we want do to, (accept, connect,
1987etc...) followed by the usual syscall params that we find on amd64.
1988
1989Also, parameters for socketcall are passed through a void* array,
1990so the socketcall syscall just takes two parameters: the call
1991number and the pointer to the parameters array.
1992
1993Googling the socketcall numbers was a bit difficult, but I
1994eventually found them in linux/net.h.
1995
1996-------------------------------------------------------------------
1997$ printf "#include <sys/syscall.h>\nblah SYS_accept" \
1998 | gcc -m32 -E - | grep blah
1999blah SYS_accept
2000
2001$ man socketcall
2002SYNOPSIS
2003int socketcall(int call, unsigned long *args);
2004
2005DESCRIPTION
2006socketcall() is a common kernel entry point for the socket system
2007calls. call determines which socket function to invoke. args
2008points to a block containing the actual arguments, which are passed
2009through to the appropriate call.
2010
2011User programs should call the appropriate functions by their usual
2012names. Only standard library implementors and kernel hackers need
2013to know about socketcall().
2014
2015$ printf "#include <linux/net.h>\nblah SYS_SOCKET" \
2016 | gcc -m32 -E - | grep blah
2017blah 1
2018
2019$ printf "#include <linux/net.h>\nblah SYS_CONNECT" \
2020 | gcc -m32 -E - | grep blah
2021blah 3
2022-------------------------------------------------------------------
2023
2024Here's an example socket application for i386 and amd64 that
2025connects to sdf.org's gopherspace (192.94.73.15:70) and dumps the
2026output for the root folder.
2027
2028I got the sockaddr_in struct from netinet/in.h and the socket
2029constants from sys/socket.h
2030
2031-------------------------------------------------------------------
2032$ cat amd64/syscalls.h
2033#define SYS_read 0
2034#define SYS_write 1
2035#define SYS_close 3
2036#define SYS_socket 41
2037#define SYS_connect 42
2038#define SYS_exit 60
2039
2040$ cat i386/syscalls.h
2041#define SYS_read 3
2042#define SYS_write 4
2043#define SYS_close 6
2044#define SYS_exit 1
2045#define SYS_socketcall 102
2046
2047$ cat socket.c
2048#define internal static
2049
2050void* syscall1(
2051 uintptr number,
2052 void* arg1
2053);
2054
2055void* syscall2(
2056 uintptr number,
2057 void* arg1,
2058 void* arg2
2059);
2060
2061void* syscall3(
2062 uintptr number,
2063 void* arg1,
2064 void* arg2,
2065 void* arg3
2066);
2067
2068/* ------------------------------------------------------------- */
2069
2070#define stdout 1
2071#define stderr 2
2072
2073internal
2074void close(int fd) {
2075 syscall1(SYS_close, (void*)(intptr)fd);
2076}
2077
2078internal
2079intptr write(int fd, void const* data, uintptr nbytes)
2080{
2081 return (uintptr)
2082 syscall3(
2083 SYS_write,
2084 (void*)(intptr)fd,
2085 (void*)data,
2086 (void*)nbytes
2087 );
2088}
2089
2090internal
2091intptr read(int fd, void* data, intptr nbytes)
2092{
2093 return (intptr)
2094 syscall3(
2095 SYS_read,
2096 (void*)(intptr)fd,
2097 data,
2098 (void*)nbytes
2099 );
2100}
2101
2102#define AF_INET 2
2103#define SOCK_STREAM 1
2104#define IPPROTO_TCP 6
2105
2106typedef struct
2107{
2108 u16 family;
2109 u16 port; /* NOTE: this is big endian!!!!!!! use flip16u */
2110 u32 addr; /* this is also big endian */
2111 u8 zero[8];
2112}
2113sockaddr_in;
2114
2115#ifdef SYS_socketcall
2116/* i386 multiplexes socket calls through socketcall */
2117#define SYS_SOCKET 1
2118#define SYS_CONNECT 3
2119
2120internal
2121int socketcall(u32 call, void* args)
2122{
2123 return (int)(intptr)
2124 syscall2(
2125 SYS_socketcall,
2126 (void*)(intptr)call,
2127 args
2128 );
2129}
2130#endif
2131
2132internal
2133int socket(u16 family, i32 type, i32 protocol)
2134{
2135#ifndef SYS_socketcall
2136 return (int)(intptr)
2137 syscall3(
2138 SYS_socket,
2139 (void*)(intptr)family,
2140 (void*)(intptr)type,
2141 (void*)(intptr)protocol
2142 );
2143#else
2144 void* args[3];
2145 args[0] = (void*)(intptr)family;
2146 args[1] = (void*)(intptr)type;
2147 args[2] = (void*)(intptr)protocol;
2148
2149 return socketcall(SYS_SOCKET, args);
2150#endif
2151}
2152
2153internal
2154int connect(int sockfd, sockaddr_in const* addr)
2155{
2156#ifndef SYS_socketcall
2157 return (int)(intptr)
2158 syscall3(
2159 SYS_connect,
2160 (void*)(intptr)sockfd,
2161 (void*)addr,
2162 (void*)sizeof(sockaddr_in)
2163 );
2164#else
2165 void* args[3];
2166 args[0] = (void*)(intptr)sockfd;
2167 args[1] = (void*)addr;
2168 args[2] = (void*)sizeof(sockaddr_in);
2169
2170 return socketcall(SYS_CONNECT, args);
2171#endif
2172}
2173
2174/* ------------------------------------------------------------- */
2175
2176internal
2177intptr strlen(char const* str)
2178{
2179 char const* p;
2180 for(p = str; *p; ++p);
2181 return p - str;
2182}
2183
2184internal
2185intptr fputs(int fd, char const* str) {
2186 return write(fd, str, strlen(str));
2187}
2188
2189/* reverses byte order of a 16-bit integer (0x1234 -> 0x3412) */
2190internal
2191u16 flip16u(u16 v) {
2192 return (v << 8) | (v >> 8);
2193}
2194
2195/* ------------------------------------------------------------- */
2196
2197#define BUFSIZE 512
2198
2199internal
2200int socket_run(int argc, char const* argv[])
2201{
2202 int res = 0; /* return code */
2203
2204 int fd;
2205 u8 ip_raw[] = { 192, 94, 73, 15 }; /* ip in big endian order */
2206 u32* pip = (u32*)ip_raw; /* pointer to ip as a 32-bit int */
2207 sockaddr_in a;
2208
2209 intptr n;
2210 u8 buf[BUFSIZE];
2211
2212 /* set up sockaddr struct with desired ip & port */
2213 a.family = AF_INET;
2214 a.port = flip16u(70);
2215 a.addr = *pip;
2216
2217 for (n = 0; n < 8; ++n) {
2218 a.zero[n] = 0;
2219 }
2220
2221 /* create a new socket */
2222 fd = socket(AF_INET, SOCK_STREAM, IPPROTO_TCP);
2223 if (fd < 0) {
2224 fputs(stderr, "socket failed\n");
2225 return 1;
2226 }
2227
2228 /* connect to sdf.org */
2229 if (connect(fd, &a) < 0)
2230 {
2231 fputs(stderr, "connect failed\n");
2232 res = 1;
2233 goto cleanup;
2234 }
2235
2236 /* request folder / */
2237 fputs(fd, "/\r\n");
2238
2239 /* read chunks of BUFSIZE bytes and relay them to stdout until
2240 there is nothing left to read or the socket errors out */
2241 while (1)
2242 {
2243 n = read(fd, buf, BUFSIZE);
2244 if (n <= 0) {
2245 break;
2246 }
2247
2248 if (write(stdout, buf, n) != n)
2249 {
2250 fputs(stderr, "write failed\n");
2251 res = 1;
2252 break;
2253 }
2254 }
2255
2256 if (n < 0) {
2257 fputs(stderr, "read failed\n");
2258 res = 1;
2259 }
2260
2261cleanup:
2262 /* make sure to not leave a dangling socket file descriptor */
2263 close(fd);
2264
2265 return res;
2266}
2267-------------------------------------------------------------------
2268
2269And as you can see, we are running flawlessly on both architectures
2270-------------------------------------------------------------------
2271$ ./build.sh && ./socket
2272iWelcome to the SDF Public Access UNIX System .. est. 1987...
2273
2274$ ./build.sh i386 && ./socket
2275iWelcome to the SDF Public Access UNIX System .. est. 1987...
2276-------------------------------------------------------------------
2277
2278###################################################################
2279 Conclusion
2280###################################################################
2281I hope this guide got you interested in understanding what happens
2282at the lowest level and knowing your programming language and OS
2283beyond the standard library! Have fun! I will add more tricks if
2284I come up with new ones.