An experiment in porting the Android init system to GNU/Linux
https://blog.darknedgy.net/technology/2015/08/05/0-androidinit/
by V.R.
Yes I’m trolling/mocking you anti-systemd retards. That’s what Schopenhauer recommends as a strategy against irrational rhetoric (here of the reactionary kind). I couldn’t care less for irrational people’s reading recommendations.
P.S.: “Self-projection” is psychobabble grounded in no science."
Preface
Being the author of the now dead uselessd project, I harbor a knack and appreciation for all things that are useless, or that use less, depending on your point of view.
While sitting in my room drinking absinthe, I was struck with a flash of brilliance. An idea for a project, so devoid of utility, that it is making John Stuart Mill do the tube snake boogie in his grave as we speak.
Take the Android init daemon… and boot a GNU/Linux system with it.
(Slackware Linux, as usual.)
Marvelous.
Now the word shall be able to point at this work and be able to see the true icon I aim to promote, that of Robert Nozick fighting the utility monster.
There are, however, some serious reasons to embark on this project:
- To study the core system platform (low-level userspace) of Android.
- To see how much Android diverts from GNU/Linux in practice, and whether porting its core daemons and libraries is a practical effort to pursue.
- To observe discrepancies between Bionic libc and glibc.
- To figure out what Android brings to the table of init daemons and process managers.
- It’s a fun idea for a hack.
The porting process itself roughly follows this order:
- Write a standalone Makefile.
- Identify all Android-specific interfaces. Insert, rewrite or remove as needed.
- Force everything into C++ semantics (see “The C/C++ quagmire” section below).
- Write a suitable init.rc file.
- Build, copy, append the init parameter on boot and give it a spin.
Android ain’t GNU/Linux, as we shall observe.
Let us dive right in to the overview, then.
Overview of Android init
The Android init daemon’s documentation is rather scant. Technical information about Android in general before its first formal unveiling in November 2007 is difficult to find, but Git logs at googlesource.com do seem to put the origin of Android init to that same time period, 2007-2008.
It thus came out after launchd and Upstart, but also predates systemd.
The intended use case for Android init is quite different compared to most init systems for Unix-like operating systems. Whereas most exist to be proactively used by the sysadmin or to empower them in some way, Android init’s purpose is for device manufacturers to have something they can configure once, shove into an initramfs, fire and never touch again.
As a result, Android init is configured in a thoroughly monolithic fashion, with a long and elaborate boot time logic being codified largely into one file, though it does support basic imports.
It is distinguished by its ad-hoc, line-oriented, section-based configuration language, which Rob Landley once said of (paraphrasing) that it “looks like a shell script, but is not”.
The likely motivation for the language was to encapsulate several common POSIX operations and process manager directives into a mini-shell that can be parsed and executed with minimal overhead, particularly not invoking any environmental and security baggage of a full Unix shell. This made sense for the limited system resources of mobile devices as recently as 2007, and still to this day given the relative heterogeneity of Android installations. Though, one wonders whether writing a basic lexer for chain loading programs, in the vein of execline, would have been a better approach.
Android init is actually composed of three programs: init, ueventd and watchdogd. However, they all hook into the basename of the argv[0] of the init program, meaning that the latter two are usually accessed as symlinks to init.
The init configuration language, as read from /init.rc, has five main statements (or semantic constructs): actions, commands, triggers, services and options.
An action is not unlike a queued event. In fact, it is quite reminiscent of the event mechanism in Upstart jobs. Given Google’s internal usage of Ubuntu as Goobuntu, and their use of Upstart in ChromeOS, it is very likely, though speculation, that the mechanism was influenced from it. The action itself actually just opens a new block scope, whereas the event itself is called a trigger. The init daemon automatically registers several intrinsic triggers like early-init, init and boot, whereas others can be user-defined either through Android system properties (described below), or by using the trigger command within another action.
They follow the syntax of:
on boot
mount tmpfs tmpfs /dev/shm nodev nosuid
exec -- /bin/mount -a
# insert other commands here
A command is a tokened directive which encapsulates a function to some POSIX, Android or init-specific interface, not unlike a command line you would execute from a shell. Commands include exec, copy, write, symlink, chmod, hostname, insmod, mount_all and others.
A service is what you’d expect it to be. It’s a definition encapsulating a system service/daemon or one-shot job that accepts an absolute pathname and program arguments. Within its scope it accepts modifiers called options, which include socket definitions for inetd-like superserver functionality, setenv, user, group, class for creating logical groups and so forth. The options are not as featureful as most server/desktop-oriented init(8)s, largely reflecting the fire-and-forget embedded heritage. For example:
service surfaceflinger /system/bin/surfaceflinger
class core
user system
group graphics drmrpc
onrestart restart zygote
The Android init comes with built-in bootcharting and some scripts to process the output, with its intended usage being booting from an emulator with the -bootchart option and interacting with files on the /data partition. The scripts are unportable, and I have not tested bootcharting itself.
There is no dependency resolution or automatic ordering of any form (other than pre-opening sockets), this being settled once in the init.rc file and tuned per a vendor’s needs. The only way to express service relationships, other than grouping them into classes (though this is more akin to a target or runlevel) is onrestart, so as to resynchronize the state of any services that interact with the one undergoing a restart from the SIGCHLD handler’s state change watcher.
init.rc files can be complemented or overridden on a hardware-specific basis. This is usually done by vendors for specific SoCs or single-boards.
The full list of commands and options can be found registered in the keywords.h file, with the implementations in builtins.cpp, and their invocations in init_parser.cpp.
The main init loop sets $PATH, mounts its reserved socket namespace at /dev/socket, as well as devtmpfs, /dev/pts, procfs and sysfs, sets the SIGCHLD handler for tracking state changes and doing autorestarts, initializes the system property workspace (described below), and then polls on the action queue.
init can optionally display a boot splash, otherwise defaulting on a console message of “A N D R O I D” in the console_init_action() function of init.cpp.
ueventd is a hotplug device node manager which subscribes to kernel uevents from a Netlink channel and performs operations on devtmpfs, much like udev, except lighter weight. It reads its rule sets from a static /uevent.rc file, a stock example of which can be found here. Android understands a kernel command line parameter android.hardwareboot for dispatching device-specific uevent-.rc files on startup. It is symlinked to init and its entry point is ueventd_main() from init.cpp.
ueventd performs as so-called coldboot process, which init explicitly anticipates. It is described in a source code comment in init/devices.cpp as such:
/* Coldboot walks parts of the /sys tree and pokes the uevent files
** to cause the kernel to regenerate device add events that happened
** before init's device manager was started
**
** We drain any pending events from the netlink socket every time
** we poke another uevent file to make sure we don't overrun the
** socket's buffer.
*/
watchdogd is just a really basic interface to the Linux kernel software watchdog (/dev/watchdog), encapsulating various hardware watchdog timers. It pings into the driver’s buffer under a specified clock interval, triggering a system reset if the response interval is not met. It is symlinked to init and its entry point is watchdogd_main() from init.cpp.
In conclusion, Android init implements a small event-driven architecture meant for ease of configurability for device manufacturers and of little relevance to most other perspectives, even reasonably technical users.
The porting process, a.k.a. remove Android-isms from the premises
The C/C++ quagmire
The Android core platform, from 2007 onwards, has haphazardly evolved from pure C into a “C with classes” usage of C++. Much of the code between utility libraries and daemons thus remains mixed, and even within C++ programs (compiled with -std=c++11) the level of C versus C++ constructs varies between source files.
For this port, I decided to force the entire source tree into C++ semantics. One of the classic pitfalls is that C++ does not implicitly convert types from a pointer to void. As such, I needed to statically cast return values of functions like calloc(3) and strrchr(3) in several places.
Some missing header files for constructs like smart pointers had to be included.
Android build system
Android uses a recursive Makefile scheme where there is only one top-level Makefile, and all project subdirectories have an Android.mk file that is ultimately sourced into top-level. A full AOSP build also involves setting up environment variables and some shell macros, but that is outside our scope. More information can be found here.
For our purposes, we want a standalone project with its own Makefile. I quickly hacked up something similar to this repetitive but sufficient cruft:
# ueventd and watchdogd are traditionally symlinked to init
# they're conditionally invoked depending on basename(argv[0])
# here we just build the binary three-fold
CC=g++
CFLAGS=-Wall -Wextra -Wno-unused-parameter -Wno-write-strings \
-Wno-missing-field-initializers \
-std=c++11 -DGTEST_LINKED_AS_SHARED_LIBRARY=1
LDLIBS=-lpthread -lrt -lgtest
INIT_SRCS = bootchart.cpp builtins.cpp devices.cpp init.cpp init_parser.cpp \
log.cpp parser.cpp signal_handler.cpp util.cpp watchdogd.cpp \
ueventd_parser.cpp ueventd.cpp
UTIL_SRCS = klog.cpp stringprintf.cpp file.cpp strings.cpp android_reboot.cpp \
iosched_policy.cpp multiuser.cpp uevent.cpp fs_mgr.cpp \
fs_mgr_fstab.cpp strlcat.cpp strlcpy.cpp logwrap.cpp
TEST_SRCS = init_parser_test.cpp util_test.cpp
INIT_OBJS = $(SRCS:.c=.o)
INIT_MAIN = android-init
all: init ueventd watchdogd
init: $(OBJS)
@echo "Building init."
$(CC) $(CFLAGS) $(INIT_SRCS) $(UTIL_SRCS) \
-o $(INIT_MAIN) $(INIT_OBJS) \
$(LDLIBS)
ueventd: $(OBJS)
@echo "Building ueventd, which is hooked to argv[0] of init."
$(CC) $(CFLAGS) $(INIT_SRCS) $(UTIL_SRCS) \
-o ueventd $(INIT_OBJS) \
$(LDLIBS)
watchdogd: $(OBJS)
@echo "Building watchdogd, which is hooked to argv[0] of init."
$(CC) $(CFLAGS) $(INIT_SRCS) $(UTIL_SRCS) \
-o watchdogd $(INIT_OBJS) \
$(LDLIBS)
tests: $(OBJS)
$(CC) $(CFLAGS) init_parser_test.cpp $(UTIL_SRCS) \
-o init_parser_test $(INIT_OBJS) \
$(LDLIBS)
$(CC) $(CFLAGS) util_test.cpp $(UTIL_SRCS) \
-o util_test $(INIT_OBJS) \
$(LDLIBS)
clean:
rm -f android-init watchdogd ueventd
Not the use of the Google Tests library in two files. The init parser test had to be modified to remove SELinux seclabel code (which is an indicator that SELinux security context information is being stored in the xattrs of the FS) and to fix the pathnames to match the Unix FHS.
libbase
AOSP refactored some of their internal functions for buffered file I/O, string building, string formatting and logging into a Base class linked to a library called libbase.
I simply copied base/stringprintf.h, base/strings.h and base/file.h verbatim with their source files and included them to the Makefile. Some headers had to be included for definitions like BUFSIZ and the stdint types. Additionally, instead of using the cutils logging macros, I replaced their usages with calls to fprintf(stderr, “…”) [corresponding to ALOGE].
Android FHS
Android does not follow the FHS, instead using its own directory layout optimized for things like easy factory reset. Though variations are seen across device vendors, in general you have everything lumped into /system (including /system/bin, /system/etc and so forth), Dalvik/ART bytecode in /cache, miscellaneous configuration, app cache, crash/debug info and backups in /data, the recovery image partition in /recovery and so forth. More details can be found here.
The init.rc and ueventd.rc files are actually stored in /. I ended up remapping several pathnames from /system/bin to /sbin (for things like e2fsck and f2fs.fsck), /data to /etc and the .rc configuration files to /etc, as well.
BSD libc
Android’s own libc, Bionic, is in fact based in large part to OpenBSD libc. As such, it inherits several of Ulrich Drepper’s archnemeses, namely strlcpy(3) and strlcat(3), in addition to getprogname(3) in the init logging functions.
I copied and included the strlc* functions, while I replaced getprogname(3) with the glibc-specific program_invocation_short_name, which is akin to the basename of a global scope argv[0] in main().
There was also the __printflike macro, specific to <sys/cdefs.h> seen in most BSD libcs, which in GCC can be recreated as __attribute__ ((format(printf, 2, 3))), as documented here, allowing for the compiler to type check function arguments as matching printf(3) format string rules.
System properties
Android has its Bionic libc-specific feature called system properties, or just properties. They represent an odd blend between sysctl-like key-value pairs (the same dot.separated.string.format) and registry keys. They are used by both low-level Android programs and Android apps running in the Dalvik/ART VM for runtime configuration and feature toggles.
Android properties are implemented as a region of shared memory called the property workspace, which resides in the property service, implemented as part of Android init and initialized in the PID1 main loop. Properties will then be loaded from the /default.prop, /system/build.prop, /system/default.prop and /data/local.prop files in persistent storage.
The property service itself listens on /dev/socket/property_service, which init polls on for events. Only the property service can manipulate the property workspace. Processes wanting to use properties go through the libcutils property functions (or a higher level Java wrapper), themselves calling the Bionic libc functions. The shared memory block is loaded into their own virtual address space for properties to be read, however writing operations are further redirected to the property service’s control socket.
In essence, they’re used as a rather ad-hoc IPC and synchronization based on top of shared memory and file descriptors. Android init itself can create custom triggers from init.rc based on reacting to property changes, and uses them in its core logic in several places to probe for subsystems.
Because they’re not exactly portable (they actually might be in theory, since the Bionic code doesn’t appear to do anything too low-level, but I didn’t bother to go that far and that might actually be a research topic and blog article for another time), I eviscerated them entirely. The property service, all property checks, set/getprop, load_all_prop, load_persist_prop commands, initialization and so forth were removed, and so the ability to have properties act as triggers.
The fs_mgr
The fs_mgr is an object bundled into a library, with an optional standalone utility, for parsing, traversing and performing mount operations based on fstab(5) operation. It is written purely in C.
Only portions of it from its source files were added, particularly the basic parsing, swapon, mount/umount, tmpfs mount and fsck logic (which simply exec()s /sbin/e2fsck). Code for low-level ext4 superblock parsing, encryptable partitions and the Verified Boot (signed bootloader and initramfs; see below) scheme was not kept. Static casts and adjustments to mount calls were made where necessary.
libcutils
libcutils are rather self-explanatory. They are AOSP’s low-level C interfaces for the kernel and libc subsystems used all throughout core system platform.
In our case, we imported the following:
<cutils/klog.h> for logging to /dev/kmsg. The __printflike macro had to be expanded as explained in the “BSD libc” section.
<cutils/android_reboot.h for the system reboot callbacks. The <linux/reboot.h> header had to be included, and the __unused macro expanded to __attribute__((unused)) in the flags parameter of the android_reboot_with_callback() function, which itself is called from the higher-level android_reboot().
<cutils/iosched_priority.h for wrapping the ioprio_get(2) and ioprio_set(2) system calls.
<cutils/multiuser.h for calculating application UIDs.
<cutils/uevent.h> for the low-level Netlink connection that ueventd uses to subscribe to kernel uevents. To avoid a C++ scope boundary crossing error where a jump label skips some struct initializations, those had to be moved to the top level of the uevent_kernel_rcv() function.
<cutils/list.h> for simple doubly linked list macros.
Of <cutils/sockets.h, all we needed was to set two macros: ANDROID_SOCKET_ENV_PREFIX and ANDROID_SOCKET_DIR (/dev/socket).
SELinux
Android init uses SELinux security contexts very liberally. Though I’m not one to yell “OH GOD SELINUX IS NSA BACKDOOR, YOU’RE ALL PART OF THE BOTNET NOW”, I legitimately am not a fan of its convoluted RBAC model. I have instead always appreciated the elegance of a nice capability-based system (not to be confused with the deceptively named POSIX capabilities).
Furthermore, Slackware has never officially supported SELinux, nor PAM. These are all blessings in my mind, but I wasn’t willing to go through the effort of being distribution SELinux maintainer just for this one-off project.
The Android init also uses AOSP-specific SELinux extensions, which are unportable.
SELinux removal entailed ridding of the setcon, restorecon, restorecon_recursive and seclabel commands, as well as SELinux initialization, policy loading and context creation/checking code sprinkled throughout init, ueventd and utilities. The open_devnull_stdio() function in utils.cpp retained its attempt to open /sys/fs/selinux/null, since it can handle failures gracefully and default to /dev/null anyway.
Keychord driver and keycodes
Android has a scarcely documented keychord driver registered as a device node at /dev/keychord, which is used for, among other things, setting keycode combinations to trigger debug actions for Android init-supervised services during adb sessions.
As it is heavily system-specific and irrelevant to our use cases, it was removed, along with the (undocumented) keycodes option in the init.rc parser.
Logwrapper
logwrapper is a basic superserver utility library that forks/execs a program while redirecting its stdout/stderr either to the Android system log by priority, or the kernel ring buffer log.
In our case, we imported it with C++ type casting modifications, and again the ALOGE macros redefined to fprintf(stderr, …).
It was needed largely because of the android_fork_execvp_ext() function, used in the fs_mgr for launching /sbin/mkswap and /sbin/e2fsck on the parsed fstab(5) structure, as well as in the built-in unmount_and_fsck() function which is called as part of the shutdown procedure.
Verified boot
Android since 4.4 “Kit Kat” has a so-called “verified boot” system based on a DeviceMapper extension which uses signed Merkle trees of an ext4 image, verifying each read block by comparing it to the tree passed during setup. It is officially documented here.
There are two commands verity_get_state and verity_load_state related to said DeviceMapper module, which have been removed. These functions internally delegate to the fs_mgr, this portion of which we have not copied over. It is specifically in fs_mgr/fs_mgr_verity.c, which is where the bulk of the verified boot logic in general resides.
ext4_utils
AOSP ships with the ext4_utils for manipulating the low-level disk layout of ext4fs, which is well specified in this kernel wiki article. Particularly, AOSP implemented their ext4_crypt_init_extensions as part of the verified boot scheme described above, later merged into upstream Linux.
Since we did not need this and did not want to meddle with aligning ext4 structure definitions, it was removed entirely in the device management along with the installkey command.
File system config definitions
Android’s system image generation tools like mkbootfs define some system properties and definitions for user and group IDs matching various subsystems, like debug bridge, camera, NFC, DHCP client and so forth. These are defined in <private/android_filesystem_config.h>.
In our case, there was a setegid(2) operation taking the value of AID_ROOT, which is unsurprisingly just 0.
That’s mostly it. On to the final phase.
Build, configure, run
Well, shit.
After building the thing, I copied it over to /sbin/android-init. I then wrote a crude init.rc (parsed in /etc/init.rc in the port) file based on the Slackware Linux rc scripts.
It’s quite ugly, but elegance was no priority here:
# Android init.rc file for a Slackware Linux 14.1 system
on early-init
loglevel 3
export PATH=/sbin:/usr/sbin:/bin:/usr/bin
export LANG=C
on boot
hostname thatmemeisdank
# RTC tick rate
sysctltkz 10000
exec -- /sbin/hwclock --hctosys
exec -- /sbin/swapon -a
rm /etc/mtab
symlink /proc/mtab /etc/mtab
# kmods
exec -- /sbin/depmod -A
insmod parport_pc
insmod lp
insmod agpgart
insmod loop
mount tmpfs cgroup_root /sys/fs/cgroup mode=0750,uid=0,gid=1000
mount tmpfs none /run mode=0755
# Create location for fs_mgr to store abbreviated output from filesystem
# checker programs.
mkdir /dev/fscklogs 0770 root system
exec -- /sbin/sysctl -e -p /etc/sysctl.conf
# new utmp entry
exec -- /bin/touch /var/run/utmp
chown root:utmp /var/run/utmp
chmod 664 /var/run/utmp
chmod 664 /var/log/dmesg
rm /etc/nologin
mount_all /etc/fstab
# entropy pool
exec -- /bin/cat /etc/random-seed > /dev/urandom
exec -- /usr/bin/dd if=/dev/urandom of=/etc/random-seed count=1 bs=512 2> /dev/null
chmod 600 /etc/random-seed
# services
class_start core
class_start network
class_start jobs
class_start gettys
service syslogd /usr/sbin/syslogd -a -d /var/log
class core
onrestart restart klogd
service klogd /usr/sbin/klogd -c 3 -x
class core
service bring-up-network-interfaces /etc/rc.d/rc.inet1 start
class network
oneshot
service port-forwarding /etc/rc.d/rc.ip_forward start
class network
oneshot
service inetd /usr/sbin/inetd
class network
service acpid /usr/sbin/acpid
class core
service crond /usr/sbin/crond -l notice
class core
service atd /usr/sbin/atd -b 15 -l 1
class core
service dbus-uuidgen /usr/bin/dbus-uuidgen --ensure
class jobs
oneshot
service dbus-system /usr/bin/dbus-daemon --system
class core
service httpd /usr/sbin/apachectl -k start
class core
service update-so-links /sbin/ldconfig
class jobs
oneshot
service update-font-cache /usr/bin/fc-cache -f
class jobs
oneshot
service gpm /usr/sbin/gpm -m /dev/mouse -t ps2
class core
service load-alsa-mixing /usr/sbin/alsactl restore
class jobs
oneshot
service update-mime-database /usr/bin/update-mime-database /usr/share/mime
class jobs
oneshot
service rc-local /etc/rc.d/rc.local
class jobs
oneshot
service pulseaudio /usr/bin/pulseaudio --start --log-target syslog
class core
service getty1 /sbin/agetty --noclear 38400 tty1 linux
class gettys
service getty2 /sbin/agetty 38400 tty2 linux
class gettys
service getty3 /sbin/agetty 38400 tty3 linux
class gettys
service getty4 /sbin/agetty 38400 tty4 linux
class gettys
service getty5 /sbin/agetty 38400 tty5 linux
class gettys
We resort to cheap exec()s a lot, beside the generally messy logic.
Slackware uses the venerable LILO, which I actually quite enjoy despite most people looking at you like you’re a fossil for using it.
Anyway, I added an entry to my /etc/lilo.conf like so:
image = /boot/vmlinuz-huge-smp-$KERNEL_VSN-smp
root = /dev/sda1
label = androidinit
append="init=/sbin/android-init"
read-only
Committed to MBR:
/sbin/lilo -v
And reboot.
The moment of truth
There’s some good news and some bad news.
The bad news is this particular experiment failed. The fs_mgr object crashes, but this in of itself isn’t too bad, we can always do a /sbin/mount -a.
However, despite the init.rc file parsing, the SIGCHLD handler is FUBAR along with the command execution logic, and my suspicion is that this is precisely because of the lack of property service and workspace. Particularly, service state change notifications relied on property getting and setting, which we don’t have, and thus there probably isn’t any information for init(8) to deduce its sequence.
Now, the interesting thing is that someone did an Android-based init thing for GNU/Linux 5 years prior. I actually don’t know if it’s from 2010, but that was the commit date on GitHub. It might actually be earlier. It doesn’t have system properties at all, and it’s a much smaller pure C program, lacking nearly all of the interfaces we described above. It thus appears to be only loosely based on the Android init code, as compared to my rather faithful porting attempt.
And it worked… but only barely. Shutdown logic is broken, I’m unaware if it even handles autorestart properly, and way I got it to boot my system was to just lazily exec the /etc/rc.d/rc.S and /etc/rc.d/rc.M scripts, then wrap a few gettys in a class and start those.
Closing remarks
(My raw source files can be found at this GitHub repository. Use at your own discretion. If you get something working, open an issue/PR or contact V.R. at the Dark n' Edgy forums.)
Android ain’t GNU/Linux, and it shows. The amount of AOSP-specific interfaces that diverge and pose practical problems for interoperability with regular GNU/Linux systems, the ones we actually call “Linux” at all in daily speech, are numerous.
As such, the only practical strategy is not to port, as much as to loosely adapt (as in the aforementioned absabs/init project). Another possibility is that system properties and other intricacies are in fact doable to port, as they simply appear to be shared memory, fds and sockets, even if residing in the libc level. That indeed might be left as an experiment for a later time (if ever), perhaps for totally orthogonal reasons to running Android init.
The Android init is generally very purpose-built, and in any event it is probably of little interest for servers and desktops. It’s a niche, system-specific thing that to reiterate, solves the problem of being a small event-driven design meant to be configured once and chucked into an initramfs never to be touched again.
But, then again, say what you will about Android init. At least it doesn’t warrant a goddamned annual conference.
