Build DMD for Android

This is a work in progress: Android/x86 is mostly done, going to work on Android/ARM next. Almost all the druntime/phobos unit tests pass on Android/x86, plus a C/OpenGLES 1.0 purely native sample app from the NDK has been ported to D. I list work that still needs to be done at the end.

Prerequisites

• 32-bit linux/x86 host on which to build dmd
  • A virtual machine like VirtualBox/VMware will work fine, but you should have at least 512 MB of memory allocated and 1 GB of swap, particularly if building the phobos unit tests, and 10 GB of disk space.
• C++ compiler and toolchain, to build dmd
• dmd/druntime/phobos source
  • Get druntime and phobos from git, as they have Android-specific changes that are not in the release branches yet.
• Android native toolchain, the 32-bit NDK and SDK
  • The "SDK Tools only" version of the SDK is enough, if you don't plan on using their IDE integration. I will only write about using the command-line tools. The SDK requires JDK 6 and Ant 1.8 or later, follow their instructions to make sure it's installed right.
• Android/x86, whether a device or VM
  • The SDK comes with an Android/x86 emulator and Intel puts out one of their own. I use the builds put out by the android-x86 project and recommend the deprecated 4.3 build, as it's the least buggy one I've dealt with. I run the iso in a VM with 512 MBs of memory without installing to disk.

To run Android/x86 in a VM

Download the Android-x86 iso and install as normal: it'll put you through a configuration process, but you can skip every step. Go to Settings->Security and disable the Verify Apps option, or it'll ask you about that every time you install a test app. Alt-F1 will take you to a root shell, which you'll want to do any time you're not using Android/x86 for a couple minutes or the screen will eventually go black and the VM will freeze up. Alt-F7 takes you back to the normal GUI screen.

Build D for Android/x86

Download the patch for dmd, apply it, and build normally:

cd dmd
git apply packed_tls_for_elf.patch
make -f posix.mak -j5

This patch adds TLS support for Android/x86, more info can be found on the pull request.

Assuming druntime and phobos are in the same directory as dmd, download and apply the patch for druntime and the patch for phobos, set the NDK environment variable to the path of wherever you installed the NDK, and build each in turn with PIC enabled:

cd ../druntime
export NDK=/path/to/your/android-ndk-r10
git apply druntime_build.patch
make -f posix.mak PIC=1

cd ../phobos
git apply phobos_build.patch
make -f posix.mak PIC=1

Both patches avoid building druntime/phobos as a shared library and set the appropriate C compiler path and flags. The druntime patch also adds an import I missed before. The remaining changes are for the unit tests and are addressed below.

Build an Android sample app

I've put up an android repository which contains several translated headers and a C sample app translated to D. Clone it into the same directory as dmd/druntime/phobos:

cd ..
git clone https://github.com/joakim-noah/android.git

Default build of the C sample app

First, let's see how it's done by default by compiling the C version. Go to the sample native-activity app in my android repo, which is almost copied verbatim from the NDK/samples. The only differences are that it also contains a translation of the given jni/main.c source to jni/main.d, changes one line in jni/Application.mk to build for x86 by default, and includes a dmd.conf for building with dmd.

Run the following commands to compile the C source into a debug app that you can install on Android, these commands are taken from the NDK instructions (scroll down to "Exploring the native-activity Sample Application"):

cd android/samples/native-activity
NDK_TOOLCHAIN_VERSION=clang $NDK/ndk-build V=1

Looking at the output, it compiles jni/main.c, a small wrapper library called android_native_app_glue, and links everything together into a shared library. Let's look at the command that compiles jni/main.c:

/home/joakim/android-ndk-r10/toolchains/llvm-3.4/prebuilt/linux-x86/bin/clang
-MMD -MP -MF ./obj/local/x86/objs/native-activity/main.o.d -gcc-toolchain 
/home/joakim/android-ndk-r10/toolchains/x86-4.8/prebuilt/linux-x86 -target
i686-none-linux-android -ffunction-sections -funwind-tables -fstack-protector
-fPIC -no-canonical-prefixes -O2 -g -DNDEBUG -fomit-frame-pointer -fstrict-aliasing 
-I/home/joakim/android-ndk-r10/sources/android/native_app_glue -Ijni -DANDROID
-Wa,--noexecstack -Wformat -Werror=format-security
-I/home/joakim/android-ndk-r10/platforms/android-9/arch-x86/usr/include 
-c jni/main.c -o ./obj/local/x86/objs/native-activity/main.o

This is where I extracted the C compiler path and flags for the previous druntime/phobos patches, leaving out the dependency file (-MMD -MP -MF).

This command links the shared library that gets packaged into the native app:

/home/joakim/android-ndk-r10/toolchains/llvm-3.4/prebuilt/linux-x86/bin/clang++
-Wl,-soname,libnative-activity.so -shared
--sysroot=/home/joakim/android-ndk-r10/platforms/android-9/arch-x86
./obj/local/x86/objs/native-activity/main.o
./obj/local/x86/libandroid_native_app_glue.a -lgcc  -gcc-toolchain
/home/joakim/android-ndk-r10/toolchains/x86-4.8/prebuilt/linux-x86 -target 
i686-none-linux-android -no-canonical-prefixes  -Wl,--no-undefined
-Wl,-z,noexecstack -Wl,-z,relro -Wl,-z,now
-L/home/joakim/android-ndk-r10/platforms/android-9/arch-x86/usr/lib
-llog -landroid -lEGL -lGLESv1_CM -llog -lc -lm
-o ./obj/local/x86/libnative-activity.so

You'll use a modified version of the above command to link your D source later.

Getting back to building the C app, set the SDK environment variable for the path to your SDK and run the commands to package an Android app/apk:

export SDK=/path/to/your/android-sdk-linux
$SDK/tools/android update project -p . -s --target 1
ant debug

Now you'll push the final app, bin/NativeActivity-debug.apk, to your Android/x86 environment. Connecting to Android will vary based on which Android/x86 you're using: I'll show you how I do it for Android/x86 installed in a VM, using the first method shown in the Android-x86 docs. Hit Alt-F1 inside the VM to go to the root shell and type "netcfg" to get its IP address: let's say it's 192.168.0.1. Hit Alt-F7 to go back to the UI, then go to Settings->Apps. It should be empty.

Going back to the linux host:

$SDK/platform-tools/adb connect 192.168.0.1:5555
$SDK/platform-tools/adb install bin/NativeActivity-debug.apk
$SDK/platform-tools/adb logcat native-activity *:S

The NativeActivity app should show up in the Settings->Apps list in Android/x86. Go to the app launcher and click on NativeActivity to run it. Move the mouse and you should see a bunch of colors continuously flashing on the screen. Looking at the log dump in the linux host, you'll see some numbers from the accelerometer, which are being reported by the app.

Build the translated D app

I've translated jni/main.c to a D version, jni/main.d. First, you'll need to recompile the android_native_app_glue library so that it calls a couple functions necessary for a D shared library. Hit ctrl-c to get out of the Android log and open $NDK/sources/android/native_app_glue/android_native_app_glue.c in an editor. Find the android_main function and add the following rt_init()/rt_term() calls before and after it:

    rt_init();
    android_main(android_app);
    rt_term();

Clean up and compile as before:

$NDK/ndk-build clean
NDK_TOOLCHAIN_VERSION=clang $NDK/ndk-build V=1

You should see the following linker error, as the linker can't find the rt_init and rt_term functions you just added:

/home/joakim/android-ndk-r10/sources/android/native_app_glue/android_native_app_glue.c:232:
error: undefined reference to 'rt_init'
/home/joakim/android-ndk-r10/sources/android/native_app_glue/android_native_app_glue.c:234:
error: undefined reference to 'rt_term'
clang++: error: linker command failed with exit code 1 (use -v to see invocation)
make: *** [obj/local/x86/libnative-activity.so] Error 1

That's fine, now let's build and link the D source in instead:

../../../dmd/src/dmd -fPIC -I../.. -ofobj/local/x86/objs/native-activity/main.o
-c jni/main.d ../../android/sensor.d

$NDK/toolchains/llvm-3.4/prebuilt/linux-x86/bin/clang
-Wl,-soname,libnative-activity.so -shared
--sysroot=$NDK/platforms/android-9/arch-x86
./obj/local/x86/objs/native-activity/main.o
./obj/local/x86/libandroid_native_app_glue.a -lgcc  -gcc-toolchain 
$NDK/toolchains/x86-4.8/prebuilt/linux-x86 -target i686-none-linux-android
-no-canonical-prefixes  -Wl,--no-undefined -Wl,-z,noexecstack -Wl,-z,relro
-Wl,-z,now  -L$NDK/platforms/android-9/arch-x86/usr/lib -llog -landroid
-lEGL -lGLESv1_CM -llog -lc -lm -fuse-ld=bfd
-L../../../phobos/generated/linux/release/32 -l:libphobos2.a 
-o ./libs/x86/libnative-activity.so

android/sensor.d is included in the first command to work around a dmd bug with unions declared in a separate file. Note the final linker command is a lightly modified version of the one issued when linking the C shared library above, with phobos added to the mix. -fuse-ld=bfd was added to force the use of the ld.bfd linker instead of the gold linker, because ld.bfd works better with the Android TLS patch.

Finally, you can build an apk and install to the Android/x86 VM as before:

ant debug
$SDK/platform-tools/adb uninstall com.example.native_activity
$SDK/platform-tools/adb install bin/NativeActivity-debug.apk
$SDK/platform-tools/adb logcat native-activity *:S

Run the app as before and you should see the same results with the D version.

Build a command-line executable

This might be useful if you want to run your unit tests on the command-line, as I'll show with the druntime/phobos unit tests later.

Let's try building one of the sample files that come with dmd:

../../../dmd/src/dmd -fPIC -c ../../../dmd/samples/sieve.d

$NDK/toolchains/llvm-3.3/prebuilt/linux-x86/bin/clang
-Wl,-z,nocopyreloc --sysroot=$NDK/platforms/android-9/arch-x86 -lgcc 
-gcc-toolchain $NDK/toolchains/x86-4.8/prebuilt/linux-x86 -target
i686-none-linux-android -no-canonical-prefixes -fuse-ld=bfd 
-Wl,--no-undefined -Wl,-z,noexecstack -Wl,-z,relro -Wl,-z,now
-Wl,--export-dynamic -lc -lm sieve.o
-L../../../phobos/generated/linux/release/32 -l:libphobos2.a -o ./sieve

The linker command was extracted by building the sample test-libstdc++ executable from the NDK and removing a handful of flags that weren't needed.

Now we can push it to Android: there are some quirks however. Android doesn't allow you to run executables from the /sdcard partition, but you can only push apps to the sdcard. I work around these restrictions by pushing to the sdcard and then copying it as root to the /data partition. There are other ways to fix this, including remounting the partitions differently, which I haven't messed with.

$SDK/platform-tools/adb push sieve /sdcard/

Switch to the Android VM and hit Alt-F1 to get a root shell:

mkdir /data/test
cd /data/test
cp /sdcard/sieve .
chmod 755 sieve
./sieve

You'll see the following output if everything worked:

10 iterations
1899 primes

Run the druntime and phobos unit tests

Go back to the linux host and build the druntime and phobos unit tests:

cd ../../../druntime
make -f posix.mak unittest PIC=1

If everything went right, you should see the following error, as the tests were compiled but won't be run on the linux host:

make: obj/32/test_runner: Command not found
posix.mak:223: recipe for target 'obj/32/object_' failed
make: *** [obj/32/object_] Error 127

The patch for phobos disables one unit test, doesn't compile std.net.curl because it's not on Android, and makes a few function call and formatting changes because "long double" and "double" are both 64-bit on Android:

cd ../phobos
make -f posix.mak unittest PIC=1

Again, it will error out as the resulting test_runner binary won't run on the linux host:

make: generated/linux/release/32/unittest/test_runner: Command not found
posix.mak:343: recipe for target 'unittest/std/algorithm.d' failed
make: *** [unittest/std/algorithm.d] Error 127

Download this list of tests to run and push both to the Android VM:

$SDK/platform-tools/adb push test_runner /sdcard/
$SDK/platform-tools/adb push test-list.txt /sdcard/

Switch to the root shell in the Android VM and run the tests:

cp /sdcard/test_runner /sdcard/test-list.txt .
chmod 755 test_runner
source test-list.txt > tests.log

I get this output when the tests are run. All tests pass but std.datetime.

Directions for future work

• Two tests fail when the druntime/phobos unit tests are run from inside an apk as opposed to the command-line, core.thread and std.parallelism. I believe these are related to the C wrapper android_native_app_glue and the thread it creates, still looking into it.

• Now that x86 is mostly done, I'll move to ldc and start getting ARM working. It will require a patch to llvm to support the same TLS scheme as I used for Android with dmd.

• You may notice that I added a main function in the D translation of the C sample app, that's a hack to build a shared library. Some of the work in druntime's rt.sections_linux can probably be integrated with Android to get rid of that.

• The android_native_app_glue C wrapper can probably be replaced by D startup code, though a bit of C might always be necessary.

• Now that we can write D code for Android, it might make sense to integrate the D compilers with the Android build scripts.

• It might make sense to try running the dmd test suite on Android/x86. Since dmd's tests depend on phobos, I couldn't run them till I got phobos working.