Build DMD for Android
Android/x86 mostly runs, going to try Android/ARM next with ldc. Almost all the druntime/phobos unit tests pass on Android/x86, plus I ported a C/OpenGLES 1.0 purely native sample app from the NDK to D. I list work that still needs to be done at the end.
While Android/x86 is not as useful as Android/ARM, it can be used as a testbed to start porting apps to Android and try the platform out, in preparation for Android/ARM.
- 32-bit linux/x86 host on which to build dmd
- A virtual machine like VirtualBox/VMware will do fine, but you should have at least 512 MB of memory allocated and 1 GB of swap, particularly if building the phobos unit tests, and 10 GB of disk space.
- C++ compiler and toolchain, to build dmd
- dmd/druntime/phobos source
- Get druntime and phobos from git, as they have Android-specific changes that are not in the release branches yet.
- Android native toolchain, the 32-bit NDK and SDK
- The "SDK Tools only" version of the SDK is enough, if you don't plan on using their IDE integration. I will only write about using the command-line tools. The SDK requires JDK 6 and Ant 1.8 or later, follow their instructions to make sure it's installed right.
- Android/x86, whether a device or VM
- The SDK comes with an emulator and Intel puts out their own. I use the builds put out by the android-x86 project and recommend the deprecated 4.3 build, as it's the least buggy one I've dealt with. I run the iso in a VM with 512 MBs of memory without installing to disk.
To run Android/x86 in a VM
Download the Android-x86 iso and install as normal. It'll put you through a configuration process, but you can skip every step. Go to Settings->Security and disable the Verify Apps option, or it'll ask you about that every time you install a test app. Alt-F1 will take you to a root shell, which you'll want to do any time you're not using Android/x86 for a couple minutes or the screen will eventually go black and the VM will freeze up. Alt-F7 takes you back to the normal GUI screen.
Build D for Android/x86
cd dmd git apply packed_tls_for_elf.patch make -f posix.mak -j5
This patch adds TLS support for Android/x86, more info can be found on the pull request.
Assuming druntime and phobos are in the same directory as dmd, download and apply the patch for druntime and the patch for phobos, set the NDK environment variable to the path of wherever you installed the NDK, and build each in turn with PIC enabled:
cd ../druntime export NDK=/path/to/your/android-ndk-r10 git apply druntime_build.patch make -f posix.mak PIC=1 cd ../phobos git apply phobos_build.patch make -f posix.mak PIC=1
Both patches avoid building druntime/phobos as a shared library and set the appropriate C compiler path and flags. The druntime patch also adds an import I missed before. The remaining changes are for the unit tests and are addressed below.
Build an Android sample app
I've put up an android repository which contains several translated headers and a C sample app translated to D. Clone it into the same directory as dmd/druntime/phobos:
cd .. git clone https://github.com/joakim-noah/android.git
Default build of the C sample app
First, let's see how it's done by default by compiling the C version. Go to the sample native-activity app in my android repo, which is almost copied verbatim from the NDK/samples. The only differences are that it also contains a translation of the given jni/main.c source to jni/main.d, changes one line in jni/Application.mk to build for x86 by default, and includes a dmd.conf for building with dmd.
Run the following commands to compile the C source into a debug app that you can install on Android, these commands are taken from the NDK instructions (scroll down to "Exploring the native-activity Sample Application"):
cd android/samples/native-activity NDK_TOOLCHAIN_VERSION=clang $NDK/ndk-build V=1
Looking at the output, it compiles jni/main.c, a small wrapper library called android_native_app_glue, and links everything together into a shared library. Let's look at the command that compiles jni/main.c:
/home/joakim/android-ndk-r10/toolchains/llvm-3.4/prebuilt/linux-x86/bin/clang -MMD -MP -MF ./obj/local/x86/objs/native-activity/main.o.d -gcc-toolchain /home/joakim/android-ndk-r10/toolchains/x86-4.8/prebuilt/linux-x86 -target i686-none-linux-android -ffunction-sections -funwind-tables -fstack-protector -fPIC -no-canonical-prefixes -O2 -g -DNDEBUG -fomit-frame-pointer -fstrict-aliasing -I/home/joakim/android-ndk-r10/sources/android/native_app_glue -Ijni -DANDROID -Wa,--noexecstack -Wformat -Werror=format-security -I/home/joakim/android-ndk-r10/platforms/android-9/arch-x86/usr/include -c jni/main.c -o ./obj/local/x86/objs/native-activity/main.o
This is where I extracted the C compiler path and flags for the previous druntime/phobos patches, leaving out the dependency file (-MMD -MP -MF).
This command links the shared library that gets packaged into the native app:
/home/joakim/android-ndk-r10/toolchains/llvm-3.4/prebuilt/linux-x86/bin/clang++ -Wl,-soname,libnative-activity.so -shared --sysroot=/home/joakim/android-ndk-r10/platforms/android-9/arch-x86 ./obj/local/x86/objs/native-activity/main.o ./obj/local/x86/libandroid_native_app_glue.a -lgcc -gcc-toolchain /home/joakim/android-ndk-r10/toolchains/x86-4.8/prebuilt/linux-x86 -target i686-none-linux-android -no-canonical-prefixes -Wl,--no-undefined -Wl,-z,noexecstack -Wl,-z,relro -Wl,-z,now -L/home/joakim/android-ndk-r10/platforms/android-9/arch-x86/usr/lib -llog -landroid -lEGL -lGLESv1_CM -llog -lc -lm -o ./obj/local/x86/libnative-activity.so
You'll use a modified version of the above command to link your D source later.
Getting back to building the C app, set the SDK environment variable for the path to your SDK and run the commands to package an Android app/apk:
export SDK=/path/to/your/android-sdk-linux $SDK/tools/android update project -p . -s --target 1 ant debug
Now you'll push the final app, NativeActivity-debug.apk, to your Android/x86 environment. Connecting to Android will vary based on which Android/x86 you're using: I'll show you how I do it for Android/x86 installed in a VM, using the first method shown in the Android-x86 docs. Hit Alt-F1 inside the VM to go to the root shell and type "netcfg" to get its IP address: let's say it's 192.168.0.1. Hit Alt-F7 to go back to the UI, then go to Settings->Apps. It should be empty.
Going back to the linux host:
$SDK/platform-tools/adb connect 192.168.0.1:5555 $SDK/platform-tools/adb install bin/NativeActivity-debug.apk $SDK/platform-tools/adb logcat native-activity *:S
The NativeActivity app should show up in the Settings->Apps list in Android/x86. Go to the app launcher and click on NativeActivity to run it. Move the mouse and you should see a bunch of colors continuously flashing on the screen. Looking at the log dump in the linux host, you'll see some numbers from the accelerometer, which are being reported by the app.
Build the translated D app
I've translated jni/main.c to a D version, jni/main.d. First, you'll need to recompile the android_native_app_glue library so that it calls a couple functions necessary for a D shared library. Hit ctrl-c to get out of the Android log and open $NDK/sources/android/native_app_glue/android_native_app_glue.c in an editor. Find the android_main function and add the following rt_init()/rt_term() calls before and after it:
rt_init(); android_main(android_app); rt_term();
Clean up and compile as before:
$NDK/ndk-build clean NDK_TOOLCHAIN_VERSION=clang $NDK/ndk-build V=1
You should see the following linker error, as the linker can't find the rt_init and rt_term functions you just added:
/home/joakim/android-ndk-r10/sources/android/native_app_glue/android_native_app_glue.c:232: error: undefined reference to 'rt_init' /home/joakim/android-ndk-r10/sources/android/native_app_glue/android_native_app_glue.c:234: error: undefined reference to 'rt_term' clang++: error: linker command failed with exit code 1 (use -v to see invocation) make: *** [obj/local/x86/libnative-activity.so] Error 1
That's fine, now let's build and link the D source in instead:
../../../dmd/src/dmd -fPIC -I../.. -ofobj/local/x86/objs/native-activity/main.o -c jni/main.d ../../android/sensor.d $NDK/toolchains/llvm-3.4/prebuilt/linux-x86/bin/clang -Wl,-soname,libnative-activity.so -shared --sysroot=$NDK/platforms/android-9/arch-x86 ./obj/local/x86/objs/native-activity/main.o ./obj/local/x86/libandroid_native_app_glue.a -lgcc -gcc-toolchain $NDK/toolchains/x86-4.8/prebuilt/linux-x86 -target i686-none-linux-android -no-canonical-prefixes -Wl,--no-undefined -Wl,-z,noexecstack -Wl,-z,relro -Wl,-z,now -L$NDK/platforms/android-9/arch-x86/usr/lib -llog -landroid -lEGL -lGLESv1_CM -llog -lc -lm -fuse-ld=bfd -L../../../phobos/generated/linux/release/32 -l:libphobos2.a -o ./libs/x86/libnative-activity.so
android/sensor.d is included in the first command to avoid a dmd bug with unions declared in a separate file. Note the final linker command is a lightly modified version of the one issued when linking the C shared library above, with phobos added to the mix. -fuse-ld=bfd was added to force the use of the ld.bfd linker instead of the gold linker, because ld.bfd works better with the Android TLS patch.
Finally, you can build an apk and install to the Android/x86 VM as before:
ant debug $SDK/platform-tools/adb uninstall com.example.native_activity $SDK/platform-tools/adb install bin/NativeActivity-debug.apk $SDK/platform-tools/adb logcat native-activity *:S
Run the app as before and you should see the same results with the D version.
Build a command-line executable
This might be useful if you want to run your unit tests on the command-line, as I'll show with the druntime/phobos unit tests later.
Let's try building one of the sample files that come with dmd:
../../../dmd/src/dmd -fPIC -c ../../../dmd/samples/sieve.d $NDK/toolchains/llvm-3.3/prebuilt/linux-x86/bin/clang -Wl,-z,nocopyreloc --sysroot=$NDK/platforms/android-9/arch-x86 -lgcc -gcc-toolchain $NDK/toolchains/x86-4.8/prebuilt/linux-x86 -target i686-none-linux-android -no-canonical-prefixes -fuse-ld=bfd -Wl,--no-undefined -Wl,-z,noexecstack -Wl,-z,relro -Wl,-z,now -Wl,--export-dynamic -lc -lm sieve.o -L../../../phobos/generated/linux/release/32 -l:libphobos2.a -o ./sieve
The linker command was extracted by building the sample test-libstdc++ executable from the NDK and removing a handful of flags that weren't needed.
Now we can push it to Android: there are some quirks however. Android doesn't allow you to run executables from the /sdcard partition, but you can only push files to the sdcard. I route around these restrictions by pushing to the sdcard and then copying the executable as root to the /data partition. There are other ways to fix this, including remounting the partitions differently, which I haven't messed with.
$SDK/platform-tools/adb push sieve /sdcard/
Switch to the Android VM and hit Alt-F1 to get a root shell:
mkdir /data/test cd /data/test cp /sdcard/sieve . chmod 755 sieve ./sieve
You'll see the following output if everything went well:
10 iterations 1899 primes
Run the druntime and phobos unit tests
Go back to the linux host and build the druntime and phobos unit tests:
cd ../../../druntime make -f posix.mak unittest PIC=1
If everything went right, you should see the following error, as the tests were compiled but won't be run on the linux host:
make: obj/32/test_runner: Command not found posix.mak:223: recipe for target 'obj/32/object_' failed make: *** [obj/32/object_] Error 127
The patch for phobos disables one unit test in std.process, doesn't compile std.net.curl because it's not on Android, and makes a few function call and formatting changes because "long double" and "double" are both 64-bit on Android:
cd ../phobos make -f posix.mak unittest PIC=1
Again, it will error out as the resulting test_runner binary won't run on the linux host:
make: generated/linux/release/32/unittest/test_runner: Command not found posix.mak:343: recipe for target 'unittest/std/algorithm.d' failed make: *** [unittest/std/algorithm.d] Error 127
Download this list of tests to run and push both to the Android VM:
$SDK/platform-tools/adb push test_runner /sdcard/ $SDK/platform-tools/adb push test-list.txt /sdcard/
Switch to the root shell in the Android VM and run the tests:
cp /sdcard/test_runner /sdcard/test-list.txt . chmod 755 test_runner source test-list.txt > tests.log
I get this output when the tests are run. All tests pass but std.datetime.
Directions for future work
- Two modules, core.thread and std.parallelism, have tests that cause the test runner to hang when the druntime/phobos unit tests are run from inside an apk as opposed to on the command line. These failures are related to the C wrapper android_native_app_glue and the thread it creates, as they only happen when thread_suspendAll() is called, either directly in one test in core.thread or indirectly when the GC runs a fullcollect on the other hanging tests. Apparently it's incorrectly suspending one of the main threads, I'll look into a workaround.
- Now that x86 is mostly done, I'll move to ldc and start getting ARM working. It will require a patch to llvm to support the same TLS scheme as I used with dmd.
- You may notice that I added an empty main function in the D translation of the C sample app: that's a hack to build a shared library. Some of the linux shared library support in druntime's rt.sections_linux may be integrated with Android to get rid of that.
- The android_native_app_glue C wrapper can be mostly replaced by D startup code, though a bit of C will always be necessary.
- Now that we can write D code for Android, it'll make building easier if the D compilers are integrated with the Android build scripts, to compile D apps with the ndk-build command.
- I'd like to try running the dmd test suite on Android/x86. Since dmd's tests depend on phobos, I couldn't run them till I got phobos passing its tests.