Porting of cURL to Android OS using NDK (from The Software Rogue)
Porting of cURL to Android OS using NDK
In continuing my journey into Android territory, I decided it would be useful to understand the NDK development kit. Given I want to transfer some files and possibly do a couple of other projects requiring low level work, I selected the cURL kit to port. I've used cURL for a number of projects and it is my hope it runs well under Android. The project, curltest, was run under Android 2.1 but it should work for other versions as well. Given that most Android phones will be getting upgrades rolled out over the next couple of months to version 2.1, seems like a reasonable target. I will be added HTTPS capability as soon as I port openssl, for now, this is strictly unsecured transfers.
I did see some examples of partial porting and even in the curl library source there is an Android.mk which describes how to set it up although not in enough detail. Both work helpful but didn't actually get me to the holy land of a usable test applications. (http://www.mail-archive.com/curl-library@cool.haxx.se/msg03433.html was helpful but didn't work for me.)
I found it worth the exercise of downloading the full android sources and building them. Look at the build process, the bionic library, and well as whats in external. The external directory is where you would add repositories to do things like run configure, etc
You won't actually need Linux since I have included the config file here but in case you want to follow way it was created, I've included it.
Summary of the steps to take for this port:
- Lots of searches and view the current state of curl porting
- Setup source repository (I used Linux, you could try windows using cygwin)
- Create curl_config.h under the full android source tree
- Untar curltest.tar.gz under $NDKROOT or Create NDK JNI App on Windows
- Cross-link or add to eclipse project to build Java front-end
- Unpack cURL into $NDKROOT/apps/curltest/jni/curl
- Copy curltest/jni/curlbuild.h to curltest/jni/curl/include/curl
- Copy curltest/jni/curl_config.h to curltest/jni/curl/lib
- Build and test
Run the script in the top curl directory (where configure resides) and it should go through many tests before ultimately creating the lib/curl_config.h file.
Create library name under $NDKROOT/apps/, I used curltest.
CD to curltest and create the Application.mk file.
APP_PROJECT_PATH := $(call my-dir)/project
APP_MODULES := curltest libcurl
Once that is done, cd to project and create your standard Java AndroidManifest.xml file. If you created an App from Eclipse you'll need to modify this file. Make sure to give it INTERNET permissions. The rest depends on what's in your Java code.
Then cd to jni and create the Android.mk file. This is where the one from cURL would go. However, I didn't use it as anything more than a reference.
Now for the actual JNI interfacing. There are lots of complaints here because everything needs to be specified up front and you need different packages for different hardware. Except for some dynamic registration mechanism, I'm not sure you can get away with writing to hardware AND getting independence from it. Actually, I am sure, you can't.
Here is my curltest.c interface:
typedef struct pageInfo_t {
char *data;
int len;
} pageInfo_t;
static size_t
HTTPData(void *buffer, size_t size, size_t nmemb, void *userData) {
int len = size * nmemb;
pageInfo_t *page = (pageInfo_t *)userData;
if (buffer && page->data && (page->len + len < (16 * 1024)) ) { memcpy(&page->data[page->len], buffer, len);
page->len += len;
}
return len;
}
//Interface function that will receive web page from Java
jstring
Java_com_mtterra_curltest_curltest_JNIGetWebpage( JNIEnv* env,
jobject entryObject,
jstring webpageJStr )
{
pageInfo_t page;
CURL *curl;
CURLcode res;
char *buffer;
const jbyte *webpage;
webpage = (*env)->GetStringUTFChars(env, webpageJStr, NULL);
if (webpage == NULL) {
return NULL; /* OutOfMemoryError already thrown */
}
page.data = (char *)malloc(16 * 1024);
page.len = 0;
if (page.data)
memset(page.data, 32, 16 * 1024);
buffer = (char *)malloc(1024);
curl = curl_easy_init();
if(curl) {
curl_easy_setopt(curl, CURLOPT_URL, webpage);
curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, HTTPData);
curl_easy_setopt(curl, CURLOPT_WRITEDATA, &page);
res = curl_easy_perform(curl);
/* always cleanup */
curl_easy_cleanup(curl);
(*env)->ReleaseStringUTFChars(env, webpageJStr, webpage);
if(res == 0) {
if (buffer) {
page.data[page.len < 256 ? page.len : 256] = '\0'; sprintf(buffer, "pagedata(%d): %s. done.\n", page.len, page.data); return (*env)->NewStringUTF(env, buffer);
}
}
sprintf(buffer, "Result %d", res);
return (*env)->NewStringUTF(env, buffer);
} else {
return (*env)->NewStringUTF(env, "Unable to init cURL");
}
}
Finally, it's time to setup Eclipse to build. There are some assumptions here, like having a functioning Eclipse build environment. In order to get your environment over to Eclipse, you can use File->Import then select 'Existing Projects into Workspace' function.
Unpack cURL into $NDKROOT/apps/curltest/jni/curl. Move the curl_config.h file created above to $NDKROOT/apps/curltest/jni/curl/lib.
Finally, we create the Android.mk file in curltest/jni and build it. The Android.mk use is as follows:
Going back to your $NDKROOT you should be able to build the library with 'make APP=curltest -B V=1'. The -B means rebuilt it all and V=1 makes it verbose.
After this, you should be able to build the application. When I get back to Eclipse I'll hit F5 to refresh the files as a habit although likely not necessary. Another interesting thing is that some people have indicated that you need to do an 'adb push' to the emulator so the library is there. I haven't found this is the case. Basically, if it doesn't find your library the interface name or some other part is broken and you need to find the issue.
The finished result looks something like:
Here is the NDK project files used. However, you'll have to add the curl sources to this.
The Software Rogue - http://thesoftwarerogue.blogspot.com
I did see some examples of partial porting and even in the curl library source there is an Android.mk which describes how to set it up although not in enough detail. Both work helpful but didn't actually get me to the holy land of a usable test applications. (http://www.mail-archive.com/curl-library@cool.haxx.se/msg03433.html was helpful but didn't work for me.)
I found it worth the exercise of downloading the full android sources and building them. Look at the build process, the bionic library, and well as whats in external. The external directory is where you would add repositories to do things like run configure, etc
You won't actually need Linux since I have included the config file here but in case you want to follow way it was created, I've included it.
Summary of the steps to take for this port:
- Lots of searches and view the current state of curl porting
- Setup source repository (I used Linux, you could try windows using cygwin)
- Create curl_config.h under the full android source tree
- Untar curltest.tar.gz under $NDKROOT or Create NDK JNI App on Windows
- Cross-link or add to eclipse project to build Java front-end
- Unpack cURL into $NDKROOT/apps/curltest/jni/curl
- Copy curltest/jni/curlbuild.h to curltest/jni/curl/include/curl
- Copy curltest/jni/curl_config.h to curltest/jni/curl/lib
- Build and test
Setup source repository on Linux
It's one thing to describe doing X, quite another to actually do it. Anyway, I ended up creating my own android repository to start the process. While I was using Windows 7 for my NDK development, I switched to Linux to create the curl_config.h file needed in the build. What this means it that you run a couple of scripts. First up, repo. Created a directory and cd to it. Download the repo script 'wget https://android.git.kernel.org/repo'. Move repo script to somewhere in your path and issue 'repo init -u git://android.git.kernel.org/platform/manifest.git' then after run 'repo sync'. These two should get you all the goodies. I had some stalls where I had to restart the sync process but after a long while it completed successfully.Create curl_config.h under the full android source tree
Next, I created a script to call the configure command to generated the curl_config.h. CD to $ANDROID_ROOT/external. Untar curl libraries, I renamed my from curl-7.20.1 to just curl. cURL has an Android.mk inside of it. In the file were some instructions for generating the curl_config.h. I found them helpful as a start but ultimately I had to fiddle with a number of things to get this to actually work. Here is the script I used:#!/bin/sh
ANDROID_ROOT="$HOME/android_src" && \
TOOLCHAIN_VER="4.4.0" \
PLATFORM_VER="5" \
CROSS_COMPILE=arm-eabi- \
PATH=$ANDROID_ROOT/prebuilt/linux-x86/toolchain/arm-eabi-$TOOLCHAIN_VER/bin:$PATH && \
CPPFLAGS="-I $ANDROID_ROOT/system/core/include -I$ANDROID_ROOT/bionic/libc/include -I$ANDROID_ROOT/ndk/build/platforms/android-5/arch-arm/usr/include -I$ANDROID_ROOT/bionic/libc/kernel/arch-arm -L $ANDROID_ROOT/prebuilt/linux-x86/toolchain/arm-eabi-$TOOLCHAIN_VER/lib/gcc/arm-eabi/$TOOLCHAIN_VER/interwork -L$ANDROID_ROOT/ndk/build/platforms/android-$PLATFORM_VER/arch-arm/usr/lib -L$ANDROID_ROOT/out/target/product/generic/system/lib " \
CFLAGS="-fno-exceptions -Wno-multichar -mthumb -mthumb-interwork -nostdlib -lc -ldl -lm " \
./configure CC=arm-eabi-gcc --host=arm-linux --disable-tftp --disable-sspi --disable-ipv6 --disable-ldaps --disable-ldap --disable-telnet --disable-pop3 --disable-ftp --without-ssl --disable-imap --disable-smtp --disable-pop3 --disable-rtsp --disable-ares --without-ca-bundle --disable-warnings --disable-manual --without-nss --enable-shared --without-zlib --without-random
# openssl/zlib version
#./configure CC=arm-eabi-gcc --host=arm-linux --disable-tftp --disable-sspi --disable-ipv6 --disable-ldaps --disable-ldap --disable-telnet --disable-pop3 --disable-ftp --with-ssl=$ANDROID_ROOT/external/openssl --disable-imap --disable-smtp --disable-pop3 --disable-rtsp --disable-ares --without-ca-bundle --disable-warnings --disable-manual --without-nss --enable-shared --with-zlib=$ANDROID_ROOT/external/zlib --without-random
ANDROID_ROOT="$HOME/android_src" && \
TOOLCHAIN_VER="4.4.0" \
PLATFORM_VER="5" \
CROSS_COMPILE=arm-eabi- \
PATH=$ANDROID_ROOT/prebuilt/linux-x86/toolchain/arm-eabi-$TOOLCHAIN_VER/bin:$PATH && \
CPPFLAGS="-I $ANDROID_ROOT/system/core/include -I$ANDROID_ROOT/bionic/libc/include -I$ANDROID_ROOT/ndk/build/platforms/android-5/arch-arm/usr/include -I$ANDROID_ROOT/bionic/libc/kernel/arch-arm -L $ANDROID_ROOT/prebuilt/linux-x86/toolchain/arm-eabi-$TOOLCHAIN_VER/lib/gcc/arm-eabi/$TOOLCHAIN_VER/interwork -L$ANDROID_ROOT/ndk/build/platforms/android-$PLATFORM_VER/arch-arm/usr/lib -L$ANDROID_ROOT/out/target/product/generic/system/lib " \
CFLAGS="-fno-exceptions -Wno-multichar -mthumb -mthumb-interwork -nostdlib -lc -ldl -lm " \
./configure CC=arm-eabi-gcc --host=arm-linux --disable-tftp --disable-sspi --disable-ipv6 --disable-ldaps --disable-ldap --disable-telnet --disable-pop3 --disable-ftp --without-ssl --disable-imap --disable-smtp --disable-pop3 --disable-rtsp --disable-ares --without-ca-bundle --disable-warnings --disable-manual --without-nss --enable-shared --without-zlib --without-random
# openssl/zlib version
#./configure CC=arm-eabi-gcc --host=arm-linux --disable-tftp --disable-sspi --disable-ipv6 --disable-ldaps --disable-ldap --disable-telnet --disable-pop3 --disable-ftp --with-ssl=$ANDROID_ROOT/external/openssl --disable-imap --disable-smtp --disable-pop3 --disable-rtsp --disable-ares --without-ca-bundle --disable-warnings --disable-manual --without-nss --enable-shared --with-zlib=$ANDROID_ROOT/external/zlib --without-random
Run the script in the top curl directory (where configure resides) and it should go through many tests before ultimately creating the lib/curl_config.h file.
Create NDK JNI App on Windows
Untar curltest.tar.gz under $NDKROOT. This has the Java side application already created. Or you can simply create a new Application under Eclipse which will generated the src & res directories that will need to be modified. I'm not going into App generation except that you will target $NDKROOT/apps/curltest/project as your directory tree.Create library name under $NDKROOT/apps/, I used curltest.
CD to curltest and create the Application.mk file.
APP_PROJECT_PATH := $(call my-dir)/project
APP_MODULES := curltest libcurl
Once that is done, cd to project and create your standard Java AndroidManifest.xml file. If you created an App from Eclipse you'll need to modify this file. Make sure to give it INTERNET permissions. The rest depends on what's in your Java code.
<?xml version="1.0" encoding="utf-8"?>
<manifest xmlns:android="/apk/res/android"
android:versionCode="1"
android:versionName="1.0" package="com.mtterra.curltest">
<application android:icon="@drawable/icon" android:label="@string/app_name">
<activity android:name=".curltest"
android:label="@string/app_name">
<intent-filter>
<action android:name="android.intent.action.MAIN" />
<category android:name="android.intent.category.LAUNCHER" />
</intent-filter>
</activity>
</application>
<uses-sdk android:minSdkVersion="5"/>
<uses-permission android:name="android.permission.INTERNET"/>
</manifest>
<manifest xmlns:android="/apk/res/android"
android:versionCode="1"
android:versionName="1.0" package="com.mtterra.curltest">
<application android:icon="@drawable/icon" android:label="@string/app_name">
<activity android:name=".curltest"
android:label="@string/app_name">
<intent-filter>
<action android:name="android.intent.action.MAIN" />
<category android:name="android.intent.category.LAUNCHER" />
</intent-filter>
</activity>
</application>
<uses-sdk android:minSdkVersion="5"/>
<uses-permission android:name="android.permission.INTERNET"/>
</manifest>
Then cd to jni and create the Android.mk file. This is where the one from cURL would go. However, I didn't use it as anything more than a reference.
Now for the actual JNI interfacing. There are lots of complaints here because everything needs to be specified up front and you need different packages for different hardware. Except for some dynamic registration mechanism, I'm not sure you can get away with writing to hardware AND getting independence from it. Actually, I am sure, you can't.
Here is my curltest.c interface:
typedef struct pageInfo_t {
char *data;
int len;
} pageInfo_t;
static size_t
HTTPData(void *buffer, size_t size, size_t nmemb, void *userData) {
int len = size * nmemb;
pageInfo_t *page = (pageInfo_t *)userData;
if (buffer && page->data && (page->len + len < (16 * 1024)) ) { memcpy(&page->data[page->len], buffer, len);
page->len += len;
}
return len;
}
//Interface function that will receive web page from Java
jstring
Java_com_mtterra_curltest_curltest_JNIGetWebpage( JNIEnv* env,
jobject entryObject,
jstring webpageJStr )
{
pageInfo_t page;
CURL *curl;
CURLcode res;
char *buffer;
const jbyte *webpage;
webpage = (*env)->GetStringUTFChars(env, webpageJStr, NULL);
if (webpage == NULL) {
return NULL; /* OutOfMemoryError already thrown */
}
page.data = (char *)malloc(16 * 1024);
page.len = 0;
if (page.data)
memset(page.data, 32, 16 * 1024);
buffer = (char *)malloc(1024);
curl = curl_easy_init();
if(curl) {
curl_easy_setopt(curl, CURLOPT_URL, webpage);
curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, HTTPData);
curl_easy_setopt(curl, CURLOPT_WRITEDATA, &page);
res = curl_easy_perform(curl);
/* always cleanup */
curl_easy_cleanup(curl);
(*env)->ReleaseStringUTFChars(env, webpageJStr, webpage);
if(res == 0) {
if (buffer) {
page.data[page.len < 256 ? page.len : 256] = '\0'; sprintf(buffer, "pagedata(%d): %s. done.\n", page.len, page.data); return (*env)->NewStringUTF(env, buffer);
}
}
sprintf(buffer, "Result %d", res);
return (*env)->NewStringUTF(env, buffer);
} else {
return (*env)->NewStringUTF(env, "Unable to init cURL");
}
}
Finally, it's time to setup Eclipse to build. There are some assumptions here, like having a functioning Eclipse build environment. In order to get your environment over to Eclipse, you can use File->Import then select 'Existing Projects into Workspace' function.
Unpack cURL into $NDKROOT/apps/curltest/jni/curl. Move the curl_config.h file created above to $NDKROOT/apps/curltest/jni/curl/lib.
Finally, we create the Android.mk file in curltest/jni and build it. The Android.mk use is as follows:
LOCAL_PATH:= $(call my-dir)
CFLAGS := -Wpointer-arith -Wwrite-strings -Wunused -Winline \
-Wnested-externs -Wmissing-declarations -Wmissing-prototypes -Wno-long-long \
-Wfloat-equal -Wno-multichar -Wsign-compare -Wno-format-nonliteral \
-Wendif-labels -Wstrict-prototypes -Wdeclaration-after-statement \
-Wno-system-headers -DHAVE_CONFIG_H
include $(CLEAR_VARS)
include $(LOCAL_PATH)/curl/lib/Makefile.inc
LOCAL_SRC_FILES := $(addprefix curl/lib/,$(CSOURCES))
LOCAL_CFLAGS += $(CFLAGS)
LOCAL_C_INCLUDES += $(LOCAL_PATH)/curl/include/
LOCAL_COPY_HEADERS_TO := libcurl
LOCAL_COPY_HEADERS := $(addprefix curl/include/curl/,$(HHEADERS))
LOCAL_MODULE:= libcurl
include $(BUILD_STATIC_LIBRARY)
# Build shared library now
# curltest
include $(CLEAR_VARS)
LOCAL_MODULE := curltest
LOCAL_SRC_FILES := curltest.c
LOCAL_STATIC_LIBRARIES := libcurl
LOCAL_C_INCLUDES += $(LOCAL_PATH)/curl/include
include $(BUILD_SHARED_LIBRARY)
CFLAGS := -Wpointer-arith -Wwrite-strings -Wunused -Winline \
-Wnested-externs -Wmissing-declarations -Wmissing-prototypes -Wno-long-long \
-Wfloat-equal -Wno-multichar -Wsign-compare -Wno-format-nonliteral \
-Wendif-labels -Wstrict-prototypes -Wdeclaration-after-statement \
-Wno-system-headers -DHAVE_CONFIG_H
include $(CLEAR_VARS)
include $(LOCAL_PATH)/curl/lib/Makefile.inc
LOCAL_SRC_FILES := $(addprefix curl/lib/,$(CSOURCES))
LOCAL_CFLAGS += $(CFLAGS)
LOCAL_C_INCLUDES += $(LOCAL_PATH)/curl/include/
LOCAL_COPY_HEADERS_TO := libcurl
LOCAL_COPY_HEADERS := $(addprefix curl/include/curl/,$(HHEADERS))
LOCAL_MODULE:= libcurl
include $(BUILD_STATIC_LIBRARY)
# Build shared library now
# curltest
include $(CLEAR_VARS)
LOCAL_MODULE := curltest
LOCAL_SRC_FILES := curltest.c
LOCAL_STATIC_LIBRARIES := libcurl
LOCAL_C_INCLUDES += $(LOCAL_PATH)/curl/include
include $(BUILD_SHARED_LIBRARY)
Going back to your $NDKROOT you should be able to build the library with 'make APP=curltest -B V=1'. The -B means rebuilt it all and V=1 makes it verbose.
After this, you should be able to build the application. When I get back to Eclipse I'll hit F5 to refresh the files as a habit although likely not necessary. Another interesting thing is that some people have indicated that you need to do an 'adb push' to the emulator so the library is there. I haven't found this is the case. Basically, if it doesn't find your library the interface name or some other part is broken and you need to find the issue.
The finished result looks something like:
Here is the NDK project files used. However, you'll have to add the curl sources to this.
The Software Rogue - http://thesoftwarerogue.blogspot.com
