GUADEC: porting GNOME to Android
https://lwn.net/Articles/510465/
By Nathan Willis
August 15, 2012
Rightly or wrongly, GNOME is regarded by many casual users as "a Linuxdesktop environment." It also runs on various BSD and Solarisflavors, of course, and there has been considerable discussionrecently of developing a "GNOME OS" deliverable suitable for tablets andother devices. But at GUADEC2012, there was yet another spin on the redefinition of GNOME: portingseveral underlying GNOME frameworks to Android.
A team of developers from Collabora hosted the relevant session,which was titled D-Bus, PulseAudio, GStreamer and Telepathy Inthe Palm of your Hand. Reynaldo Verdejo Pinochet acted asringmaster for the other speakers, and provided an introduction to theoverall effort. As many others have observed, GNOME and other opensource projects are facing a "form-factor challenge," he said.Desktop approaches to user interfaces do not work as well onsmall-screen devices, and mobile device users exhibit different usagepatterns: rapidly interleaving multiple tasks rather than camping outin one application.
The open source developer community is currently failing to provide abridge from the desktop to mobile devices, he said. Meanwhile, thesystem-on-chip vendors are turning their full attention to Android— a platform into whose development the community has no say.Collabora's solution, he said, is not to "fix" Android or to "fix" theusers, but instead to enable the GNOME project's technology to run onAndroid. As a result, developers and users will have more choice, andthe GNOME components will continue to be relevant to both.
The bulk of the session was then taken up by a series of short statusupdates from Collabora developers working on "smallifying"frameworks used by GNOME applications. Covered were Android ports ofPulseAudio, Wayland, Telepathy, and GStreamer. Despite the title,D-Bus was not discussed separately in the talk; Verdejo said in anemail that the D-Bus port was undertaken as part of the Telepathyeffort. Verdejo closed out the session with general information forapplication developers looking to port GNOME applications toAndroid.
Porting stories
Arun Raghavan spoke first, explaining the PulseAudio port for Android.Android's native audio subsystem is called AudioFlinger (AF). It is asoftware mixer like PulseAudio, he explained, but it pales incomparison to PulseAudio for features. "It's not as nice asPulseAudio, and you can look at code if you don't believe me."In particular, PulseAudio provides flexible audio routing (includingnetwork routing), dynamically-adjustable latency (which preventsglitches, saves power, and simplifies the work required fromapplications), and support for passing compressed audio (in contrastto AF, which provides only PCM audio transport).
Although the project's original goal was to install PulseAudio in parallel with AF,Raghavan said, that quickly proved impossible due to how integrated AFis into Android. So instead, Raghavan focused on building andinstalling PulseAudio as a complete replacement for AF, using the Galaxy Nexus phone asthe target platform. First, he successfully compiled PulseAudio andgot it to run and play sound on the Galaxy Nexus. He then startedwork on using PulseAudio to emulate AF's playback API "AudioTrack," atask that he said is almost complete. Still in progress isemulation of AF's policy framework (which controls volumesettings, when playback is allowed, and so forth). After that, thenext item on the to-do list is "AudioRecord," the AF audio captureAPI.
Next, Daniel Stone presented the Wayland port. Once again, the goalwas to offer Wayland as an alternative to a built-in Android service— in this case, SurfaceFlinger. SurfaceFlinger provides buffertransport functions analogous to Wayland, and composition functionslike those of Wayland's reference compositor Weston. Collabora's portis mostly complete, and is currently able to run Wayland demos. Stillin progress is integrating Wayland into the Android stack so that itcan run side-by-side with SurfaceFlinger, a challenge that includesboth running Wayland and SurfaceFlinger applications at once, andhandling clipboard events, task switching, and other functions. Stillto come is styling, which would make Wayland applications seamlesslymatch the SurfaceFlinger look.
The Wayland work ran into the usual porting problems, he said, butthere were special difficulties as well. First, Wayland relies onseveral kernel features too new to be found in the Android kernel(such as timerfd and signalfd). Second, the Android graphics driversare "terrible" at buffer management in comparison to thedesktop. Whereas Mesa provides full EGLImagesharing through the Direct Rendering Manager (DRM) and seamlesszero-copy transfers (which Wayland can take advantage of through thewl_drm extension), EGLImage is an "absolute crapshoot" onAndroid. The quality of the implementation varies drasticallybetween hardware vendors, and the graphics drivers are almost alwaysclosed.
Alvaro Soliverez presented the port of the Telepathy communicationframework. Telepathy is modular, supporting a long list of pluggableback-ends (from XMPP to SIP, to proprietary networks like MSN) andbindings for a number of programming languages and toolkits.Soliverez's port picked the subset most useful for Android: theGabble XMPP module, link-local XMPP with Salut (using Avahi servicediscovery), and Java bindings.
The port is fully functional, he said, and the patches have been sentupstream to Telepathy. Android users can download and install theTelepathy service as a stand-alone application. In addition to theprotocols mentioned above, the account management and statusmanagement functionality works, and the GLib bindings are in place fordevelopers. Ports of the rest of the framework are underway, he said,although assistance would be welcome.
Verdejo discussed the GStreamerport. GStreamer is already successfully in use on otherplatforms, where it can serve as a bridge to the operating system'snative multimedia framework (such as QuickTime on Mac OS X orDirectShow on Windows). The GStreamer Android port is intended to bea complete port of the framework, putting Android on par with theother platforms. This required the development of two new elements tointerface to Android's media layer: GSTSurfaceFlingerSink andGSTAudioFlingerSink. These are sink elements, which allowapplications' GStreamer pipelines to play their output on Android.
But the Android platform "is crippled on purpose," hesaid, offering no real native development story. Instead, all ofthe emphasis is placed on writing Dalvik applications. Consequently,Collabora decided to build GStreamer as a system-level service aswell. It developed two other elements, GSTPlayer andGSTMetaDataRetriver, which can function as replacements for Android'sbuilt-in media playback functionality. Up next are new elements tohandle direct multimedia output over OpenGL ES and OpenSL ES, andcapture elements for camera and microphone input. The OpenGL ES andOpenSL ES elements will eventually replace the existing AndroidGStreamer sinks so that GStreamer pipelines will not need to depend onthe Android API layer. All of the GStreamer work is being pushedupstream and will be included in the GStreamer SDK.
Developing applications
Verdejo Raghavan closed out the session with general advice for otherdevelopers interested in porting their software to Android. There aretwo general approaches to consider, he said: the native developmentkit (NDK) and the complete system-integration route. The NDK isGoogle's officialoffering, and it places some requirements on the developer —specifically, all of the application's functionality must be inlibraries, which are then accessed by Java code using JNI.The libraries must also be bundled with applications.
The alternative route is to develop the software integrated into theAndroid source, which means it must subsequently be distributed as acustom Android build. The system integration route does allow one tomake the software functionality available to the entire platform, hesaid, rather than just to specific applications (which is a limitationof bundling the library with an NDK application).
Whichever route one chooses, he said, there are "gotchas" to look outfor. Android exhibits not-invented-here (NIH) syndrome, he said,right down to the basic C library: it uses Bionic, a"bastardized" version of libc with missing APIs andhobbled features. Android also uses unique Git repository andpatch management tools written by Google, he added. But the mainsticking point is Android's specialized build system, which relies onAndroid-specific makefiles. Maintaining those alongside the makefilesneeded for another platform can be a considerable headache.
To simplify the process, Collabora created a tool calledAndrogenizer.Developers need to set up only a single top-level Android.mkfile for the project and Android target rules inMakefile.am. The target rules call Androgenizer, whichgenerates the makefiles Android expects (using Google's macros andAndroid-specific variables). Androgenizer can be used to generatemakefiles appropriate for either NDK or system-integration Androidbuilds, by checking for an environment variable calledANDROGENIZER_NDK.
The Androgenizer tool does not make building for Android trivial, hesaid, but once you figure it out, it only takes about 30 minutes foreach new project. Although the process is not hard, Verdejoencouraged anyone working on an Android port to get in touch, sayingthe company is interested in helping.
All things considered, the ports of these and other frameworks fromdesktop GNOME (there are several other projects on display at the company's Git repository)offer an interesting take on the desktop-versus-mobile "problem."Typically the massive increase in mobile Internet devices is talkedabout as a sign that desktop projects are doomed; Collabora'sGNOME-to-Android ports show that developers do not need to take thatas a foregone conclusion, even for a tightly-controlled platform likeAndroid.
[The author would like to thank the GNOME Foundation for travel assistance to A Coruña for GUADEC.](Log in to post comments)
GUADEC: porting GNOME to Android
Posted Aug 16, 2012 19:01 UTC (Thu) by jmorris42 (guest, #2203) [Link]
With a PC we have a open and understood boot process and fairly standardized hardware. Removing the OEM preloaded OS and installing a free OS on a PC is therefore fairly easy. Each Android device has to be undertaken as a new port. Without even a plan to crack that that problem there isn't much point trying to make a tablet or phone version of any free OS.
GUADEC: porting GNOME to Android
Posted Aug 17, 2012 2:58 UTC (Fri) by Ford_Prefect (subscriber, #36934) [Link]
I disagree with you about there not being a point in try to make a phone/tablet version -- we don't have to target every device, just as many as we can (and we can happily just aim for Google's reference devices to start with). More devices will end up being supported by the community if we provide a compelling reason to use our platform.
GUADEC: porting GNOME to Android
Posted Aug 17, 2012 18:37 UTC (Fri) by alankila (guest, #47141) [Link]
1. Autotools could be back. Android was fine without it. In general GNU makefiles are awesome and can do lots of things -- as a case in point, Android can build everything with just GNU makefiles. Can't other people just start writing those instead, and rather drop autotools entirely? If you have taken a look at Android.mk files, they tend to just list the module name, the source files, and some boilerplate that tends to be very common in all Android.mk files. Is this really too much to maintain?
2. PulseAudio. Despite the popular theory of it being superior, PA doesn't support OpenSL (audio effects on streams). Until it does, I am going to be bitterly opposed to any hint of it replacing AF. Android is one of the very few, if not the only, Linux-based systems that can do digital sound processing in user friendly way that anybody can use. The sound-producing application doesn't need to be aware of effects being applied, and nice GUIs to do it have be designed.
3. GStreamer. Android already has a media framework. I think gstreamer is responsible for videos playing without sound on OS X. I know this is an anecdotal punch under the belt, but IMHO, gstreamer has never worked properly for me. On Android, it can't be plausibly be distributed as common component to be used by multiple applications, anyway, so unless you like multi-megabyte application bundles, GStreamer remains a complete non-starter, regardless of its merits of faults.
4. Wayland. It's not vaporware, but also not ready for prime-time any time soon. The fact this plan requires complicated interaction with random vendor-provided GPU drivers that are only built for use with SurfaceFlinger makes me think that there's no hope with trying to make multiple graphics systems work simultaneously on one screen. What exactly is the story for needing Wayland at all? For instance, if GTK+ applications must run on Android, why can't GTK+ talk with SurfaceFlinger?
I know what I write here is going to come across as needlessly harsh, but the common theme I read here is "more fragmentation for android". It's not like Android wants or needs any of this. And if people seriously want to replace AF or SF with PulseAudio or Wayland, surely it would make most sense to talk with Google's Android team about it? The only consolation I have is that I can't see any of this work going anywhere, so it can be safely ignored.
GUADEC: porting GNOME to Android
Posted Aug 17, 2012 19:47 UTC (Fri) by Cyberax (✭ supporter ✭, #52523) [Link]
GUADEC: porting GNOME to Android
Posted Aug 18, 2012 17:35 UTC (Sat) by alankila (guest, #47141) [Link]
For security reasons, random DSP plugins can't be done on Android either, but the system must at boot-up time declare static list of files to be used for such purposes. But all ROM modders can change this list, so there's that. Afterwards, applications that wish to have their audio stream managed publish their session id to the wider system to work with, and DSP-related applications hear these broadcasts and can use RPC shims to connect to the main mediaserver process to add and remove these effects on a per-session basis.
It is obvious that it takes quite some effort to get from "can be done in principle" to something that can be distributed to end users. My belief is that PA's DSP story is still in the "can be done in principle" stage...
GUADEC: porting GNOME to Android
Posted Aug 18, 2012 18:05 UTC (Sat) by Cyberax (✭ supporter ✭, #52523) [Link]
Nope. DSP is often done in _hardware_ on Android (for instance, on Galaxy S phones) and sometimes sound stream doesn't even touch the CPU. So that's the reason for non-dynamic AF plugins.
GUADEC: porting GNOME to Android
Posted Aug 18, 2012 21:16 UTC (Sat) by alankila (guest, #47141) [Link]
In any case, Android vendors can provide vendor-specific implementations of AudioEffects, and I suppose it might be possible to hardware-accelerate more of the AudioFlinger API. There's a lot of maybes to this, because last time I looked there was but one digital stream that everything got mixed to, and whatever "DSP" was involved was concerned entirely with the mechanics of generating analog audio from that single PCM data.
So there's multiple kinds of things called "DSP" here. The fact that custom DSP plugins are not possible is obvious given that they must be loaded to a root-owned process implemented in C which must not crash and must be as realtime as possible. So random code, or offloadng processing by some IPC approach is probably out.
GUADEC: porting GNOME to Android
Posted Aug 17, 2012 22:24 UTC (Fri) by daniels (subscriber, #16193) [Link]
1) Static Makefiles don't support feature detection and configuration in the same way autotools do. Obviously they're not perfect, but they're a damn sight better than just writing plain static Makefiles. Converting the build system of every single project we've been working with is obviously a non-starter for any viable project.
2) This is completely possible with PulseAudio, and many vendors who've deployed it have written their own custom plugins to process streams, some extremely complex, and all without any need for custom knowledge on behalf of the applications. It also provides a number of very real benefits such as drastically reduced latency from source to hardware, and strong power savings. You can read more about this here: http://arunraghavan.net/2012/01/pulseaudio-vs-audioflinge...
3) StageFright has a number of issues, and is not particularly well-suited to more than just basic media playback (and even then ...). GStreamer, on the other hand, has been deployed in all major distributions for years, as well as part of uncountable devices (both consumer/gadget and enterprise), some of which have some hugely taxing requirements. I'm sorry to hear that it anecdotally doesn't work for you on OS X, but it certainly seems to work very well for tens of millions of people. In our Android port, it can indeed be used as part of the base system, because we're developing a branch of AOSP to use as a base for integration, rather than something you can just install from the Google Play store.
4) Wayland isn't, as you note, 1.0 yet (although it is 0.95, with a frozen core protocol), but it's something we're very interested in, as you can see from its commit stats. We think it has a lot to offer in terms of fluid and beautiful UIs which people particularly want in the mobile space, as well as media presentation. We're getting a lot of interest from both hardware vendors and their customers, so a port definitely seemed prudent.
These are all proven, solid technologies (with the exception of Wayland, which is obviously still maturing), which people have deployed and want to continue deploying on a wide range of hardware. Hardware vendors are now turning their attention to Android and only doing base enablement for the Android stack, and the number of hardware vendors who actually care about getting a proper Linux stack up on their hardware is small and diminishing every day. We don't want Android to become a completely separate silo to generic Linux, so we're trying both to offer people with Android-only hardware a much better stack, and also to use the interest from Android deployments to keep that stack more relevant so it can continue to grow and improve for everyone.
It would be nice if that weren't the reality, but what can you do?
Most of these technologies will never become mainstream Android, partially because of licensing concerns, but also partially because the Android development model is based around being almost entirely in-house within Google, rather than the traditional inclusive open source model.
Anyway, as you say, none of this will affect you in any way, so no need to be worried about it. :)
(Obviously I'm speaking for myself and not necessarily for Collabora here. One disclaimer from my brief presentation that Jake left out of the article was that I wasn't actually working on the Wayland port: it was all done by Pekka 'pq' Paalanen, but unfortunately he couldn't make it to GUADEC, so I spoke on his behalf.)
GUADEC: porting GNOME to Android
Posted Aug 17, 2012 22:56 UTC (Fri) by daniels (subscriber, #16193) [Link]
GUADEC: porting GNOME to Android
Posted Aug 18, 2012 17:56 UTC (Sat) by alankila (guest, #47141) [Link]
2. Lower latency, power usage, and freedom of sample rate are not useless features, and favor PA -- at least in some use cases. But until it has a comparable DSP story, it is inferior in other practical ways. I'm just bringing this out every time PA gets mentioned against AF so that people would keep in mind that PA is not a superior replacement in every way currently. That article you linked also mentions the OpenSL effects because I pointed them out last time too.
3. Okay. I suppose it ought to work for the basic use cases. I guess I've been really unlucky. No points for planning on distribute a modified android, though. That's precisely the sort of fragmentation Android doesn't need... ;-(
4. Does this imply that Wayland doing android-like UI is better at the job than SurfaceFlinger? Or is this just a general comparison against X? Of particular note is the fact that android 4 apparently managed to add hardware accelerated UI composition where SF apparently composes the application's UI by overlapping various buffers. It sounds fairly optimal already...
GUADEC: porting GNOME to Android
Posted Aug 19, 2012 11:14 UTC (Sun) by smcv (subscriber, #53363) [Link]
> detection which tends to be a nonstarter for cross-compiling use cases
It is possible for a configure.ac to require this, but that feature is relatively rarely-used (most Autoconf checks only require preprocessing, compiling or sometimes linking a program for the embedded architecture, not actually running it).
When a check can't be done without running a program for the embedded architecture (e.g. dbus checks for support for Linux/Solaris-style abstract Unix sockets), in a properly written configure.ac it's possible to tell configure what the result will be, which stops it from actually trying the check. For instance, when building for Android you know that the kernel does support abstract Unix sockets, and there's a configure option to say so.
GUADEC: porting GNOME to Android
Posted Aug 18, 2012 0:04 UTC (Sat) by swetland (guest, #63414) [Link]
I may be misunderstanding the goals, but it would seem like there are simpler paths to getting existing Linux apps running on Android than replacing big chunks of platform internals...
GUADEC: porting GNOME to Android
Posted Aug 25, 2012 14:02 UTC (Sat) by nilesh (guest, #37042) [Link]
Isn't Ubuntu for Android taking care of a lot of the low-level stuff that could be useful here as well? Granted, that's work in progress and not yet available, but still there could some code-sharing possible here.
