Android 2.2 (“Froyo”) and you: the gritty details behind «Apps to SD»

tl;dr version: If you can’t use Apps2SD, do adb shell, pm setInstallLocation 2, move any app to SD (ignoring possible “failed” errors at first try).

Our beloved Frozen Yoghurt came with many new features welcome to the community at large, and one feature which had a mixed reception: “Apps on external storage”, which allows the user to install applications to its phone’s external storage – mostly in order to free up internal disk space.

Many custom ROM distributions for Android already had this feature built in, going by the moniker “Apps to SD” (or “Apps2SD” or just “A2SD”).

The typical implementation of A2SD works by using an ext2/ext3 partition on the SD card of your device – and usually only works when it’s exactly the second partition. For the sake of argument, one such custom implementation of A2SD will be included at the end of this post.

What it then does is just completely move all the applications to the SD partition, leaving only the /data partition behind, and uses a bind mount to fool the system into believing that the files are still on the same file system. So, in essence, the a2sd patch “cheats” and pretends that nothing actually has happened while quietly siphoning the apps to the SD card.

This, of course, only works when you actually have root access to your device and are allowed to play around with all the interesting system data itself. If you’re working on an unrooted/stock handset and firmware, you don’t have the option of using this feature; and also if you’re too lazy or unknowing or prissy to set up an ext[23] partition on your SD card.

Thus the «official» Apps to SD comes into play – if your device is running Android 2.2, that is.

An important thing to note about understanding the official implementation is that it assumes that the user has no direct access to the /system partition. Especially: the user is not able to access any installed Android application package in any way that allows copying files.

What Froyo does when installing an application to SD is pretty simple: it creates a file on the SD card and uses this as a container to store the application in. Said container is used with a crypted loop mount, that is the actual data on the SD card is encrypted, and will be decrypted at load time when accessing the application.

The idea behind this seemingly convoluted setup is simple: if you have paid for an application, you could just store it on SD and then copy it if it is not encrypted. If it is encrypted, you cannot access the application in a “simple” way to copy (i.e. pirate) it.

Additionally, the application (with the default settings) needs to allow Android to move it to the SD card – otherwise the system does not enable the functionality, probably to ensure that applications aren’t “broken” by SD storage.

Of course this is easily manhandled by using the USB debugging interface with adb shell: just issue pm setInstallLocation 2. This tells he package manager (hence «pm») to use the external storage as a default install location, which incidentally lifts the block that does not allow an application to be stored on external storage, too.

The downside:
/dev/block/dm-41 on /mnt/asec/de.hafas.android.db-1 type vfat [...]

And yes, that’s 41 device mapper crypto loops. At least they don’t produce that much overhead as to noticeably slow down the system.

One of the boons of the Froyo implementation is that with above command, it can easily be used even with an unrooted phone and without repartitioning your SD drive. The disadvantages are that Android requires a fair bit of time after booting to mount all the crypto loop devices, which will result in your applications being accessible rather late after booting. Also, you will not be able to use widgets of any app that is on SD.

Here come the advantages of the customized A2SD approach: you can still access widgets and applications on your SD card even when it is mounted to your computer – because Android will only mount away the root partition (the FAT one), and not your ext partition. And you’ll have less overhead due to the crypto business.

And, as promised, the code that enables A2SD on most current ROMs:

#!/system/bin/sh
#
# Apps2SD using symlinks and bind mounts
# Originally by cyanogen (shade@chemlab.org)
# Modified to use a cleaner /sd-ext implementation by IEF (ief@shadowchild.nl)

# execute any postinstall script then kill it
if [ -e /dev/block/mmcblk0p2 ];
then

    # mount and set perms
    busybox mkdir /sd-ext
    busybox mount -o noatime,nodiratime -t auto /dev/block/mmcblk0p2 /sd-ext;
    busybox chown 1000:1000 /sd-ext;
    busybox chmod 771 /sd-ext;

    # clean up any old symlinks, create data directories
    for i in data;
        do
                if [ -h /data/$i ];
                then
                        rm /data/$i;
                fi;
                if [ ! -d /data/$i ];
                then
                        mkdir /data/$i;
                        busybox chown 1000:1000 /data/$i;
                        busybox chmod 771 /data/$i;
                fi;
        done;

    # don't allow /data/data on sd because of upgrade issues - move it if possible
    if [ -d /sd-ext/data ];
    then
        busybox cp -a /sd-ext/data/* /data/data/;
        busybox rm -rf /sd-ext/data;
    fi;

    # move apps from internal memory to sdcard
    for i in app app-private dalvik-cache;
    do
        if [ ! -d /sd-ext/$i ];
        then
            mkdir /sd-ext/$i;
        fi

        busybox chown 1000:1000 /sd-ext/$i;
        busybox chmod 771 /sd-ext/$i
            
        if [ -d /data/$i ] && [ ! -h /data/$i ];
        then
            busybox cp -a /data/$i/* /sd-ext/$i/;
            busybox rm -f /data/$i/*;
        fi;
    done;

    # symlink app dirs - they must be on the same filesystem
    for i in app app-private dalvik-cache;
    do
        if [ -d /data/$i ] && [ ! -h /data/$i ];
        then
            busybox rm -rf /data/$i;
            busybox ln -s /sd-ext/$i /data/$i;
        fi;
    done;

    # clean up old whiteouts
    for i in local misc property system tombstones data;
    do
        if [ -h /sd-ext/$i ]; then rm -f /sd-ext/$i; fi
    done;

    # please don't put odex files in the app directory people!
    # it causes dexopt to crash when switching builds!
    busybox rm -f /sd-ext/app/*.odex

    setprop shadow.apps2sd.active 1;
    
    echo "+++ Apps-to-SD successfully enabled";

else
    
    # replace symlinks with directories so we can boot without sd
    for i in app app-private dalvik-cache;
    do
       if [ -h /data/$i ];
       then
            rm -f /data/$i;
            mkdir /data/$i;
            busybox chown 1000:1000 /data/$i;
            busybox chmod 771 /data/$i;
        fi;
    done;


    setprop shadow.apps2sd.active 0;
fi;
sync;

This is run as an init script.

vimium mapping for Dvorak layouts

I recently stumbled upon the rather neat vimium extension for Chrom(e|ium), which does much the same as the vimperator extension for Firefox. The problem, though, as with vimperator and vim itself, is that the default keyboard mappings are a bit of a pain in the arse for Dvorak users, as hjkl isn’t on the home row anymore, much less next to each other.

Therefore, it needs some remapping to get in a halfway familiar and Dvorak-compatible layout, which would look like this:

unmapAll

map r reload
map e removeTab
map u restoreTab
map h scrollDown
map t scrollUp
map d scrollLeft
map n scrollRight
map <c-h> scrollPageDown
map <c-t> scrollPageUp
map <c-u> scrollFullPageDown
map D goBack
map N goForward
map T nextTab
map H previousTab
map <c-y> createTab
map gg scrollToTop
map G scrollToBottom
map gf toggleViewSource
map zi zoomIn
map zo zoomOut
map yy copyCurrentUrl
map i enterInsertMode
map f LinkHints.activateMode
map F LinkHints.activateModeToOpenInNewTab
map / enterFindMode
map . performFind
map , performBackwardsFind

Just paste it in the remap field of the extension’s “advanced options” menu.

pisg: patch to irssi parser for euIRC ‘admin’ user mode

As pisg is ill-equipped to handle support for ‘admin’ users in the standard configuration, I went on a quick code hunt to find the bit of code responsible for stripping nick modes from a log line. A bit counter-intuitively, this function is called normalline, and not something like normalize or strip_mode.

Anyhow, here’s a small patch to fix the problem for the Irssi parser module:

--- modules/Pisg/Parser/Format/irssi.pm.old	2008-02-13 21:40:25.000000000 +0100
+++ modules/Pisg/Parser/Format/irssi.pm	2010-03-16 02:29:42.000000000 +0100
@@ -10,7 +10,7 @@
     my ($type, %args) = @_;
     my $self = {
         cfg => $args{cfg},
-        normalline => '^(\d+):\d+[^<*^!]+<[@%+~& ]?([^>]+)> (.*)',
+        normalline => '^(\d+):\d+[^<*^!]+<[@%+~&! ]?([^>]+)> (.*)',
         actionline => '^(\d+):\d+[^ ]+ +\* (\S+) (.*)',
         thirdline  => '^(\d+):(\d+)[^-]+-\!- (\S+) (\S+) (\S+) (\S+) (\S+)(.*)',
     };

Or you could just download the diff directly.

Gaim to Pidgin log conversion

I was browsing through some older files of mine and cleaning up when I stumbled over a bunch of old instant messenging log files. These logs were still in the legacy unified log file format which Gaim (today knows as Pidgin) used in the beginning. I didn’t find a converter after about ten seconds of using Google, so I went ahead and wrote my own.

It’s mostly feature-complete, and will split up any number of [something].log you pass to it into [something]/[date].txt style files. What it can’t really do is determine what kind of protocol it’s dealing with, so you’ll still have to move the log directory manually to the appropriate protocol directory inside of ~/.purple/logs. Be wary when moving files, though, as you might accidentally overwrite other log files. Use rsync.

Anyhoo, you can either get the file directly or just try this delicious copypasta:

#!/usr/bin/perl
# gaim2pidgin.pl
# author:  towo <towo@ydal.de>
# version: 3
# license: CC-BY-DE-3.0
 
use strict;
 
# convert short month names to numbers.
my %shortmonths = (
	'Jan' => '01',
	'Feb' => '02',
	'Mar' => '03',
	'Apr' => '04',
	'May' => '05',
	'Jun' => '06',
	'Jul' => '07',
	'Aug' => '08',
	'Sep' => '09',
	'Oct' => '10',
	'Nov' => '11',
	'Dec' => '12'
);
 
# go through files
FILE: foreach my $file (@ARGV) {
	my ($header, $target);
 
	# sanity checks
	unless (-f $file) {
		warn "$file is not a file.\n";
		next FILE;
	}
	unless(open(LOG, $file)) {
		warn "Unable to open $file for reading: $!\n";
		next FILE;
	}
 
	# get file header, get target name
	chomp($header = <LOG>);
	$header =~ s#<.*?>##g;
	$target = $file;
	$target =~ s/\.log$//;
 
	# check header for correctness
	unless($header =~ m{^(<HTML><HEAD><TITLE>)?IM Sessions with .*? \
                             (</TITLE></HEAD><BODY BGCOLOR=".*?">)?$}i) {
		warn "$file does not seem to be a gaim conversation.\n";
		next FILE;
	}
 
	# read LOG to file
	my @contents = <LOG>;
	close(LOG);
 
	# parse log file (one loop ^= one chat session)
	while(@contents) {
		my ($session, $identifier, $date);
 
		# get session identifier
		chomp($session = shift @contents);
 
		# Strip HTML.
		#$session =~ s#<.*?>##g;
		$session =~ s#</?(FONT|B|I|ALIGN|HTML|HEAD|TITLE|HR|BR|BODY|H3).*?>##ig;
 
		# sanity check for the session identifier
		unless ($session =~ m/^ ?---- New Conversation @ \w{3} (\w{3}) ([0-9 ]{2}) \
                                      (\d{2}):(\d{2}):(\d{2}) (\d{4}) ----$/) {
			warn "Could not recognize session identifier: «$session»\n";
			next FILE;
		}
 
		# extract date from session identifier and create target identifier
		$date = "$6-$shortmonths{$1}-" . sprintf("%02d", $2) . ".$3$4$5";
		$identifier = "$target/$date.txt";
 
		# sanity check for target directory
		unless (-d $target) {
			unless(mkdir $target) {
				warn "Could not create directory $target: $!\n";
				next FILE;
			}
		}
 
		# open output file
		unless(open(OUTPUT, "> $identifier")) {
			warn "Could not write to $identifier: $!\n";
			next FILE;
		}
		select OUTPUT;
 
		# extract log to log file
		until($contents[0] =~ m/^(<HR><BR><H3 Align=Center>)? ?---- New Conversation/ \
                                      or !@contents) {
			my $line = shift @contents;
			$line =~ s#<.*?>##g;
			print $line;
		}
		close OUTPUT;
	}
}

Creative Commons License Licensed as CC-BY-DE-3.0.