Android 2.2 (“Froyo”) and you: the gritty details behind «Apps to SD»

tl;dr version: If you can’t use Apps2SD, do adb shell, pm setInstallLocation 2, move any app to SD (ignoring possible “failed” errors at first try).

Our beloved Frozen Yoghurt came with many new features welcome to the community at large, and one feature which had a mixed reception: “Apps on external storage”, which allows the user to install applications to its phone’s external storage – mostly in order to free up internal disk space.

Many custom ROM distributions for Android already had this feature built in, going by the moniker “Apps to SD” (or “Apps2SD” or just “A2SD”).

The typical implementation of A2SD works by using an ext2/ext3 partition on the SD card of your device – and usually only works when it’s exactly the second partition. For the sake of argument, one such custom implementation of A2SD will be included at the end of this post.

What it then does is just completely move all the applications to the SD partition, leaving only the /data partition behind, and uses a bind mount to fool the system into believing that the files are still on the same file system. So, in essence, the a2sd patch “cheats” and pretends that nothing actually has happened while quietly siphoning the apps to the SD card.

This, of course, only works when you actually have root access to your device and are allowed to play around with all the interesting system data itself. If you’re working on an unrooted/stock handset and firmware, you don’t have the option of using this feature; and also if you’re too lazy or unknowing or prissy to set up an ext[23] partition on your SD card.

Thus the «official» Apps to SD comes into play – if your device is running Android 2.2, that is.

An important thing to note about understanding the official implementation is that it assumes that the user has no direct access to the /system partition. Especially: the user is not able to access any installed Android application package in any way that allows copying files.

What Froyo does when installing an application to SD is pretty simple: it creates a file on the SD card and uses this as a container to store the application in. Said container is used with a crypted loop mount, that is the actual data on the SD card is encrypted, and will be decrypted at load time when accessing the application.

The idea behind this seemingly convoluted setup is simple: if you have paid for an application, you could just store it on SD and then copy it if it is not encrypted. If it is encrypted, you cannot access the application in a “simple” way to copy (i.e. pirate) it.

Additionally, the application (with the default settings) needs to allow Android to move it to the SD card – otherwise the system does not enable the functionality, probably to ensure that applications aren’t “broken” by SD storage.

Of course this is easily manhandled by using the USB debugging interface with adb shell: just issue pm setInstallLocation 2. This tells he package manager (hence «pm») to use the external storage as a default install location, which incidentally lifts the block that does not allow an application to be stored on external storage, too.

The downside:
/dev/block/dm-41 on /mnt/asec/de.hafas.android.db-1 type vfat [...]

And yes, that’s 41 device mapper crypto loops. At least they don’t produce that much overhead as to noticeably slow down the system.

One of the boons of the Froyo implementation is that with above command, it can easily be used even with an unrooted phone and without repartitioning your SD drive. The disadvantages are that Android requires a fair bit of time after booting to mount all the crypto loop devices, which will result in your applications being accessible rather late after booting. Also, you will not be able to use widgets of any app that is on SD.

Here come the advantages of the customized A2SD approach: you can still access widgets and applications on your SD card even when it is mounted to your computer – because Android will only mount away the root partition (the FAT one), and not your ext partition. And you’ll have less overhead due to the crypto business.

And, as promised, the code that enables A2SD on most current ROMs:

#!/system/bin/sh
#
# Apps2SD using symlinks and bind mounts
# Originally by cyanogen (shade@chemlab.org)
# Modified to use a cleaner /sd-ext implementation by IEF (ief@shadowchild.nl)

# execute any postinstall script then kill it
if [ -e /dev/block/mmcblk0p2 ];
then

    # mount and set perms
    busybox mkdir /sd-ext
    busybox mount -o noatime,nodiratime -t auto /dev/block/mmcblk0p2 /sd-ext;
    busybox chown 1000:1000 /sd-ext;
    busybox chmod 771 /sd-ext;

    # clean up any old symlinks, create data directories
    for i in data;
        do
                if [ -h /data/$i ];
                then
                        rm /data/$i;
                fi;
                if [ ! -d /data/$i ];
                then
                        mkdir /data/$i;
                        busybox chown 1000:1000 /data/$i;
                        busybox chmod 771 /data/$i;
                fi;
        done;

    # don't allow /data/data on sd because of upgrade issues - move it if possible
    if [ -d /sd-ext/data ];
    then
        busybox cp -a /sd-ext/data/* /data/data/;
        busybox rm -rf /sd-ext/data;
    fi;

    # move apps from internal memory to sdcard
    for i in app app-private dalvik-cache;
    do
        if [ ! -d /sd-ext/$i ];
        then
            mkdir /sd-ext/$i;
        fi

        busybox chown 1000:1000 /sd-ext/$i;
        busybox chmod 771 /sd-ext/$i
            
        if [ -d /data/$i ] && [ ! -h /data/$i ];
        then
            busybox cp -a /data/$i/* /sd-ext/$i/;
            busybox rm -f /data/$i/*;
        fi;
    done;

    # symlink app dirs - they must be on the same filesystem
    for i in app app-private dalvik-cache;
    do
        if [ -d /data/$i ] && [ ! -h /data/$i ];
        then
            busybox rm -rf /data/$i;
            busybox ln -s /sd-ext/$i /data/$i;
        fi;
    done;

    # clean up old whiteouts
    for i in local misc property system tombstones data;
    do
        if [ -h /sd-ext/$i ]; then rm -f /sd-ext/$i; fi
    done;

    # please don't put odex files in the app directory people!
    # it causes dexopt to crash when switching builds!
    busybox rm -f /sd-ext/app/*.odex

    setprop shadow.apps2sd.active 1;
    
    echo "+++ Apps-to-SD successfully enabled";

else
    
    # replace symlinks with directories so we can boot without sd
    for i in app app-private dalvik-cache;
    do
       if [ -h /data/$i ];
       then
            rm -f /data/$i;
            mkdir /data/$i;
            busybox chown 1000:1000 /data/$i;
            busybox chmod 771 /data/$i;
        fi;
    done;


    setprop shadow.apps2sd.active 0;
fi;
sync;

This is run as an init script.

Using grub2 to recover your system

grub2 is hailed as the all new, super modular cure-all remedy for all booting problem you’ve had, have and will have. At least that’s the way the developers and some enthusiasts see it, whereas most blokes who’ve actually had to use it with more than arrow keys and enter will paint a slightly different picture.

The thing with grub2 is that even though in theory it sounds like the end of all things booting, it’s about as well-documented as the question for life, the universe, and everything.

And as I today had to try to fight my way through googling for necessary information again, I’d thought I’d create a quick step-by-step reference with all the most interesting bits you’ll ever need already there.

Thusly, the ingredients needed to resurrect your computer with grub2. The gist is that you have the goal of booting one specific operating system on your computer, from wherein which you’ll use whatever methods you deem necessary to update your grub in the “right way” – usually a downgrade to an older version and waiting for the dust to blow over.

  1. A booting grub2. If your grub2 already fails to boot because of some random error, you need to get a grub in smelling distance of your BIOS. One of the most proven methods is to
    1. Download a USB rescue image like grml (usually from Your Other Computer or that of somebody else)
    2. Put it on an USB stick (dd if=grml-variant_version.iso of=/dev/sdx in most cases, with appropriately chosen variables)
    3. (Re)boot, eventually adjusting the priority for your USB HDD/USB key

    And that’s it, you’re in a grub. Also note that it’s recommendable to have an USB stick with a rescue image lying around for the times when you can’t just easily download it.

  2. Enter the command line/shell mode by pressing ‘c‘.
  3. Do an ‘ls‘, which will give you a listing of recognized devices. Doing an ‘ls device‘, e.g. ls (hd0,1) will give you more information about that device.
  4. If the information by your ls isn’t complete, you will have to load some modules (by using insmod modulename). Here’s a checklist:
    1. If you do not see any other devices which look like your hard drive(s), e.g. you only have an (hd0) device from your USB medium, then load a device driver. They will allow you to find the actual devices. Examples include:
      • biosdisk
      • scsi
      • fs_uuid
      • pci
      • raid
      • mdraid
      • dm_nv
    2. If you have devices, but no partitions, you’ll need a partition driver. It seems the default grub config does not load any partition driver, and debugging this is just a bit annoying. But there’s two easy choices for most people:
      • Load the module “part_msdos“.
      • If this doesn’t help, try “part_gpt“.

      These are the two most common partition tables (at least for next to everyone reading this guide in need) and should help your grub find its partitions again.

    3. Eventually, you will also have to load your filesystem drivers. I presume you already know which those are, but for the sake of completion:
      • Almost all Linux use ext2
      • Most current Windows will use ntfs, but fat is also an option.
      • Mac users will use hfsplus for newer systems, hfs for older ones.
    4. The next step depends on exactly what you want to do. There’s a fork in the road – if you just want to load your previously unbootable grub, you will try to load its configuration file, else you’ll try to boot your operating system kernel.

    5. To search for a file, you use the search -f filename command, which will give you results on where files of that name are stored. Use root device to set the resultant device as the root device for your further operations. If you only want to load your old grub config, type in configfile filename, whereas filename will usually be something like /grub/grub.cfg or /boot/grub/grub.cfg.
    6. Should this fail to resolve your problem, or not be what you’re aiming for, you’ll need to find the operating system. For most Linuxens, you’ll probably have a file called /vmlinuz or /boot/vmlinuz to search for. For Windows operating systems, look for /Windows/win.ini. For Mac: no clue. When found, set your root device (with root device).
    7. Now methods will become divergent, as operating systems differ in the way of booting them.
      Linux
      1. kernel kernel_filename
      2. initrd initrd_filename [most current kernels come with an “initial ramdisk” holding modules etc.]
      3. boot – if all goes well, you’re set.
      Windows
      1. chainloader +1
      2. boot
      MacOS
      Probably the same as Windows, using the chainloader.

    And that’s it. It should cover most cases you’d need to restore your capability of booting your operating system. You’ll probably want to fix/install your bootloader after this, though.

    A helpful tool for debugging your current grub state is probe, which will allow you to check what drivers are assigned to devices.

D&D Characters: Shamorn Fallenheart, Tiefling Bard

As a bit of a side occupation, I like to engage in some character design for role-playing games, as it just comes as a natural extension of being a hobby-ish writer person.

Thus, I present: Shamorn Fallenheart, a tiefling bard from High Imaskar.


Birth – and over misgivings

Shamorn was born in Gheldaneth, the fading Mulanian metropolis of High Imaskar, and his parents believed in the prophecies stating Shamorn to bring forth better times for the tiefling folk of the Gheldaneth slums. Being raised in a community of hired hands to accompany adventurers on dangerous treasure hunts through the depths of the sunken city, hopes were laid on him, and him alone, to liberate them from this life of unofficial slavery.

Early life

Our young tiefling was always a bit pampered. The male role models of the community were often too busy getting killed on a foolish quest, as was Shamorn’s own father – shortly before his fourth birthday. As it were, there was none of the usual goading and testing a tiefling endures as part of growing up. The consequences of this, as well as the pampering he received by his mother and other “faithfuls”, would be dire indeed.

Thus Shamorn grew to be a young adult, helping out everywhere in the community, without ever taking up a real job. He had many on and off teachers, versing him in skills as @skills and the heritage of the tiefling race, training him in the use of weapons and telling stories of heroic deeds throughout time.

Constantly surrounded by an appreciation for life, for heroism, the history and culture of his people and a will to bring good to them, it came as a great surprise to many that Shamorn Fallenheart, Prophesied Saviour of the Gheldaneth Tieflings, came to start training to be…

a bard.

There was a wandering Elven Bard in Gheldaneth at the time, and Shamorn choose to apprentice himself to him, believing that becoming a bard, a herald of their people, would be worth much more than simply slaughtering any would-be oppressors or being a leader to guide the people to their Promised Land.

As was to be expected, his decision did not sit well with some, if not most, of his elders. His mother came just short of disinheriting him, and he was forever branded as a wimp by most others. Still, there were some people who still believed in him, and he managed to stay in the community, even though everyone tried to forget about any kind of prophesy laid upon him.

The turning point

His apprenticeship was going well, all things considered. But his teacher, unbeknowest to him, was a bit of a braggart and ignorant, that is to say: not a very good bard. Still, Shamorn managed to master his natural graps of the Arcane under his tutorship, even though the social values might have been slightly distorted.

Sadly, this distortion and the infusion of heroic tales led to an unfortunate incident. A rough band of treasure hunters, with a fierce reputation for their harsh effectiveness and rumours of a brutal and unrelenting manner towards opposition, sought out their enclave to hire some of their men for help. So, after a few minutes of shouting, waving of weapons and dragging people out of their hovels, Shamorn thought it was time to act.

Bravely stepping forward, he confronted the leader of the scavengers, demanding of him to cease these despicable acts and appealing to his good sense, as a man, to respect his people’s wishes.

The screams as the leader’s minions started slaughtering the women and children are still stuck in Shamorn’s head. He still only has vague memories of that moment, but there is one thing he is quite confident of:

As his mother’s lifeless body was thrown in front of him, crumpled up in a heap, he snapped. Shamorn went into a rage, slamming into the minions and fighting them fiercely. It seemed the demon in him had taken control, for he was full of laughter at the slaughter he was causing, taunting his enemies as he smashed their faces in with his $weapon or embedded his daggers into their hearts, even just ripping into them with his claws and biting as he went along.

It did not take long for him to cut through the minions, emerging bathed in blood, eldritch powers abound and flames crackling around his body. His Elven master bard was astonished at the display, and recognized the potential of a warlock in him should he have even been trained thusly. As it was, the teacher preferred to cower in fear and observe what happened next.

Shamorn confronted the leader of the scavengers who was just standing there, shocked to his core.

“This is what happens when you try to compel my folk, human!” the bard stated in an almost neutral voice, only a hint of a burning darkfire noticable in the voice. And with that, he slew the leader of the group that brought death to his kin.

And as if by miracle, Shamorn immediately calmed down to his usual, naive self. The only hint at his monstrosity was the fact that he surveyed the slaughter he had caused without fear, shame or disgust. Looking around him, he found few people left alive. Some were cowering inside their hovels, either hiding their faces or staring out at him with fear. Others seem to have run a way, and it was eerily silent.

Shamorn cleared his throat. “My master, I will be leaving now. Do you wish to accompany me?”

His master, still shaking slightly, replied “No, my apprentice. I do not think that you need me any further. Consider your training complete.”

And with these short words, the recently orphaned Shamorn Fallenheart set out into the Realms, venturing forth to herald his people – and to leave this blighted home which has been cursed by his deeds.


The character statistics will follow as soon as I have access to the relevant documents again. I might also write a short story or two detailing the background or later adventures.

Vortrag: Datenschutz und -sicherheit in MMOs

Auf der RPC 2009 habe ich einen Vortrag über Datenschutz und -sicherheit in MMOs gehalten.

Wohl gemerkt sei der Vortrag nur als eine Art Einführung zu verstehen; richtige Möglichkeit der Forschung bietet sich leider nicht, da die Hersteller nur unzureichend kooperativ sind.

Downloadlink für die Folien: Datenschutz- und -sicherheit in MMOs
Es gab keine Ton- oder Bildaufnahmen.

OpenChaos-Vorträge: Digitale Verhütung (de)

Von September bis November gab es im Chaos Computer Club Cologne die Vortragsreihe „Digitale Verhütung” im Rahmen des OpenChaos. Diese Reihe richtete sich an den technisch weniger begabten Nutzer; sie sollte der Sensibilisierung gegenüber den Gefahren dienen, ampoule welche einem in der modernen Datenwelt drohen, cheap und was man zu seinem Schutz davor machen kann.

In der Reihe habe auch ich mehrfach vorgetragen; zum einen habe ich, case zusammen mit marcel, den Vortrag „Sichere Kommunikation” gehalten. Dort haben wir darüber geredet, wie einfach es manchmal sein kann, die Kommunikation anderer abzuhören, und was man dagegen tun kann, damit das nicht mehr so einfach geht. Dabei wurden allgemeine Grundlagen der Kommukationsverschlüsselung behandelt, sowie diverse Methoden der Verschlüsselung und deren Implementation in populären IM-Programmen.

Dann habe ich noch im November einen Teil des Metavortrages „Verschlüsselte Datengräber” gehalten; dort ging es, wie schwer zu erraten ist, um Festplattenverschlüsselung. Ich übernahm dabei den letzten Teilabschnitt, „Umgehung” und beschäftigte mich genau damit. Grundlagen, Ausnutzen von Nutzerschwachstellen sowie die Umgehung auf technischer Ebene wurden angerissen und auf einfachem Niveau erklärt; eine detaillierte Erklärung, oder gar Beispiele, waren nicht vorgesehen.