Faster USB sticks

I've been trying to set up my Linutop 2 box as a small server for a while now, but keep hitting a variety of annoying problems. The latest was that an intenso 32GB USB stick seemed to be the cause of achingly slow performance.

I read in Linux Format recently that it was wise to make sure partitions and filesystems were aligned with the structures found on the flash drives. Flash devices come set up in this way, but invariably with archaic FAT32 filesystems and I wanted to use a more up-to-date linux filesystem. I didn't know how to ensure alignment, so I embarked on a course of reading across the internet.

Unfortunately the information I found was generally very confusing, some articles told me what to do, but not why, and others went into bamboozling levels of gritty detail, and one or two even told me aligning wasn't worth the effort on USB2.0 drives. I'll provide links of the sites I found most helpful at the end, but here's my account of what I did, starting with the USB stick formatted in a completely misaligned way with three partitions - the result of my fettling before I knew any better.

First I plugged the drive into my laptop, then, as root:

mount /dev/sdb3 /mnt/hd

I tested the read speed with:

hdparm -t /dev/sdb

and this told me the speed was 23.8 MB/s - I did a few tests and found this varied by no more that +/- 0.05 MB/s. I then mounted the third partition, which had an ext2 filesystem on it, with:

mount -o sync /dev/sdb3 /mnt/hd

The sync option means that the writing isn't buffered - if it is buffered then the test will give you a misleadingly high speed because it will only include the time take to write to the buffer in memory. I then tested the write speed with:

dd count=100 bs=1M if=/dev/zero of=/mnt/hd/test

This copies one hundred 1 megabyte blocks of zeroes to a file called test. The result was a paltry 893 kB/s.

I then set about deciding how I could reformat the drive to get better performance, particularly for writing. I wish I'd kept a note of the original partitioning of the stick as that would give me vital clues about how to align partitions and filesystems. After searching the web, the best I could do was guess that the USB stick was likely to have a 4kB or 8kB page size and a 512kB erase block size.

To be honest, I haven't really had time to get to grips with the details, so please excuse all the uncertainty, but I'll try and set out my patchy understanding. The erase block size is significant because if the filesystem requires, say, 100kB to change, then the flash drive electronics will have to wipe at least a 512kB block and rewrite it all. I say "at least" because if that 100kB happens to cross the boundary between two adjacent 512kB blocks then both blocks will have to be erased. These erase blocks are grouped together to form erase block groups which, I understand, are typically 4 MB in size. This means that partitions boundaries should be on multiples of 4MB on the device.

The page size represents the smallest unit that can be addressed on the flash drive. I'm not sure what impact this has, except that clearly you don't want the partition or filesystem constantly straddling the "pages". But, if you're using a block size of 4kB in an ext filesystem and you've aligned the partition to 4MB there should not be a problem. With modern multi-GB drives 4kB will usually be the default block size used by the mkfs command and it's also the maximum permissible block size on typical systems.

From this you can get some idea of how aligning data with these blocks can help with performance. Also, because it reduces the number of writes to the disk, alignment can also help increase the life of a flash drive. One big problem in aligning is that there is no easy way to find out the page or erase block size for a particular stick because such information is often not published by manufacturers. The best you can do is look at the original factory formatting of the stick or search around on the internet to see if someone has measured it. (There are tools to do it yourself, and I tried flashbench which was intriguing but unfortunately it gave inconclusive results for my USB stick.)

I side stepped the issue of partition alignment by not creating any partitions at all (zero is a multiple of all numbers!) In linux it is quite permissible to create a file system directly on the device. I did this with the command:

mkfs.ext4 /dev/sdb -b 4096 -E stride=128,stripe-width=128

This creates an ext4 filesystem on device sdb (sda is the hard drive of my laptop). The -b 4096 specifies the use of a 4kB block size (I had tried 8192 but that is possible only on a few specialised systems. Next, I tried to make sure the filesystem was likely to be aligned with 512kB blocks by setting the stride and stripe-width both to 128 because 128 times 4kB is 512kB. These two parameters are more often used to make sure filesystems work efficiently with RAID arrays, but as we're only dealing with one drive here I understand that they should be set to be equal to each other (though I confess I don't know why).


Finally, to save on disk writes, I turned the journal off:

tune2fs -O ^has_journal /dev/sdb

The downside of this is that the disk will be more likely to suffer from data loss if powered down or removed incorrectly. (This option can also be set when creating the filesystem I believe so it should be possible to combine the above two commands.)

With this done, I performed the above tests again and found that the read speed was almost the same at 22.8 MB/s. The write speed was now 4.7 MB/s - a five-fold increase. Job well done! That wasn't all, if I repeated the test, the write speed seemed to go up reaching a little over 10 MB/s after 4 runs of the test. I'm not sure why this might be, but it could be to do with some optimisation in the onboard electronics in the flash drive combined with the artifice of the test in writing long stretches of zeroes.

But, I'm not quite comparing like with like here. In the "before" tests I was writing to an ext2 partition, so is it possible that some of the speed-up is due to inefficiencies of ext2 compared to ext4? To test this I created an ext2 partition directly on the disk with this command:

mkfs.ext2 /dev/sdb -b 4096

(The stripe and stride width options are not available for ext2.) The first thing that struck was how slow this was compared to the ext4 creation - 30 minutes compared to a few minutes. The tests revealed that performance was back to almost exactly what I found on the original ext2 partition with its completely misaligned partitions and filesystems.

Finally, I wanted to see if it was ext4 that was behind the speed gain, or if it was the setting of the stripe and stride option that brought about the improvement. To test for this I issued this command:

mkfs.ext4 /dev/sdb -b 4096

and did not disable the journal. The write speed was now 3.6 MB/s which seems to suggest that ext4 is responsible for most of the speed gain. The continued flashing of the stick's LED after the test had ended suggested that the journal was keeping the stick busy at least to some extent and this was further confirmed when I umounted the stick and the flashing ceased (this ruled out the activity as the flash doing wear-leveling maintenance. Interestingly, there was a slightly improvement in read speed, but no more than a few percent.

So, in conclusion, the most important improvement seemed to be just to use ext4 instead of ext2. Disabling the journal helped a little too, as did setting the stripe and stride options on ext4. But, I didn't find any evidence that alignment helped improve the performance of this drive.

Here are some links:

• Good starting point: http://linux-howto-guide.blogspot.co.uk/2009/10/increase-usb-flash-drive-write-speed.html
• A very thorough and detailed technical account: http://lwn.net/Articles/428584/
• Somewhere between the above too in technical detail - some good diagrams: http://blog.nuclex-games.com/2009/12/aligning-an-ssd-on-linux/