This blog is no longer updated - please go to http://blog.mcnalu.net
I've decided to move this blog from Google's Blogger platform to one that runs on my own server and that is generated by a marvellous piece of software called Pelican.
I'll probably write a blog post about Pelican, but for now it's enough to say that all existing post and image content will be present in the new blog and it will be at the same root URL as before: http://blog.mcnalu.net
Direct links to old posts probably won't work, but the old blog will remain as it is, residing on blogger.com, unchanging, preserved in aspic at http://oldblog.mcnalu.net, until such time as Google decides it should no longer exist.
Friday, November 15, 2013
Monday, September 9, 2013
Lending is good and bad
I was once told by a person I very much respect that I had an aversion to taking out loans. I suppose in the climate of free loaning (free as in loving, not as in beer) that persisted until about 2007 I did seem loan-averse to most. I thought it imprudent to view the rise in the value of a house that served as a home as a real increase in wealth. I thought it doubly imprudent to remortgage on such an increase. I warned a friend in 2002 not to take out a mortgage for nine times his salary. I didn't like the fact that a new generation of students who came after me viewed it as normal to start out one's working life in debt.
But of course I have taken out loans. I've taken out mortgages for houses. I took out a loan fifteen years ago to buy a car. Taking out these loans made sense. The mortgage helped me get on the housing ladder in 1995, which in the long run has allowed me to spend a surprisingly small amount to live in a decent home. The car was different - it was a necessity to allow me to get to work and so the loan and depreciation were unavoidable at the time.
But I would have preferred to avoid these loans because if I had the money to buy the house or car outright I'd have saved tens of thousands of pounds over the years.
So now, with the UK's economy still bumping along the bottom, as it has for several years, it might surprise some to hear me say, "yes, borrowing money to spend money is a good way to get the economy going again". The immediate retort of many is to say, "if you're in debt, it's stupid to take out more debt to pay it off." It may or may not be stupid, depending on what you do. Let's take familiar example of debt - that associated with buying a house.
A person wishes to take out a loan to buy a house to be the family home. They borrow £200,000 at a 5% interest rate and use that to buy the house. This loan is perfectly sensible (aside for possibly inflated house values) because a home is a necessary thing. The person is essentially renting the £200,000 required to purchase the home by paying £10,000 in interest per year.
Some years later they decide to remortgage for the same amount but at a 4% interest rate offered by a different lender. This means they take out the loan from the new lender and the money from that goes to pay off the original loan. This is a clear example in which taking out a new loan to pay off an existing loan is a good idea because now the cost of the £200,000 loan is only £8000 per year.
But let's imagine when it comes time to remortgage that house prices have increased and the house is now worth £250,000. If they decide to remortgage at 4% then they will be left with £50,000 to spend and still be paying £10,000 per year. This seems like a good deal, the cavaet being that if house prices ever drop and they spend their £50,000, say on a luxury car, then they might get stuck in a negative equity trap one day, i.e. their house can't be sold to pay off the £250,000 loan.
But a person who is prudent with money would take that £50,000 and invest it. In that way, they will get a return on that new money and they'll safeguard against the negative equity trap. Let's say this prudent person buys some government bonds (a safe investment which is in fact like lending money to the government) which lock the money up for 5 years but give a return of 5%. This person has increased their mortgage but is making money on the extra £50,000 they borrowed: it was borrowed at 4% and invested again at 5%. Again, another example where borrowing is a sensible course of action.
Now let's consider a different example, a business. Let's say this business has a factory that produces chairs. Suppose each chair costs £50 in material and labour to make and can be sold on for £100, making a profit of £50 on each one. If they make 100 chairs in the month, the profits are £5000. The chairs are good quality and in high demand and they have no trouble selling them and are sure they could sell more. But, there's a problem. The factory's fixed costs are £5000 a month (rent, administration etc) so they're only breaking even and can't save up any money to expand their operation and actually become profitable. The obvious solution is to take out a loan and any bank that's half-way decent would be willing to give them that loan, all else being equal.
But it will only make sense to take out this loan as long as the interest payments it incurs are less than profits of the newly expanded business. So let's say £200,000 is borrowed at 5%, i.e. £10,000 of interest needs to be paid per year. Clearly, this will only be worthwhile if overall profits are £10,000, at the very least, which means they'll have to sell at least 200 more chairs per year, and probably a bit more because the fixed costs will rise. If this investment increased their production to, say, 150 chairs per month, then the business will almost certainly become profitable.
The business example again demonstrates that taking out a loan can be a very prudent thing to do, in the right circumstances. This example is of course very well known and is sometimes referred to as gearing, by analogy with changing the gear to keep an engine working in an efficient range of revs: if the factory drops below its current output it will lose money and "stall".
The common element that exists in both the household loan (mortgage) example and business example is that the loan is only worthwhile when the interest that you need to pay on it is less than the return from investing it.
So in the UK economy, if you can borrow money cheaply at the moment (which the UK certainly can) then it is not a stupid thing to do if it is wisely invested into the productive economy. To put it simply, the UK can borrow now and borrow cheaply because interest rates are low: the Bank of England base rate is 0.5% at the time of writing, and it has been at that rock bottom for years now. It is not hard to envisage that this borrowed money can be invested to bring back a return above that base rate of 0.5%. If done sensibly, it can create jobs, fund affordable houses and more.
There's plenty more to say here, such as on the control of currency, money creation and the issue of inflation, but I'll leave that for another blog post. My main point is that borrowing is an economically sensible strategy in the right circumstances. Also, as a wee corollary, it's surprising how much of the unproductive world of banking starts to make sense once you realise there are profits to be made by borrowing money so you can lend it out again at a higher interest rate.
But of course I have taken out loans. I've taken out mortgages for houses. I took out a loan fifteen years ago to buy a car. Taking out these loans made sense. The mortgage helped me get on the housing ladder in 1995, which in the long run has allowed me to spend a surprisingly small amount to live in a decent home. The car was different - it was a necessity to allow me to get to work and so the loan and depreciation were unavoidable at the time.
But I would have preferred to avoid these loans because if I had the money to buy the house or car outright I'd have saved tens of thousands of pounds over the years.
So now, with the UK's economy still bumping along the bottom, as it has for several years, it might surprise some to hear me say, "yes, borrowing money to spend money is a good way to get the economy going again". The immediate retort of many is to say, "if you're in debt, it's stupid to take out more debt to pay it off." It may or may not be stupid, depending on what you do. Let's take familiar example of debt - that associated with buying a house.
A person wishes to take out a loan to buy a house to be the family home. They borrow £200,000 at a 5% interest rate and use that to buy the house. This loan is perfectly sensible (aside for possibly inflated house values) because a home is a necessary thing. The person is essentially renting the £200,000 required to purchase the home by paying £10,000 in interest per year.
Some years later they decide to remortgage for the same amount but at a 4% interest rate offered by a different lender. This means they take out the loan from the new lender and the money from that goes to pay off the original loan. This is a clear example in which taking out a new loan to pay off an existing loan is a good idea because now the cost of the £200,000 loan is only £8000 per year.
But let's imagine when it comes time to remortgage that house prices have increased and the house is now worth £250,000. If they decide to remortgage at 4% then they will be left with £50,000 to spend and still be paying £10,000 per year. This seems like a good deal, the cavaet being that if house prices ever drop and they spend their £50,000, say on a luxury car, then they might get stuck in a negative equity trap one day, i.e. their house can't be sold to pay off the £250,000 loan.
But a person who is prudent with money would take that £50,000 and invest it. In that way, they will get a return on that new money and they'll safeguard against the negative equity trap. Let's say this prudent person buys some government bonds (a safe investment which is in fact like lending money to the government) which lock the money up for 5 years but give a return of 5%. This person has increased their mortgage but is making money on the extra £50,000 they borrowed: it was borrowed at 4% and invested again at 5%. Again, another example where borrowing is a sensible course of action.
Now let's consider a different example, a business. Let's say this business has a factory that produces chairs. Suppose each chair costs £50 in material and labour to make and can be sold on for £100, making a profit of £50 on each one. If they make 100 chairs in the month, the profits are £5000. The chairs are good quality and in high demand and they have no trouble selling them and are sure they could sell more. But, there's a problem. The factory's fixed costs are £5000 a month (rent, administration etc) so they're only breaking even and can't save up any money to expand their operation and actually become profitable. The obvious solution is to take out a loan and any bank that's half-way decent would be willing to give them that loan, all else being equal.
But it will only make sense to take out this loan as long as the interest payments it incurs are less than profits of the newly expanded business. So let's say £200,000 is borrowed at 5%, i.e. £10,000 of interest needs to be paid per year. Clearly, this will only be worthwhile if overall profits are £10,000, at the very least, which means they'll have to sell at least 200 more chairs per year, and probably a bit more because the fixed costs will rise. If this investment increased their production to, say, 150 chairs per month, then the business will almost certainly become profitable.
The business example again demonstrates that taking out a loan can be a very prudent thing to do, in the right circumstances. This example is of course very well known and is sometimes referred to as gearing, by analogy with changing the gear to keep an engine working in an efficient range of revs: if the factory drops below its current output it will lose money and "stall".
The common element that exists in both the household loan (mortgage) example and business example is that the loan is only worthwhile when the interest that you need to pay on it is less than the return from investing it.
So in the UK economy, if you can borrow money cheaply at the moment (which the UK certainly can) then it is not a stupid thing to do if it is wisely invested into the productive economy. To put it simply, the UK can borrow now and borrow cheaply because interest rates are low: the Bank of England base rate is 0.5% at the time of writing, and it has been at that rock bottom for years now. It is not hard to envisage that this borrowed money can be invested to bring back a return above that base rate of 0.5%. If done sensibly, it can create jobs, fund affordable houses and more.
There's plenty more to say here, such as on the control of currency, money creation and the issue of inflation, but I'll leave that for another blog post. My main point is that borrowing is an economically sensible strategy in the right circumstances. Also, as a wee corollary, it's surprising how much of the unproductive world of banking starts to make sense once you realise there are profits to be made by borrowing money so you can lend it out again at a higher interest rate.
Monday, August 26, 2013
Firefox OS ZTE phone
To me, the main attraction of Firefox OS is the fact it is based on HTML5 and CSS and javascript. To those who don't know what that means: those are standards that keep the web open, i.e. you can rely on being able to view and interact with content on a web page on almost any type of PC, tablet or phone, made by any manufacturer, or under an operating system from any vendor. In contrast, mobile phone apps are not like this: android apps work only on devices with Google's android; iOS apps only work on Apple's iThings. But, if you write an app in Firefox OS, it's a simple matter to deploy it as a web app which may be used from any device with a web browser. There are of courses cons against that pro, but that's a discusson for another blog post.
About the time I received my phone, I noticed a message from a well-respected-techy-geeky-sort-of-a-chap, called Jezra, who had also recently acquired the same phone, referring to it as "shitty". Upon asking why, he told me it was because he thought it was hard to get his software on to the phone. Despite my dewy-eyed enthusiasm for this phone driven by open standards, I was soon to discover he had a point.
To start my adventure, I read this article and set about trying to install the Firefox OS Simulator. It's an Add-on to Firefox and should've been a simple matter to install, but I discovered that it's not available for Firefox 17 ESR, which is the default version in slackware 14.0, which I have on my laptop (Patrick Volkerding did provide prompt upgrades to recent Firefoxes, but at 22 he couldn't work around a troubling bug, and decided to go back to 17 ESR for slackware stable). Not wishing to mess with a very stable operating system, I downloaded the linux binary tar ball of Firefox 23.0.1, closed Firefox 17 and ran the newer Firefox binary without installing it - it just sits under my home directory. I was then able to install the Firefox OS simulator version 4.0 as I would any other Add-on, and it seemed to work fine. Soon I had this Hello World app and the Boiler-plate app working without issue.
Next up was the potential "shitty" bit - getting the app onto my phone. Now there are numerous ways of doing this, including via a website, but I wanted to get it done via USB, because that, in my experience, mainly with android, is probably best when developing and debugging an app.
According to the aforementioned article, connecting my phone to my laptop involved creating the file /etc/udev/rules.d/51-android.rules containing the following:
SUBSYSTEM=="usb", ATTR{idVendor}=="19d2", MODE="0666", GROUP="plugdev"
The bit that will vary from phone to phone is idVendor; 19d2 means ZTE. You can look these up online (there's a table somewhere in the android adb documentation), or, you can do what I did under linux, and hunt around in the output of dmesg | tail immediately after plugging the phone in. A restart of udev is recommended after you create the rules file, and the udev folk recommend restarting your system.
Luckily, I happened to have the android adb tool already installed, and from the output of running adb devices, I could see that "roamer2" was listed, which, despite the misleading name, was indeed my ZTE phone.
Now when I fired up the Firefox OS simulator it said "Device connected" and "Push" buttons appeared to the right of my apps. This all looked very encouraging. But, clicking the "Push" button did nothing. Nothing happened in the simulator, not even an error message, and the phone didn't display the connection prompt depicted on the page of instructions (mentioned above). Further down that page, a command line method of pushing apps on to the device was described. OK, I thought, if I try that, even if it doesn't work, at least I'll get some clueful error messages, such is the advantage of the CLI.
The CLI tool is called make-fxos-install, and it needs both the android adb, which I had installed, and xpcshell from Mozilla's xulrunner, which I did not. Luckily, I found xulrunner listed on slackbuilds.org and so within a few seconds I had sbopkg downloading and then building xulrunner.
The source downloaded was 80MB, so I was expecting the compilation to take a while on my rather slow laptop. To pass the time, I decided to play around editing the various demo apps and testing the results out in the Firefox OS simulator. I immediately hit problems. After even the tiniest change to an app's manifest, the simulator kept complaining it couldn't update the manifest cache and something about an NS_ERROR was mentioned amongst other gobbledegook. Frustrated, I decided to launch VirtualBox and try out the simulator under Windows to see if the trouble was caused by some linux-related issue (which I thought very odd and rare). To my astonishment, Virtual box failed to start and also gave an NS_ERROR. I thought for a moment that my dual use of Firefox had screwed up the system. I rebooted the system and found X Windows wouldn't start.
I was just about to hurl myself out the nearest window (OK, I was on the ground floor), when I noticed a little snippet that mentioned a write to /tmp/something had failed. I quickly realised that the disk had filled because of the xulrunner compilation.
I then set xulrunner building on my more powerful desktop machine which has a much larger and less full hard disk. Unfortunately, since I had done this whilst sat at my laptop downstairs, my wife inadvertently shut-down the desktop machine upstairs once our son had gone to bed. Grumble. Once my third attempt at building xulrunner had finished without incident, I scp'd the .txz slackware package over to my laptop and installed it.
I then needed to edit the Makefile that comes with make-fxos-install to specify the path to the xpcshell executable. But it didn't work. After a fair bit of pulling of what little hair I had left, I noticed that the xulrunner version I had built was 15. Guessing this was rather old, on the basis that version numbers track firefox, and unable to face any more compiling, I went to mozilla ftp site and found linux 64bit binaries for xulrunner version 23, which matched the firefox I was using for the simulator. I downloaded the xulrunner-sdk (which is where xpcshell is to be found these days) and stuck it in a directory underneath the Makefile.
But still xpcshell refused to work, complaining about a missing library, but this line in the Makefile got it to play ball:
LD_LIBRARY_PATH=xulrunner-sdk/lib xulrunner-sdk/bin/xpcshell
After a little RTFM I found the command to push the helloworld app to my phone was:
make FOLDER=/home/MyUserName/dev/firefoxos/examples/firefoxos-helloworld packaged install
Did I finally get to rejoice and dance about the room at the sight of this little app installing on my phone? No. Grrrrrr! It complained that it couldn't bind to port 6000. After looking at the Makefile I understood that the android adb tool was trying to forward port 6000 on my laptop to 6000 on my phone. Right. So if local port 6000 was unavailable, for reasons unknown, perhaps I could just tell the Makefile to use 6001 and forward that to port 6000 on the phone, which simply involved a tiny tweak to the thoughtfully constructed Makefile, namely:
PORT_LOCAL = 6001
And now executing the make command above worked! The phone prompted me to accept the connection and the app installed and ran. Phew. But, I wasn't quite satisfied; I had to know what port 6000 was being used for on my system. Executing the command
lsof -i tcp:6000
gave me no output, from which I concluded that no process was actually using the port. But this command
nmap localhost
told me that port 6000 was in some way being claimed by X windows. A bit of web research told me that it was needed for remote sessions, which I never use. To stop it using this port, I shut down X windows and started it again with
startx -- -nolisten tcp
I put PORT_LOCAL back to 6000 in the Makefile and found I could now push to the phone. More than that, I found that the "Push" button in the Firefox OS simulator now worked too.
In summary, to get apps to push to a phone from the FireFox OS simulator, in linux:
- use a recent version of Firefox (23.0.1 worked for me)
- create the udev rule file
- make sure port 6000 is open
- get a recent version of xulrunner sdk which includes xpcshell (23.0.1 worked for me)
- get the android SDK which includes the adb tool
- edit the Makefile to specify paths to xpcshell and adb, and libraries they need
I hope the details above prove useful to someone, and if not, I hope you had a geeky chuckle or two at my FireFox OS misadventure. Hopefully I'll soon write some posts about how easy it is (stifles laugh) to write web apps.
Thursday, June 13, 2013
Writing
I've enjoyed writing since I was a small boy. At school I remember writing Dr Who stories on old computer paper from my Dad's work. I also wrote a series of Goon Show-like parodies of Star Trek which seemed to go down well with folk at my school. Later on writing became a central part of my job when I worked at the Open University, writing science courses. I learned a huge amount doing that, especially from the diligent and patient science editors that would pore over my work, and sit with me, talking me through their comments sentence by sentence.
These editors made sure the writing was clear, correct in grammar and free of typos whilst also taking care not to disturb the style of writing. This surprised me because the text books we wrote involved several authors and I expected part of an editor's job was to pull it together and bring more consistency of style between chapters (writing style, not font sizes etc which was strictly controlled). I distinctly remember how one of them explained his role to me: "We do not want to make the book homogenized. Students generally like a change of tone and change of pace. Besides, you can't edit those things anyway."
It was gruelling at times. I remember being sat in one editor's office as she patiently explained when I should use 'which' rather than 'that', e.g. "This is the train which takes you to Newcastle." versus "This is the train that takes you to Newcastle." Which one is correct, dear reader?*
But, there's no other way to learn other than to write and rewrite and seek reactions from readers, then write something else and revise, and so on. I find it very hard to progress a piece of writing working by myself. Very often it would start well enough but then I would get stuck during the revisions, often abandoning the piece. So I consciously decided to write my first drafts with more care, rationalising that if I could do that, then I'd have less revising to do. But this made matters worse. My writing now slowed to a snail's pacecrawl, was thoroughly unenjoyable and the turgid result could not be rectified by any amount of disciplined editing because the style and tone were compromised. Had I stopped to think about it back then, I might have realised that that's exactly what the editor at the OU had meant.
So, for many years I hardly wrote at all. It was even a welcome relief when the rise of mobile phones and twitter meant that emails became shorter and more sloppy: I could spend less time trying to write them well, or so I thought for a while.
But I missed writing. I wanted to return to it, even though it seemed like a futile waste of my time. I had ideas that needed expressed, but repressed them because I couldn't think how to get them out. After writing nothing for years, the urge strengthened sufficiently for me to bang out a few short stories, but I didn't feel happy enough with them to let others read them. And I was slow. One story took me more than two years to complete.
I remember sitting in my car outside a supermarket listening to BBC Radio 4. The author Frederick Forsyth was being questioned about how he wrote. To my surprise he typically wrote six pages a day, but stressed that it was very important to get into the discipline of writing every day. That didn't seem like a lot to me. What did he do for the rest of the day? Presumably gathering the experiences and knowledge that gave him something to write about.
I investigated the habits of other authors and I found many did something similar. One author, I forget who, had a strict rule of writing only two pages per day and never any more. But, as Forsyth had also emphasised, even at this rate you can yield a novel-length work in a few months.
The details of how authors work varies tremendously, but I'm not aware of one that advocates binge writing. Little and often yet disciplined and regular was the recurring advice. Approaches to revising vary greatly, but most recommended that you should not aim for perfection on a first draft.
Then I happened across NanoWriMo.org, which encourages people to produce 50,000 words from scratch in the month of November (continuing an existing work is not allowed). At first I was skeptical. Isn't this just going to produce 50,000 words of drivel and probably verbose drivel at that, so as to produce the requisite 1667 words per day?
The answer is sometimes, perhaps even in most cases. But that's missing the point. What it achieves is to encourage a person to write with discipline and work towards a goal. In my case it helped prove to myself that even with two children, a day job and many other calls on my time, I could get those 1667 words out almost every day.
The quality of my writing varied tremendously of course, with some days being awful, most days being OK and a few being good. But I realised it didn't matter. As long as I was relaxed when I wrote so that my writing was natural rather than forced, what I produced was generally acceptable in that it expressed my ideas well enough so that someone else might understand them. It took me another year to get the word count to 80,000 and since then seven months have elapsed during which I have performed two editing passes over the whole thing. It's still not finished. On the second editing pass I decided to add several new sections and remove a few that weren't working. Once I've done that, I hope, it'll be ready to be read by someone else.
It's certainly the case that my writing has improved over this time and with it I'm probably being more economical with words, though I still struggle with repetition (She did that... she did this) and banishing superfluous words, especially adverbs (He just stood there, impassively). I'm also somewhat blind to typos and rely on others to help me find them. But I'm realising that staying relaxed and writing a little too much and editing out is better, for me at least, than trying to write precisely under conscious invigiliation. Also, most readers, at least ones who are not authors, will skate over pointless adverbs and other minor flaws as long they want to know what happens next.
If I had to summarise it in one pithy phrase, then I'd settle for "be as simple as possible, but no simpler".
You can read some of my writing here at http://www.mcnalu.net/writing
*That one. :)
These editors made sure the writing was clear, correct in grammar and free of typos whilst also taking care not to disturb the style of writing. This surprised me because the text books we wrote involved several authors and I expected part of an editor's job was to pull it together and bring more consistency of style between chapters (writing style, not font sizes etc which was strictly controlled). I distinctly remember how one of them explained his role to me: "We do not want to make the book homogenized. Students generally like a change of tone and change of pace. Besides, you can't edit those things anyway."
It was gruelling at times. I remember being sat in one editor's office as she patiently explained when I should use 'which' rather than 'that', e.g. "This is the train which takes you to Newcastle." versus "This is the train that takes you to Newcastle." Which one is correct, dear reader?*
But, there's no other way to learn other than to write and rewrite and seek reactions from readers, then write something else and revise, and so on. I find it very hard to progress a piece of writing working by myself. Very often it would start well enough but then I would get stuck during the revisions, often abandoning the piece. So I consciously decided to write my first drafts with more care, rationalising that if I could do that, then I'd have less revising to do. But this made matters worse. My writing now slowed to a snail's pace
So, for many years I hardly wrote at all. It was even a welcome relief when the rise of mobile phones and twitter meant that emails became shorter and more sloppy: I could spend less time trying to write them well, or so I thought for a while.
But I missed writing. I wanted to return to it, even though it seemed like a futile waste of my time. I had ideas that needed expressed, but repressed them because I couldn't think how to get them out. After writing nothing for years, the urge strengthened sufficiently for me to bang out a few short stories, but I didn't feel happy enough with them to let others read them. And I was slow. One story took me more than two years to complete.
I remember sitting in my car outside a supermarket listening to BBC Radio 4. The author Frederick Forsyth was being questioned about how he wrote. To my surprise he typically wrote six pages a day, but stressed that it was very important to get into the discipline of writing every day. That didn't seem like a lot to me. What did he do for the rest of the day? Presumably gathering the experiences and knowledge that gave him something to write about.
I investigated the habits of other authors and I found many did something similar. One author, I forget who, had a strict rule of writing only two pages per day and never any more. But, as Forsyth had also emphasised, even at this rate you can yield a novel-length work in a few months.
The details of how authors work varies tremendously, but I'm not aware of one that advocates binge writing. Little and often yet disciplined and regular was the recurring advice. Approaches to revising vary greatly, but most recommended that you should not aim for perfection on a first draft.
Then I happened across NanoWriMo.org, which encourages people to produce 50,000 words from scratch in the month of November (continuing an existing work is not allowed). At first I was skeptical. Isn't this just going to produce 50,000 words of drivel and probably verbose drivel at that, so as to produce the requisite 1667 words per day?
The answer is sometimes, perhaps even in most cases. But that's missing the point. What it achieves is to encourage a person to write with discipline and work towards a goal. In my case it helped prove to myself that even with two children, a day job and many other calls on my time, I could get those 1667 words out almost every day.
The quality of my writing varied tremendously of course, with some days being awful, most days being OK and a few being good. But I realised it didn't matter. As long as I was relaxed when I wrote so that my writing was natural rather than forced, what I produced was generally acceptable in that it expressed my ideas well enough so that someone else might understand them. It took me another year to get the word count to 80,000 and since then seven months have elapsed during which I have performed two editing passes over the whole thing. It's still not finished. On the second editing pass I decided to add several new sections and remove a few that weren't working. Once I've done that, I hope, it'll be ready to be read by someone else.
It's certainly the case that my writing has improved over this time and with it I'm probably being more economical with words, though I still struggle with repetition (She did that... she did this) and banishing superfluous words, especially adverbs (He just stood there, impassively). I'm also somewhat blind to typos and rely on others to help me find them. But I'm realising that staying relaxed and writing a little too much and editing out is better, for me at least, than trying to write precisely under conscious invigiliation. Also, most readers, at least ones who are not authors, will skate over pointless adverbs and other minor flaws as long they want to know what happens next.
If I had to summarise it in one pithy phrase, then I'd settle for "be as simple as possible, but no simpler".
You can read some of my writing here at http://www.mcnalu.net/writing
*That one. :)
Thursday, January 3, 2013
Faster USB sticks
I read in Linux Format recently that it was wise to make sure partitions and filesystems were aligned with the structures found on the flash drives. Flash devices come set up in this way, but invariably with archaic FAT32 filesystems and I wanted to use a more up-to-date linux filesystem. I didn't know how to ensure alignment, so I embarked on a course of reading across the internet.
Unfortunately the information I found was generally very confusing, some articles told me what to do, but not why, and others went into bamboozling levels of gritty detail, and one or two even told me aligning wasn't worth the effort on USB2.0 drives. I'll provide links of the sites I found most helpful at the end, but here's my account of what I did, starting with the USB stick formatted in a completely misaligned way with three partitions - the result of my fettling before I knew any better.
First I plugged the drive into my laptop, then, as root:
mount /dev/sdb3 /mnt/hdI tested the read speed with:
hdparm -t /dev/sdband this told me the speed was 23.8 MB/s - I did a few tests and found this varied by no more that +/- 0.05 MB/s. I then mounted the third partition, which had an ext2 filesystem on it, with:
mount -o sync /dev/sdb3 /mnt/hdThe sync option means that the writing isn't buffered - if it is buffered then the test will give you a misleadingly high speed because it will only include the time take to write to the buffer in memory. I then tested the write speed with:
dd count=100 bs=1M if=/dev/zero of=/mnt/hd/testThis copies one hundred 1 megabyte blocks of zeroes to a file called test. The result was a paltry 893 kB/s.
I then set about deciding how I could reformat the drive to get better performance, particularly for writing. I wish I'd kept a note of the original partitioning of the stick as that would give me vital clues about how to align partitions and filesystems. After searching the web, the best I could do was guess that the USB stick was likely to have a 4kB or 8kB page size and a 512kB erase block size.
To be honest, I haven't really had time to get to grips with the details, so please excuse all the uncertainty, but I'll try and set out my patchy understanding. The erase block size is significant because if the filesystem requires, say, 100kB to change, then the flash drive electronics will have to wipe at least a 512kB block and rewrite it all. I say "at least" because if that 100kB happens to cross the boundary between two adjacent 512kB blocks then both blocks will have to be erased. These erase blocks are grouped together to form erase block groups which, I understand, are typically 4 MB in size. This means that partitions boundaries should be on multiples of 4MB on the device.
The page size represents the smallest unit that can be addressed on the flash drive. I'm not sure what impact this has, except that clearly you don't want the partition or filesystem constantly straddling the "pages". But, if you're using a block size of 4kB in an ext filesystem and you've aligned the partition to 4MB there should not be a problem. With modern multi-GB drives 4kB will usually be the default block size used by the mkfs command and it's also the maximum permissible block size on typical systems.
From this you can get some idea of how aligning data with these blocks can help with performance. Also, because it reduces the number of writes to the disk, alignment can also help increase the life of a flash drive. One big problem in aligning is that there is no easy way to find out the page or erase block size for a particular stick because such information is often not published by manufacturers. The best you can do is look at the original factory formatting of the stick or search around on the internet to see if someone has measured it. (There are tools to do it yourself, and I tried flashbench which was intriguing but unfortunately it gave inconclusive results for my USB stick.)
I side stepped the issue of partition alignment by not creating any partitions at all (zero is a multiple of all numbers!) In linux it is quite permissible to create a file system directly on the device. I did this with the command:
mkfs.ext4 /dev/sdb -b 4096 -E stride=128,stripe-width=128This creates an ext4 filesystem on device sdb (sda is the hard drive of my laptop). The -b 4096 specifies the use of a 4kB block size (I had tried 8192 but that is possible only on a few specialised systems. Next, I tried to make sure the filesystem was likely to be aligned with 512kB blocks by setting the stride and stripe-width both to 128 because 128 times 4kB is 512kB. These two parameters are more often used to make sure filesystems work efficiently with RAID arrays, but as we're only dealing with one drive here I understand that they should be set to be equal to each other (though I confess I don't know why).
Finally, to save on disk writes, I turned the journal off:
tune2fs -O ^has_journal /dev/sdb
The downside of this is that the disk will be more likely to suffer from data loss if powered down or removed incorrectly. (This option can also be set when creating the filesystem I believe so it should be possible to combine the above two commands.)
With this done, I performed the above tests again and found that the read speed was almost the same at 22.8 MB/s. The write speed was now 4.7 MB/s - a five-fold increase. Job well done! That wasn't all, if I repeated the test, the write speed seemed to go up reaching a little over 10 MB/s after 4 runs of the test. I'm not sure why this might be, but it could be to do with some optimisation in the onboard electronics in the flash drive combined with the artifice of the test in writing long stretches of zeroes.
But, I'm not quite comparing like with like here. In the "before" tests I was writing to an ext2 partition, so is it possible that some of the speed-up is due to inefficiencies of ext2 compared to ext4? To test this I created an ext2 partition directly on the disk with this command:
mkfs.ext2 /dev/sdb -b 4096(The stripe and stride width options are not available for ext2.) The first thing that struck was how slow this was compared to the ext4 creation - 30 minutes compared to a few minutes. The tests revealed that performance was back to almost exactly what I found on the original ext2 partition with its completely misaligned partitions and filesystems.
Finally, I wanted to see if it was ext4 that was behind the speed gain, or if it was the setting of the stripe and stride option that brought about the improvement. To test for this I issued this command:
mkfs.ext4 /dev/sdb -b 4096and did not disable the journal. The write speed was now 3.6 MB/s which seems to suggest that ext4 is responsible for most of the speed gain. The continued flashing of the stick's LED after the test had ended suggested that the journal was keeping the stick busy at least to some extent and this was further confirmed when I umounted the stick and the flashing ceased (this ruled out the activity as the flash doing wear-leveling maintenance. Interestingly, there was a slightly improvement in read speed, but no more than a few percent.
So, in conclusion, the most important improvement seemed to be just to use ext4 instead of ext2. Disabling the journal helped a little too, as did setting the stripe and stride options on ext4. But, I didn't find any evidence that alignment helped improve the performance of this drive.
Here are some links:
- Good starting point: http://linux-howto-guide.blogspot.co.uk/2009/10/increase-usb-flash-drive-write-speed.html
- A very thorough and detailed technical account: http://lwn.net/Articles/428584/
- Somewhere between the above too in technical detail - some good diagrams: http://blog.nuclex-games.com/2009/12/aligning-an-ssd-on-linux/
Subscribe to:
Posts (Atom)