The Twitter API supports UTF-8 encoding. Please note that angle brackets (“<” and “>”) are entity-encoded to prevent Cross-Site Scripting attacks for web-embedded consumers of JSON API output. The resulting encoded entities do count towards the 140 character limit.

Does anyone notice the weirdness there? Apart from the MAGIC_QUOTES smell.

If I were feeling pathological, I could tweet a message of 140 characters all between the Unicode code-points U+010000-U+10FFFF. I think that would end up as 560 bytes. And I think that would be all fine with Twitter. Which is another way of saying that Twitter would, I assume, be happy to exceed 140 bytes for a message if it were written in, say, Japanese.

By contrast, while on my pathological holiday from good sense, I would only be able to tweet a message of 35 angle brackets – hence 140 characters, 140 bytes in UTF-8 – because the encoded angle-brackets count toward the number of characters. Seems a bit backwards doesn’t it?

Does anyone know the reasoning here? Or are the docs at fault?

Back to the angle-bracket quoting. Just as the PHP folk are finally ending their own embarrassing journey through that silliness, it looks to me like Twitter are now making a similar mistake. JSON should safely encapsulate angle-brackets, so perhaps I don’t understand the problem that they are trying to solve?

One more question: what if I tweet “&gt;”? When using the API, can that be distinguished from a “>”?

(You might have noticed that I’ve have so far been too lazy to experiment with all this stuff; I just wanted to write it down before I forgot. I’ll add a comment if I get the time to play.)

1. iTunes keeps skipping tracks half way through. It will suddenly start playing the next track for no obvious reason. Restarting stops this for a while.
2. The keyboard focus keeps going… elsewhere – into the void – as I type in an X11 window. The one other person I’ve asked about this has the same issue.
3. Spaces is, well, just a bit crap compared to the multiple-desktop systems that you can find for X (as in X-Windows, or X11).
4. Window/application focusing and switching are sometimes just wrong. For example, Cmd-Tab switches between the most recently used applications, but Cmd-` (back-tick) cycles through the windows of the current application. Good luck to you if your application has, say, 10 windows open.
5. Terminal is broken. I gave up on it a long time ago, hence the need for X11.
6. MacPorts is great, except for port, the package manager, which is actually less useful than, say, tar. Why o why o why did they write yet another package manager? Fink at least had the good sense to use APT, but I don’t use it because I prefer the packages in MacPorts.
7. Sleep. Specifically safe-sleep. Which doesn’t work on my MacBook Pro. If I use safe-sleep (which is the default), the laptop will take several minutes to go to sleep, several minutes to wake up, and every 3 or 4 times will simply not wake up. Sometimes it will wake up but no login dialog box will appear, which means I have to put it back to sleep (and wait), then wake it up again (and wait and hope), and if the dialog box doesn’t appear, it probably means I need to hard-reboot. It would have been quicker to just pull the battery out. Actually, I disabled safe-sleep on Tiger, so maybe it works now on Leopard, but I’m not in the mood to try.
8. External monitors. I have a nice 30″ Apple Cinema Display, which is simply great. However, using it with my MBP is a PITA:
   1. For best use, start with MBP switched off (cold, not asleep).
   2. Plug in monitor.
   3. Turn on MBP, and quickly shut the lid.
   4. Don’t open the lid.
   5. Don’t open the lid.
   6. If you open the lid, Mac OS will add the laptop screen to the available display space, then rearrange your windows, backdrops, dock and menu bar. And then, if you close the lid, the MBP will go to sleep, then take 5 minutes to wake up and show the login dialog box on the 30″ display. Then all your windows will be in different places and at different sizes.
   7. Don’t plug/unplug the monitor unless the MBP is switched off. Doing so will work 2 times in 3, the other time it will leave your machine not crashed, but not working either. Hard-reboot required. If you want to take your supposedly mobile laptop computer into the other room, think again.

I am sure there are more things that irk me, but that’s all for now.

There are some things I do like:

1. Mail. It works better than any MUA I know. I just wish it wasn’t tethered to Mac OS X, highlighted URLs properly (it doesn’t understand ~ in URLs, which is, oh, only in half the URLs I get sent every day), and came with PGP/GnuPG working out of the box.
2. Time Machine. Fantastic. I love it. It makes me feel so much better. I just hope it actually works when I need it.
3. iPhone. Okay, I could use MS Windows to work with my iPhone, but… hah, I’m not that bloody stupid. Anyway, this is not really a feature of Mac OS, it’s just a requirement. Which is, actually, somewhat annoying. And the new iPhone 2.0 software is a bit shit too.

I like 2.5 things (the iPhone “feature” gets 0.5). And I’m sure there aren’t any more.

I now realise that I would rather use Linux (again; I used to use Linux all the time on the desktop). But Ubuntu (which is my distro of choice) is a PITA all of its own to install on Apple hardware, so I’m just going to persevere with Mac OS X for now.

That, and because I have an iPhone. Grr, Apple, you now make me feel locked-in to your platform, and I don’t like that.

lsof -p $(ps ux | egrep [/]opt/local | awk '{print $2}' | xargs | sed 's/ /,/g') | fgrep opt/local | awk '{print $9}' | sort -u | xargs port provides | sed 's/.*: //' | sort -u | xargs port -f upgrade


The command line rules!

It’s meant to rebuild every MacPorts package I’ve got installed that is loaded when I start XDarwin (the xfree86 port in MacPorts) in a vague attempt to fix the completely broken mouse pointer in X.

In a company far far away, I used to be a Java developer. At that time I thought that applets could be a good thing for the web. Since then, and for most of this millennium, I’ve been somewhere between indifferent and blisteringly annoyed when encountering Java applets. I silently hoped they would go away.

But they just won’t die.

Recently I joined Facebook, and they use a crappy upload applet for photos. Which reminded me of the crappy upload applet that comes with Gallery. These might be the most beautifully crafted objects, whole Versailles of Java, but during the time the ruddy thing painfully loads I can’t use the browser, and the OS even mentions that things are not responding. Okay, Versailles was not built in a day, so I’ll go and feed the donkeys or something and come back when it’s done. But, of course, it’s not beautiful as promised. It’s an eyesore, a pig to use, and breaks just after I’ve invested 20 excruciating minutes choosing photos of family barbecues.

So I’m voting with my browser preferences, and I vote for no more Java on the web. Applets were cool for a month or two, but have been a mercilessly flogged but nevertheless dead horse ever since.

Flash, on the other hand, actually has a use. Video. After over a decade of squabbling between Real, Apple Quicktime and Microsoft SomethingMediocre, it took a vector animation program to pretty much solve cross-browser video.

That’s great. But I really really do not want to watch another whizzy website Flash-splash page or another Microsoft disinfomercial next to the news. So Flash might be next to get the chop.

I’ll be using Lynx before the year’s out.

I’ve got about 1000 tracks in Apple Lossless (.m4a) format, all ripped from my CDs. Tools like MPlayer now support it, and Amarok can play them, so why bother converting them? Well, for a long time they were not supported so I could only listen to them on my Mac. I’ve also realised, if I’m going to keep a music collection on my computer, that the format I choose better be open so I can still listen to my tunes in a decade.

You’ll need the following:

• Python 2.4 or later.
• Mutagen is a pure-Python meta-data library, and I wrote the script using version 1.8.
• MPlayer is used to convert from Apple Lossless to WAV. I was using version 1.0_rc1.
• FLAC of course, to encode the WAV. My machine has version 1.1.2-r8 installed.

So far I’ve only tested the script with one Moloko album on a Linux machine. It’ll probably work just fine on any Unix-like OS, and may just work on Windows if you peddle hard. I’m going to convert all 1000 tracks soon, but first I need to find some disk space. My advice: take backups, because there are no warranties, and, though I don’t mean to scare you, the pieces are yours to keep if things break.

Happy converting, and try the magic --help switch if you’re lost.

[Edited to fix broken link, 2009-07-06]

In other news, I want to apologise for my content-free blatherings about the Apple iPhone. I don’t think anyone reads this blog so I’m talking to myself, but I want to make my contrition public even so, for future readers :-)

Apple Mail can automatically delete old junk mail, but I also use KMail and SquirrelMail, and it’s often a long wait between the times I load up Mail, so I need something else. It requires Python 2.4 (perhaps earlier, but I’ve not tried it), and that’s it. Run it with a --help to get started.

Incidentally, the script is also in a Bazaar repository, and you can branch it if you want:

bzr branch \

http://www.premolo.net/~gavin/Scripts

Right now the script doesn’t care about SSL (I run it on the same box as my Courier IMAP server) and it uses CRAM-MD5 authentication which may not be to your taste. It’s not had a lot of testing either, so be careful, and take some backups first :-) It’s in the public domain though, so happy hacking, and I’m more than willing to merge patches.

The easiest way to install, say Alfie, is to use easy_install from setuptools:

easy_install premolo-alfie

I’ll be adding in script support for running Alfie from the command line, but in the meantime you can use this slightly unwieldy incantation:

python -c 'from premolo.alfie import template; from sys import argv; template.main(*argv[1:])'

I’m working mostly on Alfie at the moment, and hoping to get a 1.0 release out the door in the first quarter of 2007.

First, some context.

I recently launched a dating website for Luxembourg. “Oh no!” you scream, “Not another bleedin’ dating website!”. No apologies though; Luxembourg needs some loving, and I was the man for the job.

As part of my preparations I wrote a few libraries to flex my Python muscles, which is to say that I didn’t want to write any namby-pamby front-end code just yet. I’m a real hacker goddammit and I’m going to build me some libs.

So I wrote a templating system, some tools to wrap around Apache/mod_python, and collected several other little things into a common package.

I also needed an object-relational mapper, or ORM if you’re short of time and/or breath. I kept cutting my fingers on the sharp edges of SQLObject, and the sheer volume of impenetrable documentation for SQLAlchemy was intimidating, so I wrote my own. In fairness, the documentation for SQLAlchemy was probably fine, and SQLObject could have been fixed, but I wanted to do it my way, being a real hacker and all.

And thus began the journey which led me to conclude that I’m not going to use MySQL any more.

I’m not going to actually relive that journey with you. It was long and had lots of swearing. My wife can let you know all about the frantic desk-slamming, the irritated pacing, and the endless cups of tea. I did a lot of cursing during my skirmishes with mod_python too, and my choice of (human) language was generic/base enough that my wife probably could not discern the MySQL-induced tantrums from the mod_python ones.

I didn’t pull my hair out though. That’s because I hardly have any, and standing in front of a mirror trying to find strands to tear out with tweezers gets in the way of the ‘moment’.

Back to the point, my beefs with MySQL.

Doctor, …

CTRL-C does not do what it says on the tin

Firstly, and this is a bit petty, but when I type CTRL-C in the mysql command-line client, the client exits. This is really annoying, because I have learned that CTRL-C in every other interactive (*nix) command-line means either (a) ditch the current line, or (b) cancel the current command. There is no “(c) Exit”, because it doesn’t exist. Except for MySQL, where you are instantly dumped back to your shell, transactions turned to dust, and left with that bubbling sensation at the top of your head as the blood boils and takes another couple of months off your life.

REFERENCES

Declaring a column with a REFERENCES clause has no effect. In the words of a friend, this makes the baby Jesus cry.

You may ask “Why is this a problem?”, and you’d be right to ask. “It’s there for compatibility with other dialects of SQL” is a good answer, but not good enough. “It’s a feature that will be introduced later on” is another reply, to which I laugh heartily, then shed a small tear of my own. The problem is that it is misleading and creates a false sense of security. People believe that they have an extra tool to maintain the consistency of their data, but alas, it is not so. MySQL is a database, and as such it should not go playing fast and loose with the data that it holds.

This is another example of unexpected results, but more fundamental than the CTRL-C thing. I know it’s in the docs, but my brain is not big enough to remember every gotcha in the book. Someone else said this first, but software should either work, or fail hard. MySQL should not accept my request; it should totally barf on my shoes. Perhaps there should be an option to ignore these clauses, but that should be the thing that you have to look in the manual to discover, and the default should be to either respect them or throw them back in your face.

Foreign keys

I first learnt about relational databases on Oracle 7, so I quickly learnt to appreciate foreign keys and all other kinds of constraints. I could not get enough of the lovely little hairy fellows. Of course, there is a limit to how well they can keep your data in pristine condition, but they do help to catch problems early, especially during development.

So I was glad to see that MySQL supports foreign keys on InnoDB tables, and have eagerly used them. But they have shortcomings.

Like, not actually working.

I restored some invalid data from a dump, just for fun (actually, I was being stupid, but to admit that would make me appear fallible, which, as we both know, I am not). It was invalid because it should have violated at least one foreign key constraint, but MySQL didn’t notice. At first I thought that MySQL had probably disabled constraints during loading, then had problems re-enabling them at the end, and not told me about it.

But the keys were enabled, and I was unable to add new invalid data to the table. I deleted some of the invalid data and tried to re-insert it, but that was denied too. Hmm.

Data consistency is bread-and-butter stuff for a relational database, and I subverted it unexpectedly. If a constraint is enabled, I fully expect that nothing violates that constraint, and it’s not unfair to insist on that.

Foreign keys in MySQL are still useful, but they are no guarantee that the data is consistent. And to compensate, we have to write some scripts to verify our data, one of the things that we hoped to avoid by using foreign keys.

Okay, fair enough, this is probably a bug. At least, I hope it’s not a feature. But this is core functionality, and a gaping hole like this makes me tremble like a chicken at a KFC.

Default values

mysql> CREATE TABLE a (
    ->   b VARCHAR(10) NOT NULL,
    ->   c VARCHAR(10) NOT NULL,
    ->   d INT NOT NULL
    -> );
mysql> INSERT INTO a (b) VALUES ('b');
mysql> SELECT * FROM a;
+---+---+---+
| b | c | d |
+---+---+---+
| b |   | 0 |
+---+---+---+
1 row in set (0.00 sec)

Whoops, I forgot to set values for the c and d columns. But how cute, MySQL has filled them in for me. That’s why it’s called MySQL, because it always looks out for me, aah.

NO! I set them NOT NULL with no default so that I would be forced to enter a value. If I don’t enter a default, don’t guess one for me. Tell me I’m an idiot and get me to think again. Why is there an assumption that the empty string or zero is good for me, now that I’ve rejected NULL?

“I don’t want to work for NULL euros an hour!”

“Okay, let’s put you on our ZERO euros an hour rate instead.”

Roll it all up

You’ve probably seen the pattern. Most or all of my problems with MySQL comes from differences between what I expect will happen, and what actually does happen. But I know these quirks now so they won’t cost me much development time any more, and I’ve built my ORM to cover the most obvious cracks.

But something still bothers me. The water still isn’t clear and I’m afraid there’s a slimy green monster in the depths. What quirks do I not know? Will they one day bite me on the rear like a three-jawed radioactive toad on an arse biting frenzy?

The default MyISAM engine lacks foreign keys and transactions, but InnoDB feels tacked on, and foreign keys still don’t work properly. MySQL tries to please everyone by accepting, yet ignoring, things like REFERENCES clauses. When inserting it has a habit of choosing default values for NOT NULL columns instead of telling me I’ve forgotten to specify a value. In trying to be accommodating it actually ends up surprising and frustrating.

Bizarrely enough, I still like MySQL. In the end it’s worked well for me. The performance is good, and it is easy to maintain and backup. But I won’t be using it again. When storing mission-critical data it is vitally important to do the right thing, and MySQL often does not do the right thing.