Quarter Life Crisis

X.5 Time Machine¶

9787 words on Mac OS X 10.5 Leopard

Time Machine is the backup technology Apple introduced in Mac OS X.5. It is possibly the biggest new feature of the operating system. And I consider it to be a great step forward. In what follows I look at backups in general, a number of details of how Time Machine works and their ramifications as I can make them out today, after a few months of usage.

Backups

Backups are a sore topic. On the one hand, thinking about them is a painful and dull endeavor. On the other hand, not having a backup when shit hits the proverbial fan can be a disaster. The most important thing on a computer are your data. It’s not the machine itself. It’s not the operating system. It’s your files you will be worried about. Your work, your collected resources and work environment, your e-mail of the last decade or two, your photos and possibly your music collection or the playlists you created from it. Many of these things cannot be re-created if they were lost others would demand a massive effort to re-create which you wouldn’t undertake because it exceeds the ‘worth’ of the data in question.

Unfortunately many people aren’t aware of all this. They work with their data every day and spread it across machines and systems without much care. Parts of their e-mail will be at some web mail provider, other parts of their e-mail are still at their old web mail provider - each provider having a different state and subset of their address book. Phone numbers are stored in the mobile phone only. And documents reside in the folder the application creating them suggested as a default. The documents created after 2005, that is, because back then old computer with all the older documents on it was stolen…

Often people don’t worry too much about this, simply because they aren’t obsessive compulsive. But everybody seems to have weak points. Once the half finished dissertation or the wedding photos go missing because of a broken hard drive, careless operation or theft, panic strikes. And the count of people who become aware of backups increases.

Unfortunately, making backups is a bit of a hassle and thus of the few people who have a ‘backup strategy’ even fewer actually stick to it for a long time. It’s tedious, it’s boring and there is no day-to-day benefit from it as long as nothing goes wrong. Without ease-of-use and automation, backups don’t happen.

Once you start thinking about it you will discover many potential points of failure for a backup solution. Things you need are:

• Awareness of the need for backups.
• A mechanism that is easy and foolproof to set up.
• Automatically scheduled running of the backup that works reliably.
• The ability to read and recover your backup when your data are lost.
• Bonus: The ability to restore your data in a quick and foolproof way to a working condition. By Murphy’s Law you are going to be in a stressful tight-schedule situation when you need to do so.

None of these points is trivial to achieve. And - optimistically speaking - Time Machine is a brave attempt to address all these problems: Its mere existence raises awareness about backups. It creates and maintains backups automatically. It does so in a relatively simple way. And it makes restoring data easy.

Apple and Backups

All that said, Time Machine is also a product by Apple. And - as far as I can tell - Apple never went out of their way to protect their users’ data. The quality of their applications’ backwards compatibility varies widely (the OS X Address Book will export contacts [up to the photos associated to them] to the standard VCARD format, OS X Mail has no export feature at all, Mac OS X.5 doesn’t voluntarily play System 7 sound files, …). If Apple were self-confident enough to worry about their users’ data more than about locking their users in to peddle more machines, I’d certainly start off with a more positive impression.

While Apple’s software isn’t generally destructive, they have had their run of installers deleting users’ files and they also had the .mac Backup application gaining a reputation for not being something you’d want to entrust your data to. There were reports about people being not being able to restore data from backups (i.e. failure at its single most crucial function) a few years back. That seeds a bad reputation which is deadly for a backup tool - particularly for one by a tight lipped company which will neither admit errors nor document the changes they make in updates.

And from a good friend of mine I heard a possibly worse report about .mac Backup. He had a hardware failure of his hard drive. Which he wasn’t too concerned about because he had run the .mac Backup application all the time. He got a new drive and restored his data. Just to find that all his photos and iTMS purchases were there. But, the most essential data, his documents, were missing. Panic ensued and a large amount of money had to be spent for hard drive recovery to rescue his thesis.

Idiot! I hear you scream. Sure, he should have checked the recoverability of his data. But keep in mind that .mac customers are exactly the people more technical minded folks consider as ‘idiots’. If Apple tells them they have a backup and essential parts are missing at the end of the day, it doesn’t exactly build trust. It wasn’t possible to reconstruct how exactly that situation came to be. But even if he turned off backing up documents himself, the software should have told him about the implications. Just as it warns you before erasing a drive.

As a consequence of all this Apple will have to earn the trust needed for Time Machine as a viable backup software. And while a lack of actual experience with the technology as well as initial software problems still dominate the reports at this stage, I think they have a reasonable chance at succeeding in the long run.

Which I wish they do. Simply because it will help people’s data survive.

Background activity

There are two aspects to how Time Machine works: The user interface and the background activity needed to do the actual magic. Both of them are probably of equal importance as only a good user interface will entice people to create backups of their data and enable them to recover data which would be lost otherwise. But without the behind-the-scenes work nothing is going to happen. To keep up the good spirit, I will dig into the more obscure topic of background activity first before bitching about the user interface. This first part will be both lengthy and slightly technical. If you can’t take that skip it.

Ideas about what a backup is vary a bit. The most obvious - and very widespread - interpretation of the word is simply making a ‘clone’ of your hard drive which you update from time to time. Obviously that’s better than nothing but once you work with, or think about, backups, you soon realise that it’s really not enough. At least not if you want to be able to do more than just restore your system in the case of mass deletion or a hardware failure. Say, some data corruption happens to a file, but you don’t realise it at the time. You’ll run your clone ‘backup’ that evening and your only existing copy of the file’s previous version will be lost. Ouch! Suffer from this once and soon you’ll join the people who consider a backup to be an initial copy of all your data plus fresh copies of all files that have been changed since the previous backup run. Time Machine provides just that and I shall try to give a rough sketch of how that works (note for the overly optimistic: I have neither written Time Machine myself nor seen its code, so I can only go from observation of Mac OS X.5 and the documentation).

First, let’s look at the data storage: As things are, Time Machine requires a HFS Extended file system to work. This has been the default format for Mac hard drives for a long time, so it’s not a big deal. But if you don’t have a Mac OS Extended volume (but a drive shared by another Mac OS X.5 computer via Apple’s File Sharing or a drive in a different format), things will work as well as Time Machine creates a disk image in HFS Extended format and saves your data into that if necessary. In what follows I will just assume we are dealing with a HFS Extended volume.

At the top level of that volume Time Machine creates a folder named ‘Backups.backupdb’. And inside that folder there is a subfolder with the computer’s name - meaning you can back up different computers to the same Time Machine volume. Inside that folder there is a number of folders whose names are time stamps. These folders include the data of the drives Time Machine backed up at that time.

Hard links

Unlike many other backup solutions Time Machine doesn’t store backups in a single file backup database which may have an unknown format and may make it hard to extract data from it without the software that created it. Instead, Time Machine uses ordinary folders and files. That may be a bit less efficient space-wise but has the advantage of being simple as well as browsable on any machine that’s able to read HFS Extended volumes without Time Machine at hand.

Time Machine’s trick to avoid duplicates of unchanged files in its archives is to use hard links. Hard links are a very old and very basic concept. Yet they can be a bit mind boggling. When looking at how a file system works, you end up discovering that there are two things: The hierarchical folder structure and the files themselves. A folder essentially contains nothing but a bunch of references to the files themselves rather than the actual files: You can move even a huge file from one folder to another on the same volume in a split second because all that happens is that a reference to the file is inserted in the destination folder and it is removed from the source folder. The data themselves remain in the same place throughout.

In fact, that reference inserted to the folder is just a hard link. The new thing is just that once you have several such hard links to the same file, the file is listed several times in the file system’s folders but the actual data only exist exactly once. And that’s exactly what Time Machine uses.

Just to get a better feeling for this, let me answer a few questions I wished you had asked:

What is the difference between hard links and the ‘symbolic links’ Unixy people normally use?

The difference is that a symbolic link points to the path the file is located at. That is, it only points to the entry in the folder hierarchy rather than the ‘actual’ file. [Perhaps the image below or the next question clarifies this.]

Uh, what? You’re talking gibberish.

Just imagine the following situation: You have a file and a symbolic link to it. Then you delete the original file. As a result the symbolic link will point to nothing. And then you move another file to the same path that the original file was at. The symbolic link will then open that file. None of that could happen with a hard link. It always points to the same ‘actual’ file. Essentially it ‘is’ the file.

If ‘hard links’ are so great, why aren’t we using them all the time?

While they are great for many purposes, hard links also have a number of drawbacks. Most notably they can only point to things on the same volume.

How do Mac aliases figure in this game?

Mac aliases are more like symbolic links as they are separate files containing data about the location of the file they reference. However, aliases can contain information about the location of the ‘actual’ file and thus be much better at finding files even after they have been moved than symbolic links storing a path reference can hope to be.

One way in which Time Machine goes beyond what is usually done is that it makes extensive use of hard links for folders. These are usually avoided as they require very careful handling in situations when multiply hard linked folders are nested within another. But the feature is essential for Time Machine so it can just use a single hard link as a reference for a folder and everything it contains instead of having to create actual folders for everything and hard links for every single file in each snapshot.

Having multiple hard links to a folder turns out to be problematic once you start thinking about circular references and how deleting files should work: currently you can only remove a folder from the file system if it is empty; if you have multiple hard links to the folder it wouldn’t seem unreasonable to remove one of them without the folder being empty. But then imagine the only remaining link to that folder being from an item that is contained in the folder itself: You wouldn’t be able to reach that folder from the file system hierarchy anymore! In fact, the System’s ln tool doesn’t let you create hard links to folders yourself.

Perhaps a little demonstration will be a good way to finish this off. You can play around with the relevant commands in the Terminal without much effort, creating files, creating hard links with the ln command as well as soft links with the ln -s command. And keeping a close eye on the output of the file listing command ls -l. Its output contains a mysterious number after the initial permissions (e.g. drw-r--r--) bit. That number is exactly the number of ‘hard’ links to the file in question. Looking at these a bit you quickly realise that most files have a number of 1 and a folder has a number of at least 2 [the parent folder pointing to it and the folder pointing to itself - the ‘.’ item in the listing of the folder. Every additional item inside the folder will increase that number by 1].

[kalle:~] ssp% mkdir linktest
[kalle:~] ssp% cd linktest
[kalle:~/linktest] ssp% echo "test" > originalfile
[kalle:~/linktest] ssp% ls -la
total 8
drwxr-xr-x   3 ssp  ssp   102 17 Feb 20:25 .
drwxr-x---@ 41 ssp  ssp  1394 17 Feb 20:25 ..
-rw-r--r--   1 ssp  ssp     5 17 Feb 20:25 originalfile

Up to here we created a file originalfile containing the string test and looked at the folder contents. There is a link count of 1 for the file and of 3 for the folder. Next, create both a hard link and a symbolic link to that file:

[kalle:~/linktest] ssp% ln originalfile hardlink
[kalle:~/linktest] ssp% ln -s originalfile softlink
[kalle:~/linktest] ssp% ls -la
total 24
drwxr-xr-x   5 ssp  ssp   170 17 Feb 20:25 .
drwxr-x---@ 41 ssp  ssp  1394 17 Feb 20:25 ..
-rw-r--r--   2 ssp  ssp     5 17 Feb 20:25 hardlink
-rw-r--r--   2 ssp  ssp     5 17 Feb 20:25 originalfile
lrwxr-xr-x   1 ssp  ssp    12 17 Feb 20:25 softlink → originalfile

And observe how the numbers change: The file with the hard link to it has an increased link count of 2. Finally, we delete the original file and convince ourselves that it survived thanks to the other hard link to it:

[kalle:~/linktest] ssp% rm originalfile
[kalle:~/linktest] ssp% ls -la
total 16
drwxr-xr-x   4 ssp  ssp   136 17 Feb 20:36 .
drwxr-x---@ 41 ssp  ssp  1394 17 Feb 20:25 ..
-rw-r--r--   1 ssp  ssp     5 17 Feb 20:25 hardlink
lrwxr-xr-x   1 ssp  ssp    12 17 Feb 20:25 softlink -> originalfile
[kalle:~/linktest] ssp% cat hardlink
test

Once such hard links come into play, you lose the sense of which one is the ‘original’ and which one is ‘just being pointed to’ - as both of them are the original. And thus, when browsing your Time Machine folders on the backup drive, all files will look like proper files and not carry the little superimposed arrow icon which Mac OS X normally uses to indicate aliases or symbolic links. But with the Terminal commands above you can hope to get an idea of what is going on.

When peeking around your backup you can learn a few things about it if the numbers are interpreted correctly. For example for files at the top level of your startup volume the number tells you in how many backups the file has been included. E.g.

[kalle:~] ssp% ls -l  /Volumes/Snowman/Backups.backupdb/Kalle/Latest/Kalle/ | grep mach
-r--r--r--@ 43 root  admin    616052  6 Nov 06:40 mach.sym
-rw-r--r--@ 30 root  wheel  10272820  6 Feb 01:15 mach_kernel

Suggests that the ‘mach_kernel’ file has been replaced 30 backup runs ago. The same heuristic doesn’t work as easily for folders as you’ll have to subtract the number of items in them before knowing the number of hard links around. Also be careful when examining folders lower down in the file system. You will frequently see link counts of 1 there. That happens when no item in that folder or in any folder contained by it has been changed. Then Time Machine only created a hard link for the enclosing folder and thus every file inside it has only been backed up and hard linked to once.

Extended Attributes

But there’s more to see for those who like poking around. Apple introduced extended attributes in Mac OS X.4 and Time Machine uses them a lot, both inside the backup copy and on your usual files. Mac OS X.5 comes with the xattr command line tool for reading and writing them.

Poking around the ‘Backups.backupbd’ folder you will find these attributes:

Machine Name - a top level folder with the name of the machine backed up, e.g. ‘Kalle’

• com.apple.backupd.BackupMachineAddress — containing a string of the Ethernet port’s MAC address on my MacBook and that of the Airport card on a MacBook Air .

Timestamp - a folder with the date of the backup it contains as its name, e.g. ‘2008-02-15-000458’

• com.apple.backup.SnapshotNumber - a string representing the index of the backup, increasing by one in each iteration.
• com.apple.backup.SnapshotVersion - I have only seen the string ‘1’ here.
• com.apple.backupd.SnapshotCompletionDate - sounds clear, though I haven’t tried figuring out the number format yet.
• com.apple.backupd.SnapshotStartDate - dito.
• com.apple.backupd.SnapshotState - gives the string ‘4’ for all snapshots I have.
• com.apple.backupd.SnapshotType - gives the string ‘3’ in the time range when snapshots are daily, ‘2’ in the time range when they are hourly (with a little glitch where daily and hourly meet) and ‘1’ for the latest snapshot. I’m guessing a bit here, to be honest.

Volume-Name

• com.apple.backupd.SnapshotVolumeFSEventStoreUUID - as the name suggests the UUID of the current fsevents store (see below); this matches the string in the file /Volumes/Volume-Name/.fseventsd/fseventsd-uuid.
• com.apple.backupd.SnapshotVolumeLastFSEventID - the ID of the last event reflected in the snapshot, not sure how to find that one in the files.
• com.apple.backupd.SnapshotVolumeUUID - the UUID of the volume in question, matching the one you get with diskutil info /Volumes/Volume-Name.
• com.apple.backupd.VolumeIsCaseSensitive - naturally 0 on sane volumes.
• com.apple.metadata:_kTimeMachineNewestSnapshot - a binary property list containing a Date as its root object. The date is that of the newest Snapshot containing the file or 1.1.4001 if the file is still present in the newest snapshot.
• com.apple.metadata:_kTimeMachineOldestSnapshot - a binary property list containing the date of the oldest Snapshot which contains (or once contained) the file.

Further subfolders

• com.apple.metadata:_kTimeMachineNewestSnapshot - as above.
• com.apple.metadata:_kTimeMachineOldestSnapshot - as above.

With all these extended attributes in place, Time Machine’s archive is more than just a copy of the files. It also contains guides to match the backup up with local data and to see where the previous and next versions of a file are stored.

Time Machine also uses extended attributes on your working volume to mark files and folders as excluded from the backup.

Exclusion

In the way Time Machine currently works, exclusion is a vital concept for it. The software defaults to backing up a complete volume and you can control things a little by excluding folders from the backup. Say, to ensure that huge but frequently changing files - like disk images from a virtual machine or a large database - don’t clutter up the backup too much. There are various ways files can be excluded from the backup. (Don’t read too much into the order of that list as I haven’t checked which of these items takes precendence in case of conflicts. That might be an interesting thing to do.)

StdExclusions.plist in CoreService’s ‘backupd.bundle’

This file lists many paths which are excluded from backups, both globally and in user folders. These are mostly folders for temporary files or files which can otherwise be recreated like the Spotlight index. As of Mac OS X.5.2 this file excludes the content of the folders /Volumes, /Network, /automount, /.vol, /tmp, /cores, /private/tmp, /private/Network, /private/tftpboot, /private/var/automount, /private/var/log, /private/var/folders, /private/var/log/apache2, /private/var/log/cups, /private/var/log/fax, /private/var/log/ppp, /private/var/log/sa, /private/var/log/samba, /private/var/log/uucp, /private/var/run, /private/var/spool, /private/var/tmp, /private/var/vm, /private/var/db/dhcpclient, /private/var/db/fseventsd, /Library/Caches, /Library/Logs, /System/Library/Caches, /System/Library/Extensions/Caches, the files and folders /.Spotlight-V100, /.Trashes, /.fseventsd, /.hotfiles.btree, /Backups.backupdb, /Desktop DB, /Desktop DF, /Network/Servers, /Previous Systems, /Users/Shared/SC Info, /Users/Guest, /dev, /home, /net, /private/var/db/Spotlight, /private/var/db/Spotlight-V100 and for each user account the files and folders Library/Application Support/MobileSync, Library/Application Support/SyncServices, Library/Caches, Library/Logs, Library/Mail/Envelope Index, Library/Mail/AvailableFeeds, Library/Mirrors, Library/PubSub/Database, Library/PubSub/Downloads, Library/PubSub/Feeds, Library/Safari/Icons.db, Library/Safari/HistoryIndex.sk. I knew you’d find that interesting.

com.apple.TimeMachine.plist in global preferences

This property list contains two items of interest: ExcludeByPath which seems to accumulate items like ‘Library/Calendars/Calendar Cache’ or ‘Library/Mail/Envelope Index’ for users that used them (why isn’t the first one included in the previous list? Why is the second one in both?) and SkipPaths which contains a list of all items the user dragged to the exclude list in Time Machine system preference pane.

extended attributes

Applications can also mark files as not-to-be backed up. Apple’s Backup Core provides an API for that. The effect that seems to have is adding an extended attribute “:EA:com_apple_backup_excludeItem” to the file with the content “com.apple.backupd”. A Spotlight query for com_apple_backup_excludeItem = "com.apple.backupd" should reveal all files excluded from backups in this way. Currently it seems to be mainly used for the iTunes Music Library.xml file.

While there certainly is a point to many of the exclusions made by Time Machine, I do find them slightly problematic. Particularly as many people will not know they exist and it is non-obvious how to find out which files will not be backed up. Even more so, as applications can easily remove your documents from future backups by accident (or, maliciously, on purpose). It’s a shame that there isn’t an easily accessible list of files which aren’t backed up. I guess the sufficiently paranoid may want to keep an eye on the property lists and attributes mentioned above.

A point that is rather sore as well for me is the fact that Caches folders are completely excluded from backups. By Spotlight’s poor design the Caches folder contains a Metadata folder which applications have to store their data in if they want it to be found in the index. Those metadata files will not be in a backup. As a consequence restoring your system from a backup will leave you with an incomplete Spotlight index until you run all the applications which stored data in the Caches/Metadata folder and make sure they re-create them. Ultimately this is a Spotlight issue, I think, but with Time Machine being made by the same company, they should have had an eye on it.

Spotlight

Time Machine also ties in with Spotlight. Items in the backup are indexed by the Spotlight engine to help you find stuff when needing it. The index for the Time Machine backup is stored separately from the remaining index of the volume, indicating that Spotlight has become a bit more flexible in this regard. I suppose separating things in this way may also improve performance for everyday searches in the non-Time Machine content of the volume.

An interesting aspect is that extended attributes beginning with com.apple.metadata can also be found in the Spotlight index. In particular, this gives you a way to look for all files added to the backup in a certain time range. So if you sometimes wonder ‘what the heck caused my last backup run to take so long and to contain that much data, Spotlight might be helpful.

If Spotlight’s graphical user interface weren’t a complete POS, this could even be fun. But currently it’s a bit painful to do. And given Apple’s past speed of improving the Spotlight UI, I won’t expect considerable improvements there before 2050. But still, we can attempt to do it. Just be sure to take this with a lump of salt. I have seen loads of strange behaviour when testing this. ‘Strange’ behaviour included the Spotlight server process mds crashing and happily starting a complete re-index after being automatically restarted by the System. It also included recently backed-up files not being in the Spotlight index and thus not appearing in the results for these searches. All that seemed a bit random, so while in principle possible, I wouldn’t rate these steps as reliable but more as an interesting exploration at this stage.

1. In the Finder, navigate to the ‘Backups.backupdb’ folder on your Time Machine backup volume.
2. Use the Find command
3. Switch the scope of the search to the ‘Backups.backupdb folder. This switch will ensure that the Spotlight index for the backup is used instead of the general one.
4. Add a ‘System Files’ search criterion and set it to include System files. This ensures that Mail messages and files from the Unix innards which the Finder prefers to ignore are displayed as well.
5. Add a ‘Raw Query’ search criterion (still localised as Reine Daten in German - whatever that is supposed to tell me).
6. Enter a raw search query string like _kTimeMachineOldestSnapshot > $time.now(-3600) into the field in the Finder window. [This didn’t work when I initially tested it and the text used to contain a more complicated way to create a query. Thanks to tinbert for pointing out that this does work after all and encouraging me to give it another try.]
5. Determine the number of seconds between an hour ago and the beginning of 2001. Unless you have something like an NSDate handy this will suck. Luckily thanks to Cocoa’s new ‘scripting bridges’ in Mac OS X.5, you tend to have an NSDate handy. So I invested two lines of Python to get this number [note that (a) I don’t know any python, (b) probably some smartass can do the same with less resources in bash or ed or whatever and (c) Google Calculator apparently can’t evaluate ‘seconds since 1.1.2001’]

   #!/usr/bin/env python
   from Foundation import NSDate
   print NSDate.timeIntervalSinceReferenceDate() - 3600
   

   [Download a version for the current time or one for an hour ago if you’re too lazy to copy and paste.]
6. Enter a raw search query string like _kTimeMachineOldestSnapshot > 225450653.661 into the field in the Finder window. Where the lengthy number has to be replaced by the number you determined in the previous step. This is yet another good opportunity to curse the people who thoughtfully ‘designed’ the Finder to not extend the width of that query text field to grow with the window’s width.

The Finder lets you save elaborate search queries like this one as a ‘smart folder’ for easy accessibility later on.

In case you want to edit it at a later stage you need a Finder window with a toolbar and an action/cogwheel menu in there. That menu contains a command for editing the search query. Thanks to Dirk and S for pointing this out.

If you are happy using the Terminal, running a command like

mdfind -onlyin /Volumes/Snowman/Backups.backupdb/Kalle/ '_kTimeMachineOldestSnapshot >  $time.now(-3600)'

may be more convenient in practice.

fseventsd

One thing that hasn’t been looked at so far is how the System knows which files need backing up. The traditional way of just scanning all the folders and subfolders on the attached drives would be quite an undertaking with hundreds of thousands of files hiding on a normal Mac OS X startup drive these days. Particularly if you want to update your backups hourly.

But luckily Apple gave the System the ability to just know when a file has been changed in Mac OS X.4 where Spotlight needed it to know which files require re-indexing after being modified. This approach has now been extended in a way that the fseventsd background process monitors all file changes on the system and. It notes the folders containing files which were changed on a volume in a file inside the .fsevents folder at the top level of that volume. A more detailed description of how that works can be found in the Ars Technica X.5 review.

For this to work reliably it is crucial that the fseventsd logs are complete as otherwise files could accidentally be omitted from a backup. And thus if you ‘improperly’ remove a drive, say, because of the system crashing or being too daft to properly unmount it, fseventsd creates a fresh catalogue of changes the next time you use that drive. In the logs that will look like this:

23.02.08 14:47:38 fseventsd[45] event logs in /Volumes/Bender/.fseventsd
    out of sync with volume.  destroying old logs. (2394 0 2433) 
23.02.08 14:47:38 fseventsd[45] log dir: /Volumes/Bender/.fseventsd
    getting new uuid: 6AF6C56A-3E96-4D99-89F6-BB60256E0C97 

backupd

Finally, there is backupd, the workhorse of Time Machine. It uses the information collected by fseventsd to zone in on the files which have been changed since the previous backup. If that isn’t possible because there has been a problem with fseventsd, that will be recognised [see that the fsevents store gets a new UUID in the log message above and that the extended metadata for the previous backup include the UUID of the fsevents store used] and kicks off a ‘deep traversal’ of the file system. In that case it will go through all files to ensure it is aware of all changes. In the logs that’ll look like this:

23.02.08 14:48:08 /System/Library/CoreServices/backupd[26227] 
    Event store UUIDs don't match for volume: Bender 
23.02.08 14:48:10 /System/Library/CoreServices/backupd[26227] 
    Node requires deep traversal:/Volumes/Bender 
    reason:kFSEDBEventFlagMustScanSubDirs| 
    kFSEDBEventFlagReasonEventDBUntrustable| 

Once the list of files needing backup has been collected, the process will copy them over to the Time Machine volume, creating all the metadata described above. And of course making sure there is enough space to accomplish the task before starting it. It also purges old backups when more space is needed on the backup volume as well as those which aren’t needed anymore (that is, hourly backups which are older than a day as well as daily backups older than a month).

To do its job particularly well, backupd will check for any files that may have changed during a lengthy backup run and add those files to the backup at the very end.

Weakness

All this sounds fine and dandy. But you really have to keep in mind how it works: fseventsd only notes which folders have been changed. Then backupd scans them for files which have changed. And backupd doesn’t make a big effort when doing that job. Two obvious tests for its cleverness are failed with ease.

The first test is renaming. Renaming a file only changes its name in the folder but not its contents. Thus - keeping in mind all the hubub about hard links above - I’d expect Time Machine to create a hard link with the new name to the same old file in the backup when running the next time after renaming the file. So let’s try just that. Create the file

[kalle:~] ssp% mkdir /Volumes/Bender/renametest
[kalle:~] ssp% touch /Volumes/Bender/renametest/file

Run the backup and have a quick look at it:

[kalle:~] ssp% ls -l /Volumes/Snowman/Backups.backupdb/Kalle/Latest/Bender/renametest/
total 0
-rw-r--r--@ 1 ssp  admin  0 23 Feb 15:07 file

The file exists, now rename it

[kalle:~] ssp% mv /Volumes/Bender/renametest/file /Volumes/Bender/renametest/file-2

Run a backup again and check once more:

[kalle:~] ssp% ls -l /Volumes/Snowman/Backups.backupdb/Kalle/Latest/Bender/renametest/
total 0
-rw-r--r--@ 1 ssp  admin  0 23 Feb 15:07 file-2

The file with the new name exists in the backup. But its link count is just 1. Thus it is a new file. The harm done by this varies. For a tiny file like this one it’s hardly worth mentioning. For a big video project where renaming may mean a wastage of many gigabytes of storage space just because it has been renamed, that may be more painful.

What is potentially even more harmful is the following behaviour: Time Machine considers two files to be identical if their path and dates and size are the same. I.e. it relies exclusively on metadata to determine whether or not a file changed, which ironically means it checks everything but the only relevant information. Luckily this isn’t too much of a problem in most practical situations, but it still strikes me as ironic. Having something like a hash of the file’s content could improve this a lot.

And although the last changed date of a file is supposed to change when that file is edited, that isn’t necessarily the case. (In fact, with the shambles that are metadata on computers the ability of GraphicConverter to not modify the last changed date of a file when saving it is convenient as it will not cause converted versions of an image to have a date that differs from the original file’s.)

And thus I can create a file (I used a 1 by 1 gif image for a simple test), back it up, edit it (with that file not changing its size and dates in the process), run the backup again and have only the first version of my file in the backup. If my hard drive breaks now, the changes to that file will be lost.

This may seem like a tiny point and a situation that shouldn’t happen. I think it’s good to have a clear sense of what the words we use are. And in the way Time Machine uses the word ‘changed’ it means that a file that has been moved to a different location has been ‘changed’ and that a file whose content has been replaced by other content of the same size in a way which didn’t modify its other metadata is not ‘changed’. As my usual understanding of the word ‘changed’ differs from that, it’s worth noticing and keeping in mind to avoid disappointment.

User Interface

Not last but least there is the graphical user interface for Time Machine. Like many recent GUI ‘innovations’ by Apple it leaves me with a ‘WTF What were they thinking?!’ expression:

While not the biggest Bauhaus fan, I am a ‘form follows function’ type. And that concept certainly didn’t play a role when ‘designing’ the GUI for Time Machine. There are so many things wrong with it that I don’t know where to start or how to make sense of it. So I’ll just give a semi-random list of points to think about.

• Shouldn’t tasteless Sci-Fi backgrounds be an exclusive feature of Linux distributions?
• If you need to have moving stars in the background image of your backup application, can’t I expect them to move smoothly no matter what I do? Particularly if I’m using a ‘supercomputer’? With an OS that has been ‘crafted’ to perfectly match the hardware? [This can be worked around using the _FXShowBackgroundAnimation hidden preference of the Finder.]
• Wouldn’t it be helpful to have a real timeline on the side? One with a meaningful scale?
• Wouldn’t it be helpful to see which files changed in a particular snapshot? (Rumours are Apple thought about that, but obviously they didn’t ship it. The Spotlight comments above point at how doing that could be reasonably easy.)
• Wouldn’t it be helpful to be able to see the development of a folder or file rather than just having a 3D arrangement of older versions of the window behind the front one, all of which are covered up so much that they are completely useless?
• If you start Time Machine from a lean Finder window without the sidebar, is there any chance that the user doing it may want to use the Find command? (The only workaround I currently see for that is quitting Time Machine turning on the sidebar in the Finder window and turning Time Machine on again - no kidding!)
• I am not a fan of contextual menus, but shouldn’t those work in the Time Machine environment as well when it pretends to be just like the Finder?
• Is a feature like the remove from backup command really ‘discoverable’ when hidden away in a cog wheel menu that may or may not be visible on screen.
• If you try to give users the illusion that they really are working with exactly the same window they fiddled with in the Finder while using Time Machine, wouldn’t it be a good idea to actually do this in a way that scroll bar positions remain constant as well?
• … and in a way that the Show Package Contents command works?
• If you need a full screen UI for restoring and you even hide the Dock when showing it, wouldn’t it be great if the ‘shadows’ of F11-Exposé went away as well?

… and so on. When playing with just the Finder’s Time Machine interface you quickly get the idea that some Sci-Fi geek had an UI idea which they then implemented but never really tried out. If you are using a large sidebared Finder windows in column mode, it will work all right for recovering a single file. In any other situation things quickly appear a bit broken and not exactly well planned and thought-through.

Other aspects of the UI such as the navigation arrows could also be better in that what happens when you click them would be predictable. Right now clicking the backwards arrow will slide things around but I have no idea how far they will slide before seeing it. Even if it slides just ‘the right’ distance, why not indicate that in the UI beforehand by omitting the superfluous layers or greying them out? It’s not really clear to me anyway what all those Finder windows actually represent on screen. Usually they do not represent the state of the file system at the time the window in question represents. They just seem to be a copy of the frontmost window and will only be updated when they come to the front themselves.

Time Machine also comes with two UI elements: An icon which appears in the Dock by default in X.5 (Apple’s tendency to clutter your Dock with stuff starts reminding me of the Windows ‘Start Menu’ and Desktop which are littered by every installer - Apple just have a better monopoly to get more exclusive spamming opportunities for themselves) and a menu bar item which was added in the X.5.2 update. It lets you run a backup and launch the Time Machine UI. And it rotates like hell, the wrong way round, while files are being backed up. (The icons the Finder displays in its sidebar next to the icon of the Time Machine volume rotate the right way round, btw.)

Restoring Files

The most important thing about a backup is that you can get your files back if you need them. As a Time Machine backup is nothing but a folder with files in it, you always have the ability to just browse into the Backups.backupdb folder in the Finder and get the file you want.

Or you can use the ‘official’ starry sky Time Machine GUI described above and grab the files from there if you have the nerves for that. You will even be asked whether you want to use the backup file to replace the newer version of itself or whether you’d like to have both files next to each other. With the files I recovered so far, both methods worked fine.

Both these methods use the Finder for copying the files. And the Finder will strip the extended attributes relevant to the backup from them when doing that. Which is probably a good idea. Copying the files using the cp command leaves the extended attributes in place and you’ll probably want to remove them afterwards.

The Mac OS X.5 installer also offers an option to restore the system from a Time Machine backup rather than setting up or migrating accounts. That’s a very appealing option if the whole hard drive breaks. I haven’t tried it myself yet but reports I read so far seem to be mixed. While no harm is done to the backup data, doing the full restore doesn’t seem to be terribly reliable yet. It may be a good idea to keep those cloned drives handy for a while if you need to be able to recover quickly.

Other applications

Another feature of Time Machine is that applications are encouraged to integrate with it. Some Apple applications like the Address Book, Mail or iPhoto offer such integration (and applications like iTunes, iCal or Safari lack it). But I fear that many people won’t even notice that as you have to invoke Time Machine while the application is active to get the Time Machine interface for that application. And there isn’t an out-of-the-box way to invoke Time Machine for most of those applications if you haven’t cluttered your Dock or menu bar with its icons. Only iPhoto seems to have an explicit menu item for accessing backups at this stage.

I haven’t really needed this so far, so I can’t say terribly much about it. The idea behind it seems to be a laudable one. As many applications store their data in an opaque format which consists of many little files and perhaps some database keeping them together, it’s not feasible to do fine-grained restores for these applications. You can either revert the full database to a previous state - undoing many other, wanted, changes in the same move or you can’t undo anything.

Letting the application access its old database and restore the little bits and pieces from it is a sweet idea. But I wonder how many applications will offer that. I am sure that doing so may be quite a bit of effort. And doing that in a correct way for the rare situations when a user both needs it and knows the feature is there, seems to be a lot to ask.

As things are, the integration with Apple’s non-Finder applications isn’t first class anyway. Not only is half of the date at the bottom of the screen covered frequently covered by a button on my ‘small’ MacBook screen (a problem so obvious that it suggests nobody at Apple ever bothered to think about this properly or to try it out on such a machine), the general behaviour also seems to be quite erratic (iPhoto windows showing up empty at first and only filling eventually after I scrolled back and forth a bit, iTunes playback skipping while the computer made that ‘effort’, the lack of displaying actual differences in the Address Book, to give a few impressions I collected when trying things out for a few minutes).

The integration with applications also makes me slightly uneasy. The beauty of standard Time Machine backups is that they are a-file-for-a-file. It’s easy to understand what goes on there. And recovering a file from the backup is essentially a copy operation. The more ‘clever’ behaviour applications show here means that the transparency of what is going on is lost. I will have to hope that the software does the right thing.

Preferences

Time Machine also comes with its own preference pane in System Preferences. It features a huge logo with the name beneath it in Myriad which doesn’t really belong to OS X. It also features a huge ‘slider’ On/Off switch that is in no way a standard UI element and in no way behaves like the real world slider it’s supposed to represent. It works like a checkbox, so why isn’t it a checkbox?

With space being so generously used for the ‘pretties’, little remains for actual preferences. You mainly get a display of the current backup status, some information about how backups work and two odd looking buttons which in open sheets containing the actual settings. Just get that: There are a mere five settings they let you make for Time Machine and you need a whole preference pane and two sheets for them.

In fact those sheets are quite anaemic. They are tiny and all they let you do is select the backup volume and exclude stuff from backups. The excluded items from various volumes are then displayed alphabetically in a single list with no sense of their location. So if you excluded a folder named ‘Pictures’, good luck finding out which one it actually is. You’ll have to hover over the line in the list to get a ‘tooltip’ with a Unix style path to the folder in question. Very user friendly and Mac like!

And that’s before I get into real bitching mode, my friends! Let’s say you have a hard drive and a single folder on that hard drive which you want to include in your backup. If you naïvely assumed that this should be a ‘super easy’ single drag-and-drop operation in Apple’s OS, you’d be wrong. As far as I can tell you are expected to add all other folders on the disk in question to the exclude list and of course update its state whenever you add another folder. Excellent!

Security

Security seems a topic which they mostly ignored when designing Time Machine. If you consider your data to be private, you can be screwed in two ways by Time Machine.

If you really are more concerned about your data remaining private than about it being unaccessible, you are probably using FileVault on your machine. It stores your home folder in an encrypted disk image (these started off with a reputation for easily being irreparably broken in system crashes or so - I heard the format improved since, but never really checked that). And you get plenty of inconvenience for that - such as your home folder not being available via file sharing and possibly a bit of a speed hit. Oh - and you will only get very limited Time Machine goodies as well. Because, rather than saving the files in your account, the computer will only make a backup copy of your complete encrypted disk image if you are using File Vault. To make that a bit more efficient, Apple introduced a new disk image format, the ‘bundle image’ which splits its contents up in several ‘slices’. Meaning that Time Machine only has to back up the slices which changed rather than the complete image. This isn’t exactly neat but - assuming that they got the encryption right - it will keep your data encrypted even in the backup. But your only option for restoring will be to restore the whole image in one go, rather than the individual files contained in it.

The other aspect is encryption of backups. Usually backup software offers it as an option. And it makes a lot of sense to me. Even if you say that someone has to break into your home to steal your data and that you don’t need local encryption because of that, it seems entirely possible that you would (or rather, should) store your backup somewhere else. Put differently: There will be another copy of all your data now which you rarely need to access. Thus the inconvenience of the encryption plays less of a role and it would be nice to have.

Unfortunately Time Machine doesn’t offer to encrypt backups. Which is a shame - it will use disk images anyway and encrypted disk images do exist, so it seems like this would be an easy thing to do. Of course that extra level of encryption would require a bit of careful design about where the password for the backup image is stored on a shared computer. And it could lead to agony if someone ran their backup for years and ended up needing it without remembering the password. It’d still be a nice option.

Once you start thinking about network backups - possibly storing your data on servers of some third party on the internet - which hopefully becomes an option in the future, encryption of backups would become pretty much mandatory. Call me paranoid, but at least I wouldn’t want all my data to be stored in clear text on their machines.

What Else

Even though this text is both wordy (for sure) and rich in information (hopefully), it still doesn’t present a full coverage of Time Machine. Mostly because I am running a single machine in a single environment which means that I am not able to know about the full range of usage cases, the opportunities, the conceptual problems, the practical problems. All I can go by is word of mouth. Where word of mouth isn’t even that but just what I picked up from the garbage dump known as the internet.

The impression I get from this - my limited experience spanning a few months and the reports I read - is the following: Time Machine is worth having because it does make an additional copy of your data and is unlikely to accidentally destroy that additional copy.

On the other hand there are still plenty of practical issues at this stage. Ranging from the - not necessarily Time Machine intrinsic - fact that OS X.5’s support for external drives could be much better performance-wise and even seems kernel panic prone (from my own experience). Then there’s the sub-par UI, both for setting up the backup (no way tell Time Machine ‘include this specific folder in future backups’) and for recovering files (which is both tacky and giving much less information than would be helpful and possible). I’d say that technically those UI points aren’t biggies. And perhaps we can hope for someone givng us a better UI for it. With Apple’s personality being a bit too far on the self-consumed and narcissistic side to admit they shipped something less than ideal or - gasp! - poorly designed, I wouldn’t hold my breath while waiting for them to improve things.

And then there are future questions, most of which were touched somewhere above: Quotas trying to fairly split backup space among several users? Dealing with small changes to large files (frankly, I consider Apple’s idea of simply splitting them up - as demonstrated for bundle disk images - to be a cheap hack rather than a good solution)? The apparent problems with applications like Aperture (no first hand experience)? Getting the ability to easily revert system updates you didn’t like with Time Machine? The numerous glitches which have been reported by people trying to do full restores (i.e. your data are saved but recovering them may not be totally fool proof). The whole issue of encryption. And in a next step file systems (there will always been someone mentioning ZFS which isn’t a reality yet but sounds like it might be a good match for Time Machine although it may require a number of changes to the concept). And in yet another step, networked file systems for backups to another location (with encryption being a must-have feature there). Which leads to more questions about which file server volumes Apple support (currently OS X.5 computers and Time Capsule devices by default and give or take a hidden preference for everybody who likes doing unsupported things with their backups). This is far less than they initially promised and it’s usually said technical rather than marketing issues cause the scrapping of that feature. Of course with Apple always achieving all their goals we will never learn what the real reason was. And while them limiting the features of their software just to shift more Time Capsule hardware would be a shame and really do injustice to a topic as important as backups, it wouldn’t seem completely along the lines of Apple’s ‘personality’ to do exactly this.

It appears that Apple’s mid-March 2008 Airport/Time Machine update at least enables volumes connected via their own Airport Extreme base stations to be used for backups.

Fin

So here we are - I hope you enjoyed the trip. Time Machine looks like a great step forward for preserving data. But we will have to keep an eye on Apple whether they are serious about it or not.

Want some Time Machine goodness?

March 5, 2008, 1:09

Tagged as backup, time machine, X.5.