macpro:~ sam$ sqlite3 ~/Library/Mail/Envelope\ Index
SQLite version 3.6.3
Enter ".help" for instructions
Enter SQL statements terminated with a ";"
sqlite>


Everything about your mailboxes is stored within this database and the structure of the database is normalized so its very easy to navigate. The tables of most interest for mail analysis are:

sqlite> .tables
addresses              mailboxes              todo_notes
alarms                 messages               todos
associations           properties             todos_deleted_log
attachments            recipients             todos_server_snapshot
calendars              subjects
feeds                  threads

Fortunately, accessing a SQLite database is quite easy from just about any language that you decide to use. I’m just going to do all the queries in straight sqlite3 rather than a language, but they could be embedded in your application. First things first, copy your Envelope Index to another directory:

macpro:tmp sam$ cp ~/Library/Mail/Envelope\ Index .

Now you can use that database without worrying about messing up the locking or corrupting data while Mail.app is using it. Since we might as well do an example that is interesting rather than merely educational, how about we answer the question: “Who are my coworkers with whom that I collaborated?”. This is going to be a multi-query process to extract the information — there may be more efficient ways to do it — but think of this as instructive rather than prescriptive. First I need to limit the query to only those mailboxes which contain work email:

DROP TABLE coworkermailboxes;
CREATE TABLE coworkermailboxes(id);
CREATE INDEX coworkermailboxes_index ON coworkermailboxes(id);
INSERT INTO coworkermailboxes SELECT rowid FROM mailboxes
WHERE
url like 'imap://samp@snv-webmail.corp.yahoo.com/%' OR
url like 'imap://sam@mail.sampullara.com/Yahoo%20Inc%20Archive';

That gives us a table with several mailboxes that I have in Mail.app including Sent Messages. I would peruse the list of mailboxes to ensure that you are grabbing all the correct information. For me I had to also search my archives. Now I am going to take a series of steps to get to the final out put by iteratively processing successive tables of information. The first table, is a list of those people that you have both sent and received an email with directly (they were the sender and you were a receiver or you were the sender and they were the receiver):

DROP TABLE coworkers;
CREATE TABLE coworkers(id);
CREATE INDEX coworkers_index ON coworkers(id);
INSERT INTO coworkers SELECT a.rowid FROM addresses a, messages m, recipients r
WHERE
m.sender = a.rowid AND
m.mailbox IN (SELECT id FROM coworkermailboxes) AND
r.message_id = m.rowid AND r.address_id = 4
INTERSECT
SELECT a.rowid FROM addresses a, messages m, recipients r
WHERE
m.sender = 4 AND
m.mailbox IN (SELECT id FROM coworkermailboxes) AND
r.message_id = m.rowid AND r.address_id = a.rowid
;

Note I have directly inserted my addresses rowid into this query for the sender on the one hand and the receiver on the other. The next step will be to count the actual number of emails you have received from each of those on the list:

DROP TABLE coworkers2;
CREATE TABLE coworkers2(id, recv);
CREATE INDEX coworkers2_index ON coworkers2(id);
SELECT "Get the received mail";
INSERT INTO coworkers2 SELECT w.id, COUNT(*) FROM messages m, recipients r, coworkers w
WHERE m.sender = 4 AND
m.mailbox IN (SELECT id FROM coworkermailboxes) AND
r.message_id = m.rowid AND r.address_id = w.id
GROUP BY w.id ORDER BY COUNT(*)
;

Finally, we count the number of sent emails and also derive a ratio of sent/received so we can judge how collaborative the exchanges have been:

DROP TABLE coworkers3;
CREATE TABLE coworkers3(id, sent float, recv float, ratio float);
CREATE INDEX coworkers3_index ON coworkers3(id);
SELECT "Get the sent mail";
INSERT INTO coworkers3 SELECT w.id, COUNT(*), w.recv, COUNT(*)*1.0/w.recv FROM messages m, recipients r, coworkers2 w
WHERE
m.sender = w.id AND
m.mailbox IN (SELECT id FROM coworkermailboxes) AND
r.message_id = m.rowid AND r.address_id = 4
GROUP BY w.id ORDER BY COUNT(*)
;

You will now have a table named coworkers3 that can be mined for information about your level of correspondence with them. For example, here is way to find relatively equal sends and receives:

SELECT a.comment FROM addresses a, coworkers3 w
WHERE
a.rowid = w.id AND
ratio >= .5 AND
ratio <=2 AND
sent > 10
ORDER BY sent
LIMIT 20;

When I do this I see the people that either I use to find information or that use me to find information. Every interaction is usually a request and then a response. On the other hand, this query will find those that typically made announcements out to the groups that I also worked with:

SELECT a.comment FROM addresses a, coworkers3 w
WHERE
a.rowid = w.id AND
ratio <= 1 AND
sent > 10
ORDER BY ratio
LIMIT 20;

And so on. Adding more filters on top of this you could easily derive your team at work for a particular time period and other insights. With the wealth of information contained in this meta-data store you could figure out all kinds of things:

• Who sent you an email that you didn’t reply to yet?
• Who do you respond to the most quickly?
• Who responds to you most quickly?
• What are you and your coworkers approximate working hours?
• What groups of CCs could be made into aliases?

There really is no limit to how far the analysis could go. Ideally, it would be possible to setup a dashboard in Mail.app that let you cut and slice the data in a far more precise way than smart folders currently allow today. Maybe they should come out with super-sql-smart folders!

I can’t stand inefficiency. Time Machine is fundamentally a very inefficient mechanism for backing up large files that change. So bad actually that most things like Parallels and VMWare disable backups of your disk images. Here is the basic algorithm:

1) Get the list of files that have changed since the last backup
2) Create new directory in backup store
3) Copy any file that has changed since the last backup
4) Create hard links to any file or even whole directory in the new backup to the last backup for any file that has not changed

Step 1 is pretty efficient for Time Machine as they keep hooks into the filesystem to track those changes as they occur. Step 2 is obviously easy. Step 3 is a doosy. If you change 1 byte in a VMWare image it will copy the several gigs over to the backup store. Not a great result from such small change and that would quickly consume your disk flushing valuable older changes out of the system. Step 4 is also very efficient because hard links are trivial to create and use virtually no space, though they did have to make special changes to HFS+ so that you could hard link directories to make Time Machine more efficient.

The obvious big problem here is that in the case that a file changes at all you need to copy the whole thing to you backup device. Not that viable over the internet or even WiFi for really big files that are updated often like VM images. You might have wondered why Apple is considering integrating ZFS directly into Mac OS X, now you know why. ZFS lets you do something very special: create a snapshot of a whole filesystem. Essentially a copy of that filesystem at a particular point in time and they do this without copying whole files when they change but instead at the block level. This amazing capability is critical in this more efficient way to backup your system with multi-level snapshots.

Enter rsync. Rsync has been around for a long time. It is used by system adminstrators everywhere to efficiently update files in one location with files from another location, even over the internet. It does this by comparing them at the block level and only sending diffs when needed to update files on the other end. Using the right command line options you can essentially make one filesystem look like a carbon copy of another filesystem. Using this in combination you can make a backup solution that is much better than most out there:

1) Rsync your current filesystem to a ZFS filesystem — remote or attached storage
2) Take a snapshot of the resulting filesystem to forever capture its state

Those are the two steps. Nothing more. Here is the script that I use to backup my Macbook Air to my server at home:

#!/bin/sh
cd /Users
time rsync -av --delete sam 192.168.1.90:/Volumes/zdisk/macbookair
ssh 192.168.1.90 sudo zfs snapshot zdisk/macbookair@`date "+%s"`

This results in a set of filesytems that looks like this:

zdisk/macbookair             14.9G   898G  14.6G  /Volumes/zdisk/macbookair
zdisk/macbookair@1225350709   125M      -  14.6G  -
zdisk/macbookair@1225351248   117M      -  14.6G  -
zdisk/macbookair@1225418584  21.7M      -  14.6G  -

This obviously isn’t as awesome as using Time Machine to recover my files because I don’t have a great UI, I have to run a script and generally have to know more about the system than a Time Machine user. However… I can update a VM without sending gigs of data over the internet to back it up or deal with not having a backup at all.

The only downside is that an empty backup still takes about 8 minutes to go through all my files. Next step would be to integrate into the fslogger into the solution and only look at those files that changed for sure.

Might as well put up my predictions for what will be announced at Macworld this year. I think it will be a good one.

• iMac, Mac Mini, Macbook and Macbook Pro lines upgraded to Penryn processors
  • Same prices, better specs
• Ultra portable MacBook Pro ($2-3K)
  • 64G SSD drive standard (32G and 128G available), no DVD drive
  • 12″ OLED screen
  • 8-12 hour battery life
• iPhone
  • SDK: Still WebKit-based but with native Javascript APIs for the phone
  • Better connectivity
  • Video recording
• Monitors
  • Same panel as the new 30″ dell
  • Built-in iSights
  • Same prices
• Blu-ray
  • Added as an option for MacPro and MacbookPro
  • Support for playing Blu-ray movies added to 10.5.2
• AppleTV / iTunes
  • Movie rentals
  • Software upgrade for current devices
  • Possibly a new version with a blu-ray player

All these I consider relatively expected and if they don’t show up, some people will be disappointed. Now for the truly outrageous prediction:

APPLE WILL WIMAX ENABLE THEIR LAPTOPS AND IPHONE BY PLACING NODES IN THEIR APPLE STORES AND STARBUCKS.

That would be the most awesome thing ever and hopefully that is what is implied with “Something is in the air…”. The other possibility is just 3G enabled systems or both for those times when you are out of range