Today YQL introduced a new feature that allows 3rd parties to define new tables and then share those table definitions with whomever they like for them to use. For example, let’s say you are the New York Times or you are a developer that likes the New York Times APIs and would like to make them more accessible to someone using YQL. Yesterday, they released the article search API, so I will use that one among others as an example, to get an api-key to execute these examples go to their developer site. This is a pretty sophisticated API that allows you to search using a variety of parameters. If you were to use YQL without modification, you would simply use the dynamic JSON endpoint to parse out the results from their service. The big issue with this though is that you would be unable to easily construct the URLs required and would have to write that code that collected all the parameters and created the URL. If you had a YQL table, those parameters would be defined and how they are expressed in the URL codified and you would be able to individually address the keys.

So without this ability you would use something like this:

select * from json where url='http://api.nytimes.com/svc/search/v1/article?api-key=...
&query=yahoo&begin_date=19990112&end_date=19993112' and
itemPath='json.results'

If we instead defined the API using the YQL open data tables specification you would be able to do this (by the way, don’t read the headlines, you’ll just be depressed):

use 'http://www.javarants.com/nyt/nyt.article.search.xml' as articles;
select * from articles where apikey='...' and query='yahoo' and begin_date='19990112' and
end_date='19993112'

Why is this superior? For a number of reasons. In the second case not only does it make it easier for anyone to use it, it also brings those keys from the query into columns which allows you to do joins that you cannot do with the first abstraction.  Here is an example join:

select * from bs where apikey='...' and query in ('yahoo', 'google', 'microsoft') and begin_date='19990112' and end_date='20000101'

This will actually do 3 searches for you in parallel and then return the combined result. By creating a YQL open table we can really off-load processing that you would normally do on your client or server to the YQL engine. You’ll note with that query that the user-time spent is actually about half or less than the actual service-time thus drastically decreasing the latency through asynchronous processing.

One of the really nice APIs out there is FriendFeed’s API. It really is very well designed and easily works with straight-forward table definitions. Here is an example of how to get the public feed:

use 'http://www.javarants.com/friendfeed/friendfeed.feeds.xml' as ff;
select * from ffconsole

By defining different endpoints with the same table definition we will automatically select the correct API based on the keys included in the query. Using that same table we can also get my public entries from twitter:

select * from ff where nickname='spullara' and service='twitter'console

This won’t include my private entries though. However, if you had my remotekey you could generate and pass YQL the authorization header required and it would pass it on to authenticate the API call. Another popular example that has been on the forums is the ability to use weather.com’s API to tease out international locations. That is actually really easy and you can even use that data to join with their weather forecasts. Here is an example where we pull the weather for all the Moscow’s of the world:

use 'http://www.javarants.com/weather/weather.search.xml' as ws;use 'http://www.javarants.com/weather/weather.local.xml' as wl;
select * from wl where location in (select id from ws where query='moscow')console

I’ve actually started up a project on github called yql-tables to store useful table definitions and will be taking submissions from the community. You can try them out by ‘use’ing them directly from the git repository or by pulling them onto your own server accessible from the YQL servers.

macpro:~ sam$ sqlite3 ~/Library/Mail/Envelope\ Index
SQLite version 3.6.3
Enter ".help" for instructions
Enter SQL statements terminated with a ";"
sqlite>


Everything about your mailboxes is stored within this database and the structure of the database is normalized so its very easy to navigate. The tables of most interest for mail analysis are:

sqlite> .tables
addresses              mailboxes              todo_notes
alarms                 messages               todos
associations           properties             todos_deleted_log
attachments            recipients             todos_server_snapshot
calendars              subjects
feeds                  threads

Fortunately, accessing a SQLite database is quite easy from just about any language that you decide to use. I’m just going to do all the queries in straight sqlite3 rather than a language, but they could be embedded in your application. First things first, copy your Envelope Index to another directory:

macpro:tmp sam$ cp ~/Library/Mail/Envelope\ Index .

Now you can use that database without worrying about messing up the locking or corrupting data while Mail.app is using it. Since we might as well do an example that is interesting rather than merely educational, how about we answer the question: “Who are my coworkers with whom that I collaborated?”. This is going to be a multi-query process to extract the information — there may be more efficient ways to do it — but think of this as instructive rather than prescriptive. First I need to limit the query to only those mailboxes which contain work email:

DROP TABLE coworkermailboxes;
CREATE TABLE coworkermailboxes(id);
CREATE INDEX coworkermailboxes_index ON coworkermailboxes(id);
INSERT INTO coworkermailboxes SELECT rowid FROM mailboxes
WHERE
url like 'imap://samp@snv-webmail.corp.yahoo.com/%' OR
url like 'imap://sam@mail.sampullara.com/Yahoo%20Inc%20Archive';

That gives us a table with several mailboxes that I have in Mail.app including Sent Messages. I would peruse the list of mailboxes to ensure that you are grabbing all the correct information. For me I had to also search my archives. Now I am going to take a series of steps to get to the final out put by iteratively processing successive tables of information. The first table, is a list of those people that you have both sent and received an email with directly (they were the sender and you were a receiver or you were the sender and they were the receiver):

DROP TABLE coworkers;
CREATE TABLE coworkers(id);
CREATE INDEX coworkers_index ON coworkers(id);
INSERT INTO coworkers SELECT a.rowid FROM addresses a, messages m, recipients r
WHERE
m.sender = a.rowid AND
m.mailbox IN (SELECT id FROM coworkermailboxes) AND
r.message_id = m.rowid AND r.address_id = 4
INTERSECT
SELECT a.rowid FROM addresses a, messages m, recipients r
WHERE
m.sender = 4 AND
m.mailbox IN (SELECT id FROM coworkermailboxes) AND
r.message_id = m.rowid AND r.address_id = a.rowid
;

Note I have directly inserted my addresses rowid into this query for the sender on the one hand and the receiver on the other. The next step will be to count the actual number of emails you have received from each of those on the list:

DROP TABLE coworkers2;
CREATE TABLE coworkers2(id, recv);
CREATE INDEX coworkers2_index ON coworkers2(id);
SELECT "Get the received mail";
INSERT INTO coworkers2 SELECT w.id, COUNT(*) FROM messages m, recipients r, coworkers w
WHERE m.sender = 4 AND
m.mailbox IN (SELECT id FROM coworkermailboxes) AND
r.message_id = m.rowid AND r.address_id = w.id
GROUP BY w.id ORDER BY COUNT(*)
;

Finally, we count the number of sent emails and also derive a ratio of sent/received so we can judge how collaborative the exchanges have been:

DROP TABLE coworkers3;
CREATE TABLE coworkers3(id, sent float, recv float, ratio float);
CREATE INDEX coworkers3_index ON coworkers3(id);
SELECT "Get the sent mail";
INSERT INTO coworkers3 SELECT w.id, COUNT(*), w.recv, COUNT(*)*1.0/w.recv FROM messages m, recipients r, coworkers2 w
WHERE
m.sender = w.id AND
m.mailbox IN (SELECT id FROM coworkermailboxes) AND
r.message_id = m.rowid AND r.address_id = 4
GROUP BY w.id ORDER BY COUNT(*)
;

You will now have a table named coworkers3 that can be mined for information about your level of correspondence with them. For example, here is way to find relatively equal sends and receives:

SELECT a.comment FROM addresses a, coworkers3 w
WHERE
a.rowid = w.id AND
ratio >= .5 AND
ratio <=2 AND
sent > 10
ORDER BY sent
LIMIT 20;

When I do this I see the people that either I use to find information or that use me to find information. Every interaction is usually a request and then a response. On the other hand, this query will find those that typically made announcements out to the groups that I also worked with:

SELECT a.comment FROM addresses a, coworkers3 w
WHERE
a.rowid = w.id AND
ratio <= 1 AND
sent > 10
ORDER BY ratio
LIMIT 20;

And so on. Adding more filters on top of this you could easily derive your team at work for a particular time period and other insights. With the wealth of information contained in this meta-data store you could figure out all kinds of things:

• Who sent you an email that you didn’t reply to yet?
• Who do you respond to the most quickly?
• Who responds to you most quickly?
• What are you and your coworkers approximate working hours?
• What groups of CCs could be made into aliases?

There really is no limit to how far the analysis could go. Ideally, it would be possible to setup a dashboard in Mail.app that let you cut and slice the data in a far more precise way than smart folders currently allow today. Maybe they should come out with super-sql-smart folders!

If I was going to declare an overall country winner for the games I would likely choose Australia.  I’ve highlighted the top-10 total medal winners in the table in blue and they are far ahead of anyone else in the top-10 on a people / medal basis.

 

Why 4hoursearch?  It took 4 hours to write the initial code, 4 hours for it to go from unknown to 20 hits / second, 4 hours looking for a domain name and 4 hours to build the brand new UI.  Fortunately, it won’t take 4 hours to find something with it

If you want the classic list of links, now enhanced with SearchMonkey results, you can always start here.

Updated Again: Yuil is dead. However, you can always get the same great search results here.

Updated: Using Glue I was able to add some simple category functionality.

I’m sure everyone saw the recent announcement of a new search engine, Cuil. I thought I would have a little fun with it and put together a quick parody of it by mashing up their UI and Yahoo!’s search results. As usual, the biggest problems I had were related to my pathetic Python skills. I’d love to add the category stuff in (Yahoo! has that info as you can see in search assist) but BOSS doesn’t yet have that in the API.  But it does have web and image search and even search suggestions. Here is the one, the only, the amazing:

It was great fun to hack together. Check out the BOSS APIs. Maybe I should have converted the UI to YUI as well…

What I would like to do is give some more options for the user.  Eventually I expect that SearchMonkey might allow per user preferences, but in the interim, I’ll produce links for 1.4.2, 1.5, 1.6 and a link to the entries package page:

This gives you direct access to other versions of the classes documentation from the search result page without having to qualify your search terms or scroll through pages of results looking for the one most relevant to you as a developer.  To create this enhanced result go to the SearchMonkey Developer Tool and create a new application.  Choose Enhanced Result rather than Infobar.  The URL pattern that I used was “*.java.sun.com/*”.  Obviously the real work is done in the PHP code for the appearance of the enhanced result:

public static function getOutput() {
    $ret = array();   

$classname = Data::get('yahoo:index/dc:identifier');
    $pattern = "/.*\/docs\/api\/(.*\/[A-Z].*).html/";
    if (preg_match($pattern, $classname, $matches)) {
        $classname = $matches[1];
        $link = $classname;
        $classname = str_replace("/", ".", $classname);
    } else {
        return $ret;
    }

/* pull the package reference out */
    if (preg_match("/(.*)\.([^.]+)/", $classname, $matches)) {
        $packagename = $matches[1];
    }

/* change the title to the name of the class */
    $ret['title'] = $classname;

// Deep links - up to 4
    $ret['links'][0]['text'] = "1.6.0";
    $ret['links'][0]['href'] = "http://java.sun.com/javase/6/docs/api/" . $link . ".html";
    $ret['links'][1]['text'] = "1.5.0";
    $ret['links'][1]['href'] = "http://java.sun.com/j2se/1.5.0/docs/api/" . $link . ".html";;
    $ret['links'][2]['text'] = "1.4.2";
    $ret['links'][2]['href'] = "http://java.sun.com/j2se/1.4.2/docs/api/" . $link . ".html";
    $ret['links'][3]['text'] = $packagename;
    $ret['links'][3]['href'] = "http://java.sun.com/javase/6/docs/api/" . str_replace(".", "/", $packagename) . "/package-summary.html";

return $ret;
}

Once that is done you confirm that you are finished and you will then see these enhanced result when you use alpha.search.yahoo.com.  Here are links to my applications that you can import into your own developer environment:

javadoc-smapp

mysql-smappкомпютри

from django.utils import simplejson

instead of including it. Makes this even easier.

Yahoo! Pipes has always been a great tool for manipulating data but often you have to go to great contortions to get it to do what you want because of its very simple data flow programming model.  Google’s App Engine opens up the possibility of extending Yahoo! Pipes in very interesting ways through Pipes’ Web Service operator.  Currently this operator sees little use as it requires you to be running an external server somewhere on the internet that is always available for the Pipe execution which is quite a high barrier to entry for the typical Pipes developer. Here is what a Pipe that is using web service looks like and our example pipe:

With the launch of Google App Engine there is now a very simple way to get code up on the internet quickly in order to include arbitrary processing in the interior of your Pipes.

To demonstrate how this works, let’s first build a very simple web service that simply mirrors the data that it receives from Pipes.   If you don’t have a Google App Engine account you can still follow along by download the SDK and executing all the stuff locally though it will have to be accessible from the public internet if you want Pipes to send you requests.

First create a new application directory:

mkdir pipes-mirror
cd pipes-mirror 

Now create an application descriptor called app.yaml:

application: javarants
version: 1
runtime: python
api_version: 1

handlers:
- url: /.*
  script: pipes.py

This application descriptor basically tells Google how to deploy your application. Your application name should match an application name that you create within the GAE administration tool:

Now we need to process the data coming from pipes. Pipes is going to pass this web service some data in JSON format and we need to parse it. GAE doesn’t include ‘simplejson‘ in the Python container so you are going to have to include it with your application. I downloaded simplejson-1.8.1 and symbolically linked its simplejson directory into my application directory. When the request comes in the JSON data will be in the ‘data‘ parameter so we are going to pull it out, parse it, grab the items array and write it back over the wire in pipes.py:

import simplejson
import wsgiref.handlers

from google.appengine.ext import webapp

class MirrorPipesWebService (webapp.RequestHandler):
	def post(self):
		data = self.request.get("data")
		obj = simplejson.loads(data)
		obj = obj["items"]
		self.response.content_type = "application/json"
		simplejson.dump(obj, self.response.out)

def main():
  application = webapp.WSGIApplication([('/mirror', MirrorPipesWebService)],
                                       debug=True)
  wsgiref.handlers.CGIHandler().run(application)

if __name__ == "__main__":
  main()

Now you should have a directory structure that looks a lot like this:

-rw-r--r--@ 1 sam  sam  106 Apr 13 18:55 app.yaml
-rw-r--r--  1 sam  sam  559 Apr 13 19:28 pipes.py
lrwxr-xr-x  1 sam  sam   47 Apr 13 17:40 simplejson -> /Users/sam/Software/simplejson-1.8.1/simplejson

Now that we have all the pieces we can deploy the application to GAE with a simple command from the GAE SDK:

appcfg.py update .

At this point you should be able to replace my web service URL that you find in my example Pipe with your application URL which will be

http://[application name].appspot.com/mirror

and get the same results as mine.

What kind of uses can you put this great power? I currently have a web service that I run that combines RSS entries from the same day into a single entry and have it deployed on my own server. I will likely port that to GAE as it doesn’t require a lot of CPU and it is a pain having to administer it. In fact, most of the functionality that you see in a service like FeedBurner would be easy to build on top of this framework. More exotic use cases can be found on Y! Pipes itself where at least one person uses web services to pass in photo URLs and return the coordinates of human faces in the images.