Another project that was done in someone’s free time is Zach Grave’s TableSaw that helps you generate tables from URLs. I am going to combine these two things in this post to show you how to make a new table and an environment to more easily use it. Ultimately I could see TableSaw and the YQL Storage Editor combined, to that end, I’ve released the source for this blog post on github.

I’m going to use a real example that I just recently walked through, creating an open data table for the bit.ly API. The table will be pretty simple, just allowing you to pass in a bit.ly link and it will return the long URL for that link. First thing to do is to go to the bit.ly API documentation for the expand REST API. That gives us a nice example URL that we can use in Table Saw to generate a table:

http://api.bit.ly/expand?version=2.0.1&shortUrl=http://bit.ly/31IqMl&login=bitlyapidemo&apiKey=R_0da49e0a9118ff35f52f629d2d71bf07

If you head over to Table Saw and put that in the text field and load URL you get:

Initial generated table for api.bit.ly/expand

To complete the form and update, fill in an author name and the Produces should be JSON. One current bug with Table Saw is that it doesn’t seem to properly remember the Produces field so we will have to edit that after we grab the definition. Highlight everything under “Your new YQL Table!” and copy it into the clipboard and then go to the YQL Storage Editor. On that page you will see a set of execute/select/update text fields for the various urls that you need to use a yql.storage entry. Everything is empty right now and we want to create a new one, so click the New button. That should fill in those URL fields with brand new store:// URLs. Put focus on the Bespin editor component and paste in the table we copied. If you then click save it should say that it updated the entry in YQL with this new information. You’ll notice that there is a text area with the URLs listed out, if you want to keep track of yql.storage entries you need to keep these special identifiers around so you can perform the various operations on the content. I generally have a list that I keep locally of the entries that I am using. If you paste the list into the file as a comment you can then also just remember the ‘select’ URL in order to come back later and find the other two. Lastly, when the execute key is changed, the two links on the right get updated. The first link goes to the YQL console with your execute key used as an environment and the second link goes to the YQL console with your execute key used as a table definition. These two links can help you quickly try out changes that you make to your environment or table without a lot of cutting and pasting.

Table Saw isn’t yet perfect, so I make a few edits: change the produces from XML to JSON, add a description, reformat the sample query, set the default version to 2.0.1 and make all the other keys required, this should leave you with a page that looks like:

Edited version of the bit.ly table

When you click save it should alert you that it was saved successfully by saying that it was updated. You can now test this table in the YQL console with the following query:

use 'store://8tLqFvgCm8kdOAp60lPz24' as bit.ly; select * from bit.ly where version='2.0.1' and shortUrl='http://bit.ly/31IqMl' and login='bitlyapidemo' and apiKey='R_0da49e0a9118ff35f52f629d2d71bf07'

I noticed at this point that I forgot the itemPath and then set it to “json.results” in order to cut out the error messages and some other things that I wasn’t interested in. Nagesh Susarla had a great blog post on using YQL storage for secrets. We’ll also do that by using the YQL storage editor to create a new environment for us to use that could store our secret values that bit.ly needs in order to run the API. Create another YQL Storage Editor tab, click New and then we can make an environment that uses the table, sets the login and sets the API key:

Secret bit.ly environment with their API keys

Now using that execute URL as your env value in your query URLs you can have very simple queries against this API. For example, if you use this URL for the YQL Console you can use this much simpler query that gets all its authentication implicitly:

select * from bit.ly where shortUrl='http://bit.ly/31IqMl'

This allows you to not embed those valuable keys directly in your client software or even in your source control system but hidden behind what amounts to a very secure random password.

Update: One issue we still have is that after many hours of deployment with this configuration the heap fragments and we start to get concurrent mode failures.  We only saw this though during periods of peak activity.

We were struggling with some long pause times due to GC for YQL that we couldn’t stomach for our internal property SLAs. The secret turned out to be a collection of parameters for the Java 6 garbage collector:

-XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode \
-XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+CMSIncrementalPacing

I have so far been blown away by the results of this combination of parameters. Not only does it work very efficiently under moderate load, under crushing load it rises the occasion and aggressively keeps down the heap in such a way as to never cause the:

Concurrent Mode Failure
The concurrent collector uses one or more garbage collector threads that run simultaneously with the
application threads with the goal of completing the collection of the tenured and permanent generations
before either becomes full. As described above, in normal operation, the concurrent collector does most
of its tracing and sweeping work with the application threads still running, so only brief pauses are seen
by the application threads. However, if the concurrent collector is unable to finish reclaiming the
unreachable objects before the tenured generation fills up, or if an allocation cannot be satisfied with
the available free space blocks in the tenured generation, then the application is paused and the
collection is completed with all the application threads stopped. The inability to complete a collection
concurrently is referred to as concurrent mode failure and indicates the need to adjust the concurrent
collector parameters.

If you have any other GC secrets for the JVM, leave them here. Just as a heads up I compared it with all the other GC’s available for Java 6 including the experimental G1 and none of them were as effective.

Today YQL introduced a new feature that allows 3rd parties to define new tables and then share those table definitions with whomever they like for them to use. For example, let’s say you are the New York Times or you are a developer that likes the New York Times APIs and would like to make them more accessible to someone using YQL. Yesterday, they released the article search API, so I will use that one among others as an example, to get an api-key to execute these examples go to their developer site. This is a pretty sophisticated API that allows you to search using a variety of parameters. If you were to use YQL without modification, you would simply use the dynamic JSON endpoint to parse out the results from their service. The big issue with this though is that you would be unable to easily construct the URLs required and would have to write that code that collected all the parameters and created the URL. If you had a YQL table, those parameters would be defined and how they are expressed in the URL codified and you would be able to individually address the keys.

So without this ability you would use something like this:

select * from json where url='http://api.nytimes.com/svc/search/v1/article?api-key=...
&query=yahoo&begin_date=19990112&end_date=19993112' and
itemPath='json.results'

If we instead defined the API using the YQL open data tables specification you would be able to do this (by the way, don’t read the headlines, you’ll just be depressed):

use 'http://www.javarants.com/nyt/nyt.article.search.xml' as articles;
select * from articles where apikey='...' and query='yahoo' and begin_date='19990112' and
end_date='19993112'

Why is this superior? For a number of reasons. In the second case not only does it make it easier for anyone to use it, it also brings those keys from the query into columns which allows you to do joins that you cannot do with the first abstraction.  Here is an example join:

select * from bs where apikey='...' and query in ('yahoo', 'google', 'microsoft') and begin_date='19990112' and end_date='20000101'

This will actually do 3 searches for you in parallel and then return the combined result. By creating a YQL open table we can really off-load processing that you would normally do on your client or server to the YQL engine. You’ll note with that query that the user-time spent is actually about half or less than the actual service-time thus drastically decreasing the latency through asynchronous processing.

One of the really nice APIs out there is FriendFeed’s API. It really is very well designed and easily works with straight-forward table definitions. Here is an example of how to get the public feed:

use 'http://www.javarants.com/friendfeed/friendfeed.feeds.xml' as ff;
select * from ffconsole

By defining different endpoints with the same table definition we will automatically select the correct API based on the keys included in the query. Using that same table we can also get my public entries from twitter:

select * from ff where nickname='spullara' and service='twitter'console

This won’t include my private entries though. However, if you had my remotekey you could generate and pass YQL the authorization header required and it would pass it on to authenticate the API call. Another popular example that has been on the forums is the ability to use weather.com‘s API to tease out international locations. That is actually really easy and you can even use that data to join with their weather forecasts. Here is an example where we pull the weather for all the Moscow’s of the world:

use 'http://www.javarants.com/weather/weather.search.xml' as ws;use 'http://www.javarants.com/weather/weather.local.xml' as wl;
select * from wl where location in (select id from ws where query='moscow')console

I’ve actually started up a project on github called yql-tables to store useful table definitions and will be taking submissions from the community. You can try them out by ‘use’ing them directly from the git repository or by pulling them onto your own server accessible from the YQL servers.