Javarants

Delicious/spullara on 11/17/2008

spullara — Mon, 17 Nov 2008 16:04:26 -0800

Murdoch to media: You dug yourself a huge hole | Coop's Corner - CNET News
Cool, I was one of the bloggers! :) "To make his point, Murdoch criticized the media reaction after bloggers debunked a "60 Minutes" report by former CBS anchor, Dan Rather, that President Bush had evaded service during his days in the National Guard. "Far from celebrating this citizen journalism, the establishment media reacted defensively. During an appearance on Fox News, a CBS executive attacked the bloggers in a statement that will go down in the annals of arrogance. '60 Minutes,' he said, was a professional organization with 'multiple layers of checks and balances.' By contrast, he dismissed the blogger as 'a guy sitting in his living room in his pajamas writing.' But eventually it was the guys sitting in their pajamas who forced Mr. Rather and his producer to resign."

Delicious/spullara on 11/15/2008

spullara — Sat, 15 Nov 2008 15:12:10 -0800

10 Tips for Optimizing MySQL Queries (That don’t suck) | 20bits
Excellent practical advice for optimizing your mysql (or almost any database) queries.

Delicious/spullara on 11/14/2008

spullara — Fri, 14 Nov 2008 14:49:19 -0800

The Death Of Packaged Software: Microsoft Store Opens On The Web For U.S. Shoppers
It is insane that Apple didn't do this first with iTunes.

Delicious/spullara on 11/10/2008

spullara — Mon, 10 Nov 2008 17:39:27 -0800

Someday You Will Not Hate the CSS3 Advanced Layout - Webmonkey
Seems a hell of a lot easier than floats and clears.

Delicious/spullara on 11/09/2008

spullara — Sun, 09 Nov 2008 10:23:27 -0800

Nagesh Susarla’s Blog » Blog Archive » OAuth-ify this: 2 Legged OAuth service for YQL
I have OAuthproxy.com if you want to deploy it on there :)

Delicious/spullara on 11/08/2008

spullara — Sat, 08 Nov 2008 23:48:16 -0800

unto.net
Seems like we ought to standardize the mapping of email addresses to openid providers through DNS or some similar channel.

Master of 500 Hats: Successful Developer Platforms Have 3 Things: Features, Users, Money.
"Yahoo continues to choke on both feet planted firmly next to its own tonsils, but they still do some great work on platforms & developer communities. If only they can show a little business leadership & stability, they'll be around."

Mini nuclear plants to power 20,000 homes | Environment | The Observer
This would quite possibly be the most awesome thing ever. Thank god people have woken up and realized that nuclear power can be used for good. Though it would really be $2500/home.

Delicious/spullara on 11/07/2008

spullara — Fri, 07 Nov 2008 16:52:46 -0800

Vivek Pandey's Blog: New and improved GlassFish gem
Super simple rails deployment.

Worst. Bug. Ever. | Ed Burnette’s Dev Connection | ZDNet.com
Hmm. That is an extremely amazing bug. Don't type rm -rf /.

Grails - GrailsUI Plugin
Very cool. Can't wait to play with this. I'm a big fan of YUI.

Proposition 8 Contributions
It seems like a transparent government could quickly lead to voter intimidation. Showing who/what individuals contributed to is about the same as exposing their actual vote. I could imagine this being used for evil.

Delicious/spullara on 11/04/2008

spullara — Tue, 04 Nov 2008 15:53:46 -0800

rectalogic: Virtualizing Mac OS X Leopard Client
Cool. Have to try this when I get home.

Oliver Burkeman's Campaign Diary: CNN's hologram plan for election night interviews |
This sounds really cool. I wonder if the Emperor will also be joining.

EFS-Web Help
Eletronic submission of patent applications... interesting.

Time Machine vs. ZFS + rsync

sam — Thu, 30 Oct 2008 20:17:18 -0700

Update: I actually got the fslogger thing at the end of this entry working so I can do incremental backups. Not really a product yet but it isn’t hard to do. Here is the super rough version of it.

I can’t stand inefficiency. Time Machine is fundamentally a very inefficient mechanism for backing up large files that change. So bad actually that most things like Parallels and VMWare disable backups of your disk images. Here is the basic algorithm:

1) Get the list of files that have changed since the last backup
2) Create new directory in backup store
3) Copy any file that has changed since the last backup
4) Create hard links to any file or even whole directory in the new backup to the last backup for any file that has not changed

Step 1 is pretty efficient for Time Machine as they keep hooks into the filesystem to track those changes as they occur. Step 2 is obviously easy. Step 3 is a doosy. If you change 1 byte in a VMWare image it will copy the several gigs over to the backup store. Not a great result from such small change and that would quickly consume your disk flushing valuable older changes out of the system. Step 4 is also very efficient because hard links are trivial to create and use virtually no space, though they did have to make special changes to HFS+ so that you could hard link directories to make Time Machine more efficient.

The obvious big problem here is that in the case that a file changes at all you need to copy the whole thing to you backup device. Not that viable over the internet or even WiFi for really big files that are updated often like VM images. You might have wondered why Apple is considering integrating ZFS directly into Mac OS X, now you know why. ZFS lets you do something very special: create a snapshot of a whole filesystem. Essentially a copy of that filesystem at a particular point in time and they do this without copying whole files when they change but instead at the block level. This amazing capability is critical in this more efficient way to backup your system with multi-level snapshots.

Enter rsync. Rsync has been around for a long time. It is used by system adminstrators everywhere to efficiently update files in one location with files from another location, even over the internet. It does this by comparing them at the block level and only sending diffs when needed to update files on the other end. Using the right command line options you can essentially make one filesystem look like a carbon copy of another filesystem. Using this in combination you can make a backup solution that is much better than most out there:

1) Rsync your current filesystem to a ZFS filesystem — remote or attached storage
2) Take a snapshot of the resulting filesystem to forever capture its state

Those are the two steps. Nothing more. Here is the script that I use to backup my Macbook Air to my server at home:

#!/bin/sh
cd /Users
time rsync -av --delete sam 192.168.1.90:/Volumes/zdisk/macbookair
ssh 192.168.1.90 sudo zfs snapshot zdisk/macbookair@`date "+%s"`

This results in a set of filesytems that looks like this:

zdisk/macbookair 14.9G 898G 14.6G /Volumes/zdisk/macbookair
zdisk/macbookair@1225350709 125M - 14.6G -
zdisk/macbookair@1225351248 117M - 14.6G -
zdisk/macbookair@1225418584 21.7M - 14.6G -

This obviously isn’t as awesome as using Time Machine to recover my files because I don’t have a great UI, I have to run a script and generally have to know more about the system than a Time Machine user. However… I can update a VM without sending gigs of data over the internet to back it up or deal with not having a backup at all.

The only downside is that an empty backup still takes about 8 minutes to go through all my files. Next step would be to integrate into the fslogger into the solution and only look at those files that changed for sure.

Yahoo! Application Platform (YAP) and Yahoo! Query Language (YQL) launch today

sam — Tue, 28 Oct 2008 15:40:32 -0700

Since I moved into the platform group at the beginning of the year I had worked with the YAP and YQL teams to help them define their strategy and direction but without being part of the day-to-day operations. In August, the head of the Y!OS project asked me to step in to take them through their final run to launch. It has been a great couple of months working with the teams. They both had an amazing showing at Hack Day and now today we are launching the platforms worldwide.

Over the next few months and into 2009 YAP will be rolling across Yahoo!. Developers will be able to embed their applications in places like My Yahoo, Frontpage, Media properties and Mail. YQL will continue to be extended to other services by Yahoo! and through a plugin API in the future. Please check out the announcements at the Yodel, YDN and OpenSocial blog.

2008 Olympic Medal Counts by Population

sam — Sun, 24 Aug 2008 12:25:18 -0700

There are obviously a lot of ways to measure how well a country did at the Olympics. This post takes a view that we should look at how many people the country had to draw on in order to send the athletes to China to compete. There are a lot of problems with this including: ex-pats competing for their home country, vast disparity in wealth between countries and the relative interest in the Olympic games of the cultures. One of the things that jumps out immediately is that island nations that draw on a larger related population do very well in the games. They likely have inherited not only the interest in the competition but are also wealthy enough to train and compete in the games.

One of the things that was interesting in putting this together was that I eventually settled on PowerSet as the easiest way to lookup the population of a country. Both Yahoo and Google will give shortcuts to many of them, they don’t do it for all of them. Though PowerSet gets this population data through Freebase, Freebase itself doesn’t have a great search interface.

If I was going to declare an overall country winner for the games I would likely choose Australia. I’ve highlighted the top-10 total medal winners in the table in blue and they are far ahead of anyone else in the top-10 on a people / medal basis.

Yuil is dead! 4hoursearch is now online.

sam — Fri, 01 Aug 2008 15:22:07 -0700

As this was really just a demonstration of the power of Yahoo! BOSS, I have brought the site back as a demonstration site. Additionally, Yahoo! is making the source code to the new site available so anyone with a knack for Python, HTML and CSS can take a swipe at making a better search experience. In order to make a nice UI I teamed up with another Sam, Sam Lind. I put together the skeleton using Yahoo!’s amazing YUI tools and he created the look and feel. Please try it out and take advantage of Yahoo!’s open search API:

Why 4hoursearch? It took 4 hours to write the initial code, 4 hours for it to go from unknown to 20 hits / second, 4 hours looking for a domain name and 4 hours to build the brand new UI. Fortunately, it won’t take 4 hours to find something with it

If you want the classic list of links, now enhanced with SearchMonkey results, you can always start here.

Yahoo! BOSS is easy — meet Yuil

sam — Wed, 30 Jul 2008 02:01:40 -0700

Updated Yet Again: Relaunched as 4hoursearch including the source code. See this blog entry.

Updated Again: Yuil is dead. However, you can always get the same great search results here.

Updated: Using Glue I was able to add some simple category functionality.

I’m sure everyone saw the recent announcement of a new search engine, Cuil. I thought I would have a little fun with it and put together a quick parody of it by mashing up their UI and Yahoo!’s search results. As usual, the biggest problems I had were related to my pathetic Python skills. I’d love to add the category stuff in (Yahoo! has that info as you can see in search assist) but BOSS doesn’t yet have that in the API. But it does have web and image search and even search suggestions. Here is the one, the only, the amazing:

It was great fun to hack together. Check out the BOSS APIs. Maybe I should have converted the UI to YUI as well…

Better Javadoc results using SearchMonkey

sam — Mon, 19 May 2008 12:00:13 -0700

When you are searching for things like java.util.HashMap one of the issues that you run into is that it will give you the result with the highest rank which more often than not is the 1.4.2 version of the documentation. I’ve moved on from that version of Java and would much rather see results for version 6. I actually did this plugin back in December for the first SearchMonkey hackday and won “most useful” as it could be extended to any type of versioned documentation you might find on the web. Today I’ll also include my plugin for MySQL but I’ll use Java as the example.

Here is the normal search result that you get on Yahoo:

What I would like to do is give some more options for the user. Eventually I expect that SearchMonkey might allow per user preferences, but in the interim, I’ll produce links for 1.4.2, 1.5, 1.6 and a link to the entries package page:

This gives you direct access to other versions of the classes documentation from the search result page without having to qualify your search terms or scroll through pages of results looking for the one most relevant to you as a developer. To create this enhanced result go to the SearchMonkey Developer Tool and create a new application. Choose Enhanced Result rather than Infobar. The URL pattern that I used was “*.java.sun.com/*”. Obviously the real work is done in the PHP code for the appearance of the enhanced result:

public static function getOutput() {
    $ret = array();    $classname = Data::get('yahoo:index/dc:identifier');
    $pattern = "/.*\/docs\/api\/(.*\/[A-Z].*).html/";
    if (preg_match($pattern, $classname, $matches)) {
    $classname = $matches[1];
    $link = $classname;
    $classname = str_replace("/", ".", $classname);
    } else {
    return $ret;
    } /* pull the package reference out */
    if (preg_match("/(.*)\.([^.]+)/", $classname, $matches)) {
    $packagename = $matches[1];
    } /* change the title to the name of the class */
    $ret['title'] = $classname; // Deep links - up to 4
    $ret['links'][0]['text'] = "1.6.0";
    $ret['links'][0]['href'] = "http://java.sun.com/javase/6/docs/api/" . $link . ".html";
    $ret['links'][1]['text'] = "1.5.0";
    $ret['links'][1]['href'] = "http://java.sun.com/j2se/1.5.0/docs/api/" . $link . ".html";;
    $ret['links'][2]['text'] = "1.4.2";
    $ret['links'][2]['href'] = "http://java.sun.com/j2se/1.4.2/docs/api/" . $link . ".html";
    $ret['links'][3]['text'] = $packagename;
    $ret['links'][3]['href'] = "http://java.sun.com/javase/6/docs/api/" . str_replace(".", "/", $packagename) . "/package-summary.html"; return $ret;
}

Once that is done you confirm that you are finished and you will then see these enhanced result when you use alpha.search.yahoo.com. Here are links to my applications that you can import into your own developer environment:

javadoc-smapp

mysql-smappкомпютри

Code Changes on 05/07/2008

Wed, 07 May 2008 16:02:02 -0700

dbmapper 23: [ISSUE-6] close those result sets.
spullara on Wed May 07 16:02:02 PDT 2008

Changed paths:
M /trunk/src/main/java/com/moonspider/dbmap/DB.java
M /trunk/src/main/java/com/moonspider/dbmap/DBTable.java

Idiomatic Python?

sam — Wed, 23 Apr 2008 11:19:51 -0700

I’ve been working my way through compiling Java into Python code but the Python back end of my isn’t that good (my brain). I would call my stage of Python development the “magic incantation” stage. This is the stage where you really aren’t comfortable yet with the way things work in a new language but you can still get things done by miming other developers. I’ve also had some help from some friends on Twitter: @lhl, @precipice and @jkwatson. My distributed information system is now getting some redundancy. Little did they know that I was doing parallel invocations of identical requests for reliability and incrementally higher performance — and the results were verified using a quorum of responders.

Here is my first service that I am porting. It takes an RSS feed (in JSON format from Pipes) and combines all the entries from each day into a single entry:

import logging
import wsgiref.handlers from datetime import date
from google.appengine.ext import webapp
from django.utils import simplejson class DayBinPipesWebService (webapp.RequestHandler): def post(self): now = date.today() now = now.strftime("%m/%d/%Y") data = self.request.get("data") items = simplejson.loads(data)["items"] bins = {} for item in items: published = item["y:published"] updateDay = "%(month)02d/%(day)02d/%(year)04d" % published if now != updateDay: bin = bins.get(updateDay, []) bin.append(item) bins[updateDay] = bin entries = [] for bin in bins.items(): dayDate = bin[0] binEntries = bin[1] first = binEntries[0].copy() first["description"] = "" for e in binEntries: first["description"] += "<p><a href='%(link)s'>%(title)s</a><br>%(description)s</p>" % e first["title"] = "Items from " + dayDate first["link"] = "" entries.append(first) self.response.content_type = "application/json" simplejson.dump(entries, self.response.out)

How would you write this in idiomatic Python as opposed to my rudimentary translation? Would you change the whole design?

Tivo targeted advertising

sam — Wed, 23 Apr 2008 00:52:43 -0700

Tivo targeted advertising, originally uploaded by Sam Pullara.

This looks like it might be both effective and also something that TV advertisers would like to buy.

Using Google App Engine to Extend Yahoo! Pipes

sam — Sun, 13 Apr 2008 19:59:45 -0700

Update: A commenter pointed out that you can

from django.utils import simplejson

instead of including it. Makes this even easier.

Yahoo! Pipes has always been a great tool for manipulating data but often you have to go to great contortions to get it to do what you want because of its very simple data flow programming model. Google’s App Engine opens up the possibility of extending Yahoo! Pipes in very interesting ways through Pipes’ Web Service operator. Currently this operator sees little use as it requires you to be running an external server somewhere on the internet that is always available for the Pipe execution which is quite a high barrier to entry for the typical Pipes developer. Here is what a Pipe that is using web service looks like and our example pipe:

With the launch of Google App Engine there is now a very simple way to get code up on the internet quickly in order to include arbitrary processing in the interior of your Pipes.

To demonstrate how this works, let’s first build a very simple web service that simply mirrors the data that it receives from Pipes. If you don’t have a Google App Engine account you can still follow along by download the SDK and executing all the stuff locally though it will have to be accessible from the public internet if you want Pipes to send you requests.

First create a new application directory:

mkdir pipes-mirror
cd pipes-mirror

Now create an application descriptor called app.yaml:

application: javarants
version: 1
runtime: python
api_version: 1 handlers:
- url: /.* script: pipes.py

This application descriptor basically tells Google how to deploy your application. Your application name should match an application name that you create within the GAE administration tool:

Now we need to process the data coming from pipes. Pipes is going to pass this web service some data in JSON format and we need to parse it. GAE doesn’t include ‘simplejson‘ in the Python container so you are going to have to include it with your application. I downloaded simplejson-1.8.1 and symbolically linked its simplejson directory into my application directory. When the request comes in the JSON data will be in the ‘data‘ parameter so we are going to pull it out, parse it, grab the items array and write it back over the wire in pipes.py:

import simplejson
import wsgiref.handlers from google.appengine.ext import webapp class MirrorPipesWebService (webapp.RequestHandler): def post(self): data = self.request.get("data") obj = simplejson.loads(data) obj = obj["items"] self.response.content_type = "application/json"
 simplejson.dump(obj, self.response.out) def main(): application = webapp.WSGIApplication([('/mirror', MirrorPipesWebService)], debug=True) wsgiref.handlers.CGIHandler().run(application) if __name__ == "__main__": main()

Now you should have a directory structure that looks a lot like this:

-rw-r--r--@ 1 sam sam 106 Apr 13 18:55 app.yaml
-rw-r--r-- 1 sam sam 559 Apr 13 19:28 pipes.py
lrwxr-xr-x 1 sam sam 47 Apr 13 17:40 simplejson -> /Users/sam/Software/simplejson-1.8.1/simplejson

Now that we have all the pieces we can deploy the application to GAE with a simple command from the GAE SDK:

appcfg.py update .

At this point you should be able to replace my web service URL that you find in my example Pipe with your application URL which will be

http://[application name].appspot.com/mirror

and get the same results as mine.

What kind of uses can you put this great power? I currently have a web service that I run that combines RSS entries from the same day into a single entry and have it deployed on my own server. I will likely port that to GAE as it doesn’t require a lot of CPU and it is a pain having to administer it. In fact, most of the functionality that you see in a service like FeedBurner would be easy to build on top of this framework. More exotic use cases can be found on Y! Pipes itself where at least one person uses web services to pass in photo URLs and return the coordinates of human faces in the images.

Code Changes on 03/25/2008

Tue, 25 Mar 2008 10:21:46 -0700

cli-parser 17: fix array defaut values in the usage printout
spullara on Tue Mar 25 10:21:46 PDT 2008

Changed paths:
M /trunk/src/main/java/com/sampullara/cli/Args.java

JPA 2.0 with Criteria

sam — Sat, 15 Mar 2008 18:51:12 -0700

(see: JSR 317 Persistently Improving)

I love the idea of adding a criteria API to JPA, the only thing I hope that they do differently than Hibernate is to implement that API in addition to string queries. In Gauntlet we had issues where we wanted to use EJB-QL for selecting the right data and then a criteria-like API for applying security and filtering constraints on the query. We ended up writing a criteria-like API that augmented the WHERE clause of the query to get the behavior that we needed (like described here). For example, you could do this:

Query q = em.createQuery("SELECT p FROM Project p");
q.addExpression(Expression.notEqual("id", 2));

Or something like that. This would give you the best of both worlds, where you have the expressiveness of the textual query and the ability to further hone that query programmatically.

Code Changes on 03/11/2008

Tue, 11 Mar 2008 03:57:17 -0700

dbmigrate 55: Ignore maven target directory
mick.brooks on Tue Mar 11 03:57:17 PDT 2008

Changed paths:
M /trunk

dbmigrate 54: Add a simple parser for SQL statements: fixes case where a semi-colon character inside a single-quoted string would prematurely terminate the current statement.
mick.brooks on Tue Mar 11 03:52:15 PDT 2008

Changed paths:
A /trunk/src/test/java/com/sampullara/db/SqlStatementParserTest.java
M /trunk/src/main/java/com/sampullara/db/Migrate.java
A /trunk/src/main/java/com/sampullara/db/SqlStatementParser.java

Code Changes on 03/06/2008

Thu, 06 Mar 2008 09:49:05 -0800

dbmapper 22: map BINARY columns to byte[] instead of java.io.Serializable
realjolly on Thu Mar 06 09:49:05 PST 2008

Changed paths:
M /trunk/src/main/java/com/moonspider/dbmap/ColumnConfig.java

Code Changes on 12/23/2007

Sun, 23 Dec 2007 16:00:24 -0800

dbmigrate 53: update dependencies and make the scripts a little more undeprecated
spullara on Sun Dec 23 16:00:24 PST 2007

Changed paths:
M /trunk/plugin/scripts/CreateMigration.groovy
M /trunk/plugin/scripts/Migrate.groovy
M /trunk/pom.xml

Code Changes on 12/15/2007

Sat, 15 Dec 2007 13:43:10 -0800

dbmigrate 52: add a full assembly packaging step for making a standalone executable
spullara on Sat Dec 15 13:43:10 PST 2007

Changed paths:
D /trunk/test
M /trunk/pom.xml

dbmapper 21: add a full assembly packaging step for making a standalone executable
spullara on Sat Dec 15 13:41:13 PST 2007

Changed paths:
M /trunk/pom.xml

dbmapper 20: convert to maven project
spullara on Sat Dec 15 13:27:59 PST 2007

Changed paths:
A /trunk/src/main/resources/com/moonspider/dbmap/hibernate.template
A /trunk/src/main
A /trunk/src/main/java
A /trunk/src/test/resources
A /trunk/src/main/resources/com/moonspider/dbmap/jpa.template
A /trunk/src/test/java/com
D /trunk/src/com
A /trunk/src/main/resources/com/moonspider/dbmap
R /trunk/src/test/java/com/moonspider
A /trunk/src/main/resources/com/moonspider
D /trunk/test
D /trunk/dbmap.ipr
D /trunk/test-data/generate.xml.template
D /trunk/dbmap.iml
D /trunk/src/main/java/com/moonspider/dbmap/gorm.template
A /trunk/src/test
R /trunk/src/main/java/com/moonspider
A /trunk/pom.xml
A /trunk/src/main/java/com
A /trunk/src/test/resources/generate.xml
M /trunk/src/test/java/com/moonspider/dbmap/GenerateBeansTest.java
A /trunk/src/main/resources
A /trunk/src/test/java
D /trunk/src/main/java/com/moonspider/dbmap/jpa.template
D /trunk/build.xml
D /trunk/src/main/java/com/moonspider/dbmap/hibernate.template
A /trunk/src/main/resources/com
A /trunk/src/main/resources/com/moonspider/dbmap/gorm.template

dbmapper 19: add index verification and some more types
spullara on Sat Dec 15 13:03:47 PST 2007

Changed paths:
D /trunk/test-lib
M /trunk/dbmap.ipr
M /trunk/src/com/moonspider/dbmap/DB.java
M /trunk/src/com/moonspider/dbmap/DBTable.java
M /trunk/src/com/moonspider/dbmap/ColumnConfig.java

dbmigrate 51: have to include the custom repo since i don't have it in the settings
spullara on Sat Dec 15 12:58:29 PST 2007

Changed paths:
M /trunk/pom.xml

cli-parser 16: get rid of intellij files and try and use maven only
spullara on Sat Dec 15 12:19:58 PST 2007

Changed paths:
D /trunk/cli.iml
D /trunk/cli.ipr

cli-parser 15: convert to maven project
spullara on Sat Dec 15 12:14:37 PST 2007

Changed paths:
A /trunk/src/test/java
D /trunk/test/com
A /trunk/src/main
D /trunk/build.xml
A /trunk/src/main/java
D /trunk/test-lib
A /trunk/src/test/java/com
A /trunk/src/test
D /trunk/ant-lib
A /trunk/pom.xml
A /trunk/src/main/java/com
D /trunk/src/com

Code Changes on 12/05/2007

Wed, 05 Dec 2007 13:31:51 -0800

dbmapper 18: make the test very generic
spullara on Wed Dec 05 13:31:51 PST 2007

Changed paths:
M /trunk/test-data/simplelock.sql
M /trunk/test/com/moonspider/dbmap/GenerateBeansTest.java
M /trunk/test-data/generate.xml.template

Code Changes on 12/04/2007

Tue, 04 Dec 2007 23:16:14 -0800

dbmapper 17: add a test case that shows dbmapper generating illegal java when a database column is the same name as the table it is a foreign key to. The fix is probably to disambiguate with something like set_Foo() and get_Foo() for the field setter/getter and leave the Relation setter/getter as setFoo() and getFoo()
realjolly on Tue Dec 04 23:16:14 PST 2007

Changed paths:
M /trunk/test/com/moonspider/dbmap/GenerateBeansTest.java
A /trunk/test-data/test1.sql

Code Changes on 11/18/2007

Sun, 18 Nov 2007 14:53:11 -0800

dbmigrate 49: modify for the new PropertiesArgs stuff in cli
spullara on Sun Nov 18 14:53:11 PST 2007

Changed paths:
A /trunk/lib/cli-r14.jar
D /trunk/lib/cli-10.jar
M /trunk/dbmigrate.iml
M /trunk/dbmigrate.ipr
M /trunk/src/com/sampullara/db/Migrate.java

dbmigrate 48: fix locking on mysql
spullara on Sun Nov 18 10:58:13 PST 2007

Changed paths:
M /trunk/test/com/sampullara/db/MigrateTest.java
M /trunk/dbmigrate.iml
M /trunk/dbmigrate.ipr
M /trunk/build.xml
M /trunk/src/com/sampullara/db/db.properties
M /trunk/src/com/sampullara/db/Migrate.java

cli-parser 14: Extract the properties file stuff so you don't have to include it. Also makes it easier to implement other formats. Might want to make this more generic at some point.
spullara on Sun Nov 18 09:29:34 PST 2007

Changed paths:
A /trunk/test/com/sampullara/cli/PropertiesArgsTest.java
M /trunk/src/com/sampullara/cli/Args.java
M /trunk/test/com/sampullara/cli/ArgsTest.java
A /trunk/src/com/sampullara/cli/PropertiesArgs.java

Code Changes on 11/04/2007

Sun, 04 Nov 2007 15:35:28 -0800

imapsync 4: Add the ability to connect to IMAPS servers that don't have valid certificates.
spullara on Sun Nov 04 15:35:28 PST 2007

Changed paths:
M /trunk/src/com/sampullara/imapsync/IMAPSync.java
M /trunk/IMAPSync.ipr
A /trunk/src/com/sampullara/imapsync/ssl
A /trunk/src/com/sampullara/imapsync/ssl/DummyTrustManager.java
A /trunk/src/com/sampullara/imapsync/ssl/DummySSLSocketFactory.java

imapsync 3: make the build a little better
spullara on Sun Nov 04 14:25:16 PST 2007

Changed paths:
D /trunk/IMAPSync.iws
M /trunk/module_imapsync.xml

imapsync 2: initial import of project
spullara on Sun Nov 04 14:06:31 PST 2007

Changed paths:
A /trunk/IMAPSync.iml
A /trunk/LICENCES/LICENSE.commons-logging
A /trunk/lib/mstor/jdom.jar
A /trunk/LICENCES/LICENCE.CDDL
A /trunk/src/com
A /trunk/src/com/sampullara/imapsync
A /trunk/IMAPSync.iws
A /trunk/LICENCES/LICENSE.mstor
A /trunk/lib/mstor/xstream.jar
A /trunk/lib/mstor/mstor.jar
A /trunk/LICENCES
A /trunk/src
A /trunk/lib
A /trunk/LICENCES/LICENSE.cli
A /trunk/IMAPSync.ipr
A /trunk/src/com/sampullara
A /trunk/lib/mail-1.4.jar
A /trunk/LICENCES/LICENSE.xstream
A /trunk/src/com/sampullara/imapsync/IMAPSync.java
A /trunk/LICENCES/LICENSE.jdom
A /trunk/imapsync.properties
A /trunk/build.xml
A /trunk/README
A /trunk/lib/mstor/commons-logging.jar
A /trunk/lib/cli-969.jar
A /trunk/module_imapsync.xml
A /trunk/lib/mstor
A /trunk/lib/activation-1.1.jar

imapsync 1: Initial directory structure.
null on Sun Nov 04 14:03:47 PST 2007

Changed paths:
A /branches
A /trunk
A /tags

Code Changes on 10/21/2007

Sun, 21 Oct 2007 13:18:09 -0700

dbmigrate 47: add the ability to use the Migrate class as an Ant task
spullara on Sun Oct 21 13:18:09 PDT 2007

Changed paths:
M /trunk/test/com/sampullara/db/MigrateTest.java
A /trunk/lib/groovy-all-1.1-rc-1.jar
M /trunk/dbmigrate.iml
M /trunk/dbmigrate.ipr
M /trunk/build.xml
D /trunk/lib/groovy-all-1.1-beta-3.jar
M /trunk/src/com/sampullara/db/Migrate.java

cli-parser 13: make it show that its a revision number, not a version
spullara on Sun Oct 21 12:06:43 PDT 2007

Changed paths:
M /trunk/build.xml

cli-parser 12: Increase code coverage in the tests and optimize some code paths.
spullara on Sun Oct 21 12:04:44 PDT 2007

Changed paths:
M /trunk/cli.ipr
M /trunk/src/com/sampullara/cli/Args.java
M /trunk/test/com/sampullara/cli/ArgsTest.java

Code Changes on 10/20/2007

Sat, 20 Oct 2007 17:40:56 -0700

dbmapper 16: Edited wiki page through web user interface.
spullara on Sat Oct 20 17:40:56 PDT 2007

Changed paths:
M /wiki/Grails.wiki

dbmapper 15: fix issues 1 and 2. added CHAR as a recognized type. added Schema as a parameter to the metadata.
spullara on Sat Oct 20 11:26:21 PDT 2007

Changed paths:
M /trunk/dbmap.ipr
M /trunk/src/com/moonspider/dbmap/DB.java
M /trunk/src/com/moonspider/dbmap/Generator.java
M /trunk/src/com/moonspider/dbmap/DBTable.java
M /trunk/src/com/moonspider/dbmap/GenerateConfig.java
M /trunk/test/com/moonspider/dbmap/DBTest.java
M /trunk/src/com/moonspider/dbmap/ColumnConfig.java

Code Changes on 10/15/2007

Mon, 15 Oct 2007 11:40:49 -0700

dbmapper 14: add eclipse project files
dave.brown on Mon Oct 15 11:40:49 PDT 2007

Changed paths:
A /trunk/.classpath
A /trunk/.project

Code Changes on 10/13/2007

Sat, 13 Oct 2007 13:23:20 -0700

cli-parser 11: left out the description for flags in the usage message
spullara on Sat Oct 13 13:23:20 PDT 2007

Changed paths:
M /trunk/src/com/sampullara/cli/Args.java

Code Changes on 10/07/2007

Sun, 07 Oct 2007 16:11:12 -0700

dbmigrate 46: add externals
spullara on Sun Oct 07 16:11:12 PDT 2007

Changed paths:
A /trunk/checkout

Code Changes on 09/15/2007

Sat, 15 Sep 2007 20:39:20 -0700

cli-parser 10: fix it up so that the name is set propertly in the error message
spullara on Sat Sep 15 20:39:20 PDT 2007

Changed paths:
M /trunk/src/com/sampullara/cli/Args.java

Code Changes on 09/08/2007

Sat, 08 Sep 2007 21:59:29 -0700

cli-parser 9: get rid of some of the build stuff
spullara on Sat Sep 08 21:59:29 PDT 2007

Changed paths:
D /trunk/emma-macros.xml
M /trunk/build.xml
D /trunk/test-lib/emma-2.0.5312.jar
D /trunk/test-lib/emma_ant-2.0.5312.jar
D /trunk/repository-macros.xml

cli-parser 8: add the ability to use a properties file rather than command line arguments
spullara on Sat Sep 08 17:25:12 PDT 2007

Changed paths:
M /trunk/cli.iml
M /trunk/cli.ipr
M /trunk/src/com/sampullara/cli/Args.java
M /trunk/test/com/sampullara/cli/ArgsTest.java
D /trunk/cli.iws

Using the Yahoo! Mail SOAP API 1.1 from Java's JAX-WS 2.1

Fri, 31 Aug 2007 13:22:39 -0700

On YDN they have samples and documentation on how to use the Yahoo! Mail SOAP API from Axis2. I'm not a big fan of that method so I went ahead and used JAX-WS to do my dirty work. As an example, I will build an RSS feed of the users unread messages.

First get JAX-WS 2.1.x or possibly just JDK 1.6 (though I did all my testing with the former and JDK 1.5). Since its easiest to work with typed APIs we are first going to generate the classes needed to talk to the the Yahoo! Mail web service. From their site, we find that the latest WSDL URL. To generate the API from the WSDL we use the 'wsimport' command from JAX-WS like this:

wsimport.sh -extension -s src -p com.yahoo.mail http://mail.yahooapis.com/ws/mail/v1.1/wsdl

The key command line option there is the '-extension' option as the Yahoo! Mail WSDL has a few things that by default would get named the same thing by the schema compiler. By using Sun's extensions we can automatically rename them rather than making our own binding. This will generate 120+ classes representing each part of the complex API along with some special classes like ObjectFactory. This gives us the foundation for accessing Y! Mail but there are still a few quirks that we need to understand. Normally when you use JAX-WS you would simply access the web service using the main Ymws class that was generated like this:

 // Instantiate the SOAP proxy  Ymws service = new Ymws(); YmwsPortType stub = service.getYmws();

Then you would be able to make calls on that stub directly like this:

 UserData userData = stub.getUserData();

You'd find though if you tried to do this that you would not be authenticated with the service nor would there be anyway to select what user for which you are making this call. Yahoo! Mail's web services have their own authentication scheme that is built on Yahoo's BBAuth -- a 3rd party authentication system. In order to make use of BBAuth you will need to register your application with Yahoo. Make sure that you use a publicly available URL for your application and also select the third option at the bottom: "Yahoo! Mail (via BBAuth) with Read/Write access" so that you will be able to use this application ID to access the Y! Mail API. Once you have registered you will be asked to authenticate your URL by placing a special file at the root of the domain of your URL. This means you needs write access to the root of the web server so don't attempt this on a domain that you don't control in that way. After this is complete you continue to the success page where it provides you with your application ID (appid) and your shared secret (secret). These will be required for you to access the Y! Mail APIs on behalf of Y! users.

The core of the way BBAuth works is for you to redirect to their server when you want to authenticate a user and then they send back the token that is required to access Y! as that user to the registered application URL. Here is the code that we can use to generate the URL required to get authenticated:

 long ts = date.getTime() / 1000; String uri; uri = "/WSLogin/V1/wslogin?send_userhash=1&appid=" + URLEncoder.encode(appid, "UTF-8") + "&ts=" + ts; MessageDigest md; md = MessageDigest.getInstance("md5"); String sig = new BigInteger(1, md.digest((uri + secret).getBytes())).toString(16); return LOGIN_URL + uri + "&sig=" + sig;

Essentially this code creates a URI with our appid and signs it with the secret which is then used to create a URL that includes the signature. This ensures that only someone with the secret can work on behalf of the application that we registered. Once the user is authenticated you will receive a callback at the registered URL that includes the information passed here plus a token that can be used to retrieve a WSSID and cookie that can then be used to construct authenticated web service requests. There is an additional login URL parameter that you can pass called appdata that will be returned to you when Y! redirects the user back to your application. The token that is returned is generally good for 2 weeks of authenticated access. The send_userhash=1 option tells Yahoo to return to us a unique identifier tied to your application id that will always be the same so you can use it to tie back to a particular user in your application.

Now that we have gotten the token back from Yahoo we can get our WSSID and Cookie. Included with the Yahoo! Mail sample code they have an inner class called BrowserBasedAuthManager. We'll just use that rather than rewrite it but basically it does something similar to the code above and retrieves the two values from an XML document returned from a URL:

 // Instantiate the auth manager and set it up with the date, application ID,  // shared secret and the user token.  BrowserBasedAuthManager authManager = new BrowserBasedAuthManager(date, appid, secret, token);

These two values, wssid and cookie, are then needed to construct our web service requests. JAX-WS doesn't expose this functionality directly in the API but instead allows you to set various properties on the request context:

 Map<String, Object> requestContext = ((BindingProvider) stub).getRequestContext(); requestContext.put(BindingProvider.ENDPOINT_ADDRESS_PROPERTY,  "http://mail.yahooapis.com/ws/mail/v1.1/soap?appid=" + URLEncoder.encode(appid, "UTF-8") + "&wssid=" + URLEncoder.encode(authManager.getWssid(), "UTF-8")); Map<String, List<String>> cookies = new HashMap<String, List<String>>(); cookies.put("Cookie", Arrays.asList(authManager.getCookie())); requestContext.put(MessageContext.HTTP_REQUEST_HEADERS, cookies);

The first property allows us to change the actual endpoint of the web service call to include the WSSID that we got from our authentication request. The second call allows us to set cookies on the HTTP request that is used to make the specific call. Together these will give us the access we need in order to use the stub securely. Actually making use of the API is quite easy now that we are authenticated. For instance, we can trivially discover whether or not the user is a Y! Mail Plus subscriber and has access to the full API functionality (non-premium users can't get the contents of messages for instance):

 UserData userData = stub.getUserData(); boolean isPremium = userData.getUserFeaturePref().isIsPremium()

Here is the code to pull all the unread messages from a folder:

 // List out up to 100 new messages in the folder  ListMessages lm = new ListMessages(); Flag flag = new Flag(); flag.setIsRead(FALSE); lm.setFilterBy(of.createListMessagesFilterBy(flag)); lm.setFid(folder.getFid()); lm.setNumInfo(BigInteger.valueOf(NUM_MESSAGES)); ListMessagesResponse lmResp = stub.listMessages(lm); return lmResp.getMessageInfo();

Notice how we use the ObjectFactory to create the filterBy element. Whenever you see a reference like JAXBElement<Flag> in the API you will likely want to use one of the convenience APIs within ObjectFactory to create the argument.

Now lets get to our RSS feed of unread messages example. To create our RSS feeds we could have just written out the feed directly but instead I'm going to use ROME as we might want to extend the example to a real application later. ROME has one dependency, JDOM-1.0 so we will have to get that as well. We can encapsulate the application into a single servlet that serves both the authentication feed and the mail feed. Here is the core servlet method:

 try { String token = httpServletRequest.getParameter("token"); if (token == null) { // If there is no token we are not authenticated  throw new AuthException("No token"); } // Current date, needed for many API calls  Date date = new Date(); // Instantiate the SOAP proxy  Ymws service = new Ymws(); YmwsPortType stub = service.getYmws(); // Instantiate the auth manager and set it up with the date, application ID,  // shared secret and the user token.  BrowserBasedAuthManager authManager = new BrowserBasedAuthManager(date, appid, secret, token); // Set up the web service call  setupWebServiceCall(authManager, stub); // Create the feed  SyndFeed sf = createFeed(date); // Go and get the list of folders and pull out the inbox  Fid inbox = getInbox(stub); if (inbox != null) { List<MessageInfo> messages = listUnreadMessagesInFolder(stub, inbox); List<SyndEntry> entries = new ArrayList<SyndEntry>(); for (MessageInfo message : messages) { SyndEntry se = createEntry(message); entries.add(se); } sf.setEntries(entries); } writeFeed(httpServletResponse, sf); } catch (AuthException e) { unauthorizedFeedResponse(httpServletResponse); }

This code should be fairly self-explantory and it builds on all the work that we have done so far. The only new things are the actual calls into the Y! Mail API that retrieve the messages from Y! Mail, converts them into an RSS 2.0 feed, and writes that feed to the wire. There are a couple of issues with this code that we don't address, like generating a permanent URL for the user to use. Right now whenever their authentication is reset (at least every 2 weeks) they will have to get a new feed URL from the application.

Throughout these simple examples there are opportunities for optimization. Many of the pieces of data can be cached for some amount of time and regenerated later like the token, the wssid and the cookie when the user fails to authenticate. The user hash can, of course, be used forever as a unique identifier for the user. Other opportunities for optimization include the ability to multiply dispatch requests to the API using batchExecute. Our example is an unoptimized version so that you can see how everything works before its made more complicated through the optimization process.

Here is a link to the full application, the source code is under WEB-INF/src. You will need to configure the web.xml with your own appid and secret but otherwise it should run in a standard JEE Servlet 2.4 container out of the box.