java rants » pipes

Idiomatic Python?

Sam Pullara — Wed, 23 Apr 2008 18:19:51 +0000

I’ve been working my way through compiling Java into Python code but the Python back end of my isn’t that good (my brain). I would call my stage of Python development the “magic incantation” stage. This is the stage where you really aren’t comfortable yet with the way things work in a new language but you can still get things done by miming other developers. I’ve also had some help from some friends on Twitter: @lhl, @precipice and @jkwatson. My distributed information system is now getting some redundancy. Little did they know that I was doing parallel invocations of identical requests for reliability and incrementally higher performance — and the results were verified using a quorum of responders.

Here is my first service that I am porting. It takes an RSS feed (in JSON format from Pipes) and combines all the entries from each day into a single entry:

import logging
import wsgiref.handlers
from datetime import date
from google.appengine.ext import webapp
from django.utils import simplejson
class DayBinPipesWebService (webapp.RequestHandler):
 def post(self):
 	now = date.today()
 	now = now.strftime("%m/%d/%Y")
 	data = self.request.get("data")
 	items = simplejson.loads(data)["items"]
 	bins = {}
 	for item in items:
 		published = item["y:published"]
 		updateDay = "%(month)02d/%(day)02d/%(year)04d" % published
 		if now != updateDay:
 			bin = bins.get(updateDay, [])
 			bin.append(item)
 			bins[updateDay] = bin
 	entries = []
 	for bin in bins.items():
 		dayDate = bin[0]
 		binEntries = bin[1]
 		first = binEntries[0].copy()
 		first["description"] = ""
 		for e in binEntries:
 			first["description"] += "%(title)s
%(description)s" % e
 		first["title"] = "Items from " + dayDate
 		first["link"] = ""
 		entries.append(first)
 	self.response.content_type = "application/json"
 	simplejson.dump(entries, self.response.out)

How would you write this in idiomatic Python as opposed to my rudimentary translation? Would you change the whole design?

Using Google App Engine to Extend Yahoo! Pipes

Sam Pullara — Mon, 14 Apr 2008 02:59:45 +0000

Update: A commenter pointed out that you can

from django.utils import simplejson

instead of including it. Makes this even easier.

Yahoo! Pipes has always been a great tool for manipulating data but often you have to go to great contortions to get it to do what you want because of its very simple data flow programming model. Google’s App Engine opens up the possibility of extending Yahoo! Pipes in very interesting ways through Pipes’ Web Service operator. Currently this operator sees little use as it requires you to be running an external server somewhere on the internet that is always available for the Pipe execution which is quite a high barrier to entry for the typical Pipes developer. Here is what a Pipe that is using web service looks like and our example pipe:

With the launch of Google App Engine there is now a very simple way to get code up on the internet quickly in order to include arbitrary processing in the interior of your Pipes.

To demonstrate how this works, let’s first build a very simple web service that simply mirrors the data that it receives from Pipes. If you don’t have a Google App Engine account you can still follow along by download the SDK and executing all the stuff locally though it will have to be accessible from the public internet if you want Pipes to send you requests.

First create a new application directory:

mkdir pipes-mirror
cd pipes-mirror

Now create an application descriptor called app.yaml:

application: javarants
version: 1
runtime: python
api_version: 1
handlers:
- url: /.*
  script: pipes.py

This application descriptor basically tells Google how to deploy your application. Your application name should match an application name that you create within the GAE administration tool:

Now we need to process the data coming from pipes. Pipes is going to pass this web service some data in JSON format and we need to parse it. GAE doesn’t include ‘simplejson‘ in the Python container so you are going to have to include it with your application. I downloaded simplejson-1.8.1 and symbolically linked its simplejson directory into my application directory. When the request comes in the JSON data will be in the ‘data‘ parameter so we are going to pull it out, parse it, grab the items array and write it back over the wire in pipes.py:

import simplejson
import wsgiref.handlers
from google.appengine.ext import webapp
class MirrorPipesWebService (webapp.RequestHandler):
	def post(self):
		data = self.request.get("data")
		obj = simplejson.loads(data)
		obj = obj["items"]
		self.response.content_type = "application/json"
		simplejson.dump(obj, self.response.out)
def main():
  application = webapp.WSGIApplication([('/mirror', MirrorPipesWebService)],
                                       debug=True)
  wsgiref.handlers.CGIHandler().run(application)
if __name__ == "__main__":
  main()

Now you should have a directory structure that looks a lot like this:

-rw-r--r--@ 1 sam  sam  106 Apr 13 18:55 app.yaml
-rw-r--r--  1 sam  sam  559 Apr 13 19:28 pipes.py
lrwxr-xr-x  1 sam  sam   47 Apr 13 17:40 simplejson -> /Users/sam/Software/simplejson-1.8.1/simplejson

Now that we have all the pieces we can deploy the application to GAE with a simple command from the GAE SDK:

appcfg.py update .

At this point you should be able to replace my web service URL that you find in my example Pipe with your application URL which will be

http://[application name].appspot.com/mirror

and get the same results as mine.

What kind of uses can you put this great power? I currently have a web service that I run that combines RSS entries from the same day into a single entry and have it deployed on my own server. I will likely port that to GAE as it doesn’t require a lot of CPU and it is a pain having to administer it. In fact, most of the functionality that you see in a service like FeedBurner would be easy to build on top of this framework. More exotic use cases can be found on Y! Pipes itself where at least one person uses web services to pass in photo URLs and return the coordinates of human faces in the images.

Yahoo! Pipes just launched

Sam Pullara — Fri, 09 Feb 2007 16:16:11 +0000

A really cool application, Pipes, from Yahoo! launched the other day and has the potential to really make it easy to mashup web applications and repurpose data.

It is loosely based on the Unix pipes model of using small programs chained together to process data where the output of one program is the input to another program. With the Pipes application you use RSS (or ATOM) feeds as your source data rather than text files and then use the available transforms and other data source to mashup a new feed that can be displayed using a number of different renderers including a map output. Here is an example of how you can build a pipe with variables and multiple stages:

The whole project reminds me a little bit of my XQuery project I did so long ago though much better executed and more approachable. Though as soon as they open up the ability to create more transformation types I will probably drop XQuery in there to do the dirty work.

The ServerSide Symposium

Sam Pullara — Thu, 03 Mar 2005 09:42:09 +0000

I arrived in Vegas this morning on the 6:30am flight from San Francisco. There are very few things that cause me to get up that early.

The day started with Floyd again telling everyone how great they were and how they were really smart and how cool TSSS is. I really think they should do this with facts rather than statements. The TechTarget people that acquired TSSS have a pretty cool instant polling system that I think would be great to have access to as a speaker. I hope the speakers knew about it ahead of time and are going to poll their audiences. Here are the results of the poll:

Should java be open sourced:
59% yes

This is an inexact question because it is not qualified with what form this would take but you can see it is somewhat of a community divider. Myself, I voted no, mostly because I don’t think that without further explanation it is a valid question. For instance, lets say it was released with GPL and all Java code that used it needed to be GPL’d. I don’t think you would get a 59% yes response worded that way. For my needs, Java is sufficiently open sourced. I can download it and file bugs and patches against the source that they expose.

Star Wars versus Star Trek:
Star Wars about 65% — he moved quickly.

I believe that you have to compare the best vs the best. In that regard I compare Empire Strikes Back to Star Trek 2: The Wrath of Khan. For me, in this case, Star Wars still wins by a nose,

What is the most important thing when choosing a web application framework?
application maintainability 34%
ease of development 27%
existing architecture 19%
leverage skills 10%
code reuse 11%

This is clearly the result of asking this question to advanced java developers. People choose Java because you can go back to code that you or someone else wrote and figure out what it was doing and how to change it. There are quite a few technologies that this doesn’t apply to that people are trying to push.. *cough* PHP *cough*. These weightings generally reflect my biases.

What Java IDE do you use?
Eclipse 53%
Idea 20%
jbuilder and wsad 7% each

No surprise here except that I think IDEA has gained on Eclipse from this same written poll last year. Remember that WSAD is also Eclipse which puts it at 60%. There were a bunch of other ones on the slide but they were all noise.

What web framework would you use?
struts 47%
spring 21%
tapestry 8
webwork 2
other 15

I voted other on this one because I really think that JSP 2.0 is going to be sufficient for my needs. With the new automatic prelude/codas, tag files, and scriptfree pages I can build and extremely clean system without using any of these containers. I was a bit surprised that JSF wasn’t listed, however I’m not going to use it because of its incompatibility with JSP EL, etc.

After the introduction to the conference Floyd introduced Mark Hapner, a lifer at the JCP who has had his hands in virtually every interesting (to me) JSR. His talk was mostly about how the community now owns most of the infrastructure it needs to build services on the internet and that we should never again accept a proprietary wire protocol. He then drifted into something about mixing SE and EE technologies that didn’t make much sense. I’ve never had any problem mixing things that were designed for SE into WebLogic. Blah blah blah, collaboration, potential, all kinds of vague innuendo about lightweight containers without talking directly about it. Somehow he went from this to message and exchange patterns within an application and between them with some hand waving about unix pipe composition. The only problem is that unix pipes are incredibly ad hoc and have no meta data associated with them while the XML messages that he is talking about compositing are typically in well known self-describing formats. Perhaps unix processes should be able to output a schema for their output with a command line option so you could grab known fields instead of randomly grepping through stuff like you do today. I think Microsoft has some sort of shell for .NET that does things like that, though it goes a little too far to the object side. He also Seems to have a lot of Random Capital letters for Things in his Talk. Somehow I think that is supposed to elevate them to “ideas” but really they are just words and without specifics things like Service Component is not that interesting.

Finally he gets to the point, “here are the specs I’m working on, use them:”

J2SE as the basis
JMX for management
JBI for collaboration
J2EE for its Service (capitalization his) technologies
Build it and other Service technologies will come

I’m not so sure on that last point. From what I can tell only .NET and Java really have the support for advanced WS-I technologies (this is what he is really talking about I assume). Not that it is a bad thing, but I don’t think you are going to see much Ruby, Python, and PHP playing in this service platform or even talking to it without a lot of work. That is of course unless you are using an implementation that runs on the JVM and just leverage all the Java libraries for dealing with this stuff.

Now that he’s made his point he goes back into vague arguments about making the Java Platform the web build-out technology. Why he’s making this argument to a bunch of Java developers who are already using the Java platform to do all this stuff, I’m not sure. We all either believe in this vision already or are spies for Microsoft or the collection of duct tape technologies (the Ps).