Idiomatic Python?

I’ve been working my way through compiling Java into Python code but the Python back end of my isn’t that good (my brain). I would call my stage of Python development the “magic incantation” stage. This is the stage where you really aren’t comfortable yet with the way things work in a new language but you can still get things done by miming other developers. I’ve also had some help from some friends on Twitter: @lhl, @precipice and @jkwatson.  My distributed information system is now getting some redundancy.  Little did they know that I was doing parallel invocations of identical requests for reliability and incrementally higher performance — and the results were verified using a quorum of responders.
Continue reading

Posted in Technology | Tagged , , , | 1 Comment

Tivo targeted advertising



Tivo targeted advertising, originally uploaded by Sam Pullara.

This looks like it might be both effective and also something that TV advertisers would like to buy.

Posted in Technology | Tagged , , , | Leave a comment

Using Google App Engine to Extend Yahoo! Pipes

Update: A commenter pointed out that you can

from django.utils import simplejson

instead of including it. Makes this even easier.

Yahoo! Pipes has always been a great tool for manipulating data but often you have to go to great contortions to get it to do what you want because of its very simple data flow programming model.  Google’s App Engine opens up the possibility of extending Yahoo! Pipes in very interesting ways through Pipes’ Web Service operator.  Currently this operator sees little use as it requires you to be running an external server somewhere on the internet that is always available for the Pipe execution which is quite a high barrier to entry for the typical Pipes developer. Here is what a Pipe that is using web service looks like and our example pipe:

Web Service Pipes Example 

With the launch of Google App Engine there is now a very simple way to get code up on the internet quickly in order to include arbitrary processing in the interior of your Pipes.

To demonstrate how this works, let’s first build a very simple web service that simply mirrors the data that it receives from Pipes.   If you don’t have a Google App Engine account you can still follow along by download the SDK and executing all the stuff locally though it will have to be accessible from the public internet if you want Pipes to send you requests.

First create a new application directory:

mkdir pipes-mirror
cd pipes-mirror 

Now create an application descriptor called app.yaml:

application: javarants
version: 1
runtime: python
api_version: 1
handlers:
- url: /.*
  script: pipes.py

This application descriptor basically tells Google how to deploy your application. Your application name should match an application name that you create within the GAE administration tool:

Application Name

Now we need to process the data coming from pipes. Pipes is going to pass this web service some data in JSON format and we need to parse it. GAE doesn’t include ‘simplejson‘ in the Python container so you are going to have to include it with your application. I downloaded simplejson-1.8.1 and symbolically linked its simplejson directory into my application directory. When the request comes in the JSON data will be in the ‘data‘ parameter so we are going to pull it out, parse it, grab the items array and write it back over the wire in pipes.py:

import simplejson
import wsgiref.handlers
from google.appengine.ext import webapp
class MirrorPipesWebService (webapp.RequestHandler):
	def post(self):
		data = self.request.get("data")
		obj = simplejson.loads(data)
		obj = obj["items"]
		self.response.content_type = "application/json"
		simplejson.dump(obj, self.response.out)
def main():
  application = webapp.WSGIApplication([('/mirror', MirrorPipesWebService)],
                                       debug=True)
  wsgiref.handlers.CGIHandler().run(application)
if __name__ == "__main__":
  main()

Now you should have a directory structure that looks a lot like this:

-rw-r--r--@ 1 sam  sam  106 Apr 13 18:55 app.yaml
-rw-r--r--  1 sam  sam  559 Apr 13 19:28 pipes.py
lrwxr-xr-x  1 sam  sam   47 Apr 13 17:40 simplejson -> /Users/sam/Software/simplejson-1.8.1/simplejson

Now that we have all the pieces we can deploy the application to GAE with a simple command from the GAE SDK:

appcfg.py update .

At this point you should be able to replace my web service URL that you find in my example Pipe with your application URL which will be

http://[application name].appspot.com/mirror

and get the same results as mine.

What kind of uses can you put this great power? I currently have a web service that I run that combines RSS entries from the same day into a single entry and have it deployed on my own server. I will likely port that to GAE as it doesn’t require a lot of CPU and it is a pain having to administer it. In fact, most of the functionality that you see in a service like FeedBurner would be easy to build on top of this framework. More exotic use cases can be found on Y! Pipes itself where at least one person uses web services to pass in photo URLs and return the coordinates of human faces in the images.

Posted in Technology | Tagged , , , , , , , , , | 34 Comments

JPA 2.0 with Criteria

(see: JSR 317 Persistently Improving)

I love the idea of adding a criteria API to JPA, the only thing I hope that they do differently than Hibernate is to implement that API in addition to string queries.  In Gauntlet we had issues where we wanted to use EJB-QL for selecting the right data and then a criteria-like API for applying security and filtering constraints on the query.  We ended up writing a criteria-like API that augmented the WHERE clause of the query to get the behavior that we needed (like described here).  For example, you could do this:

Query q = em.createQuery("SELECT p FROM Project p");
q.addExpression(Expression.notEqual("id", 2));

Or something like that. This would give you the best of both worlds, where you have the expressiveness of the textual query and the ability to further hone that query programmatically.

Posted in gauntlet, Java, Technology | Tagged , , , , , | 4 Comments

Macworld 2008 followup

Let’s see:

iPhone: failed to mention 1.1.4 update but the SDK is coming in Feb. The rest nope.  C

AppleTV/iTunes: Movie Rentals yes, missed buy on-iPhone support. AppleTV 2.0 — no computer required. Flickr + DVD + HD + Dolby 5.1, software upgrade! price drop! :) A

MacBook Air: “The World’s Thinnest Notebook” 0.16″ to 0.76″, 13.3″ screen, SSD option, 1.6/1.8 Core 2 Duo,  5 hours battery life, B

Blu-ray: Nothing. F

Monitors: Nothing. F

Other MacBooks: Nothing. F

Obviously the crazy prediction also didn’t make it, not even a wireless option for the Macbook Air…

Posted in Technology | Tagged | Leave a comment

Macworld 2008 predictions

UPDATE 1/13/08: This looks like confirmation of a new ultralight portable with some sort of wireless connectivity.

Might as well put up my predictions for what will be announced at Macworld this year. I think it will be a good one.

  • iMac, Mac Mini, Macbook and Macbook Pro lines upgraded to Penryn processors
    • Same prices, better specs
  • Ultra portable MacBook Pro ($2-3K)
    • 64G SSD drive standard (32G and 128G available), no DVD drive
    • 12″ OLED screen
    • 8-12 hour battery life
  • iPhone
    • SDK: Still WebKit-based but with native Javascript APIs for the phone
    • Better connectivity
    • Video recording
  • Monitors
    • Same panel as the new 30″ dell
    • Built-in iSights
    • Same prices
  • Blu-ray
    • Added as an option for MacPro and MacbookPro
    • Support for playing Blu-ray movies added to 10.5.2
  • AppleTV / iTunes
    • Movie rentals
    • Software upgrade for current devices
    • Possibly a new version with a blu-ray player

All these I consider relatively expected and if they don’t show up, some people will be disappointed. Now for the truly outrageous prediction:

APPLE WILL WIMAX ENABLE THEIR LAPTOPS AND IPHONE BY PLACING NODES IN THEIR APPLE STORES AND STARBUCKS.

That would be the most awesome thing ever and hopefully that is what is implied with “Something is in the air…”. The other possibility is just 3G enabled systems or both for those times when you are out of range :)

Posted in Apple, Technology | Tagged , , , , , , | Leave a comment

Using Groovy to make Maven2 more modular and readable

I started a very very small project this week to increase the usability of Maven 2. I find that although I love Maven as a back-end build processor, I kinda hate the XML configuration file and all the pain that goes with it. So rather than just simply complain, I will also do some token work to fix it. Inspired by Gant, this project attempts to use Groovy to make the creation of POM files easier, more readable, and modular. It will also allow best Maven POM practices to be put into libraries and reused rather than cut and pasted into a POM or painstakingly edited in a GUI. Further, for those who are more adventurous, you can add dynamism to your POM files.With graven (gvn), you can make use of the included standard library of Mavenisms and have this appear within your pom.groovy file:

 1     build {
 2         plugins {
 3             groovy()
 4         }
 5     }

For example, rather than add this to your POM file to make your Groovy code compile:

 1   <build>
 2     <plugins>
 3       <plugin>
 4         <artifactId>maven-antrun-plugin</artifactId>
 5         <executions>
 6           <execution>
 7             <id>compile</id>
 8             <phase>compile</phase>
 9             <configuration>
10               <tasks>
11                 <taskdef name='groovyc' classname='org.codehaus.groovy.ant.Groovyc'>
12                   <classpath refid='maven.compile.classpath' />
13                 </taskdef>
14                 <mkdir dir='${project.build.outputDirectory}' />
15                 <groovyc destdir='${project.build.outputDirectory}' srcdir='${basedir}/src/main/groovy' listfiles='true'>
16                   <classpath refid='maven.compile.classpath' />
17                 </groovyc>
18               </tasks>
19             </configuration>
20             <goals>
21               <goal>run</goal>
22             </goals>
23           </execution>
24           <execution>
25             <id>test-compile</id>
26             <phase>test-compile</phase>
27             <configuration>
28               <tasks>
29                 <taskdef name='groovyc' classname='org.codehaus.groovy.ant.Groovyc'>
30                   <classpath refid='maven.compile.classpath' />
31                 </taskdef>
32                 <mkdir dir='${project.build.testOutputDirectory}' />
33                 <groovyc destdir='${project.build.testOutputDirectory}' srcdir='${basedir}/src/test/groovy' listfiles='true'>
34                   <classpath refid='maven.compile.classpath' />
35                 </groovyc>
36               </tasks>
37             </configuration>
38             <goals>
39               <goal>run</goal>
40             </goals>
41           </execution>
42         </executions>
43       </plugin>
44     </plugins>
45   </build>

Voilà. You can check out the complete project here. I’d love help creating the most complete standard library if anyone is interested.

Posted in Technology | Tagged , , , , | Leave a comment

My Current Config

Since I get asked a lot for advice about what kinds of electronics to buy, I am starting a new page on my site that tracks “my current config“. I’m limiting it to a small set of categories that I think include the most important purchases that people make. Am I leaving anything out? Should I also have software on there?

Posted in Technology | Tagged , , | Leave a comment

Generate JPA (or GORM) classes from your database for Java and Grails

Whether you start with the database or start with code, no one wants to do the other one. Certainly by the DRY principle it is a waste of time and potentially a place where you could introduce bugs into your code. Today Dave and I are releasing a command-line utility that will handle the case where you start with the database and want to use JPA to access it. There are other utilities that do this that are generally built into IDEs, however this one fills the niche for those that want to do it in a more automated fashion and don’t want to edit the actual generated code. In fact, we used a previous version of this tool at Gauntlet throughout the development process to keep our database and JPA classes in sync with one another.

Introducing… dbmapper:

Usage: com.moonspider.dbmap.GenerateConfig
  -type (-t) [String] The type to generate, either 'jpa' or 'gorm' (experimental) (jpa)
  -destinationDirectory (-d) [String] Destination directory
  -url [String] The url of the database
  -pkg (-package) [String] The target package ()
  -user (-u) [String] Database user (sa)
  -password (-p) [String] Database password ()
  -globalExtends (-extends) [String] Class for all Java classes to extend
  -globalImplements (-implements) [String] Class for all Java classes to implement
  -driver [String] Database drive class
  -extension (-ext) [String] File extension for the generated code (java)
  -hibernate [String] Generate hibernate.cfg.xml to this directory
  -jaxb [flag] Enable xml binding generation

By default, the tool has a number of rules built into it that I will call ‘best practices’ at least as far as we are concerned:

  1. Tables should have a primary key column named id and it should auto increment.
  2. Foreign keys should be named ${foreignTableName}_id and be specified in the database.
  3. Many-to-many join tables should be named ${tableName1}_${tableName2}.

If for some reason you want to break one of these rules you will need to dig into the configuration of dbmapper. On the JPA side there are a number of conventions that it uses:

  1. Class and property names are converted from _ separated to CamelCase.
  2. Many-to-one and one-to-one relationships are marked as eager.
  3. One-to-many and many-to-many relationships are marked as lazy.
  4. Relationship collections are suffixed with List rather than making them plural.

Again, these are adjustable through a configuration file that I suggest that you never use unless absolutely necessary. The most typical use case for the configuration file is when a database identifier conflicts with an identifier used in Java or Grails. In this case you can either configure dbmapper or change your database. DBMapper is designed to work well with the dbmigrate utility and in fact I recommend you use them together for the most leverage. It is great to be able to add a new migration, execute the migration, generate new classes and see exactly the effect that your database change has on the domain model.

The default functionality is also available directly from Grails. On the command-line simply type:

grails install-plugin dbmapper

This will contact the plugin repository and install the latest dbmapper plugin directly into your Grails application. Assuming that you have configured your database in the normal fashion for Grails you can then generate domain classes directly from that connection with a single command:

grails generate-domain-classes

Assuming that is successful you will find all of your domain classes in the grails-app/domain directory and also a hibernate configuration file in your grails-app/conf/hibernate directory. You should then be able to use tasks likes generate-all to create views and controllers for each of your domain classes.

There is also experimental support for creating GORM classes instead of JPA classes, however this really is experimental at this time and I suggest that you only try it if you are interested in fixing it. GORM doesn’t seem to have the flexibility dbmapper needs quite yet.

Another feature of dbmapper is the ability to create JAXB annotations on the JPA classes so the objects can be read from and written to XML in addition to the database. The dbmapper framework itself is designed to be reasonably extensible. By adding another template you can extend it to generate mappings to other persistence frameworks.

Posted in Java, Technology | Tagged , , , , , , , , , , | 8 Comments

Parallels vs VMWare vs Bootcamp vs Codeweavers Crossover Office revisited

I have reviewed these products in the past but I think they have rev’d enough times to make it worth my while to revisit them and see how far they have come. As a reminder, lets again review a couple of use cases that you might have:

  1. Native Microsoft Office (any version) support
  2. Latest Microsoft Office
  3. Microsoft Vista
  4. Games
  5. IE7

Again with the ideal requirements:

  1. Runs applications at full speed
  2. Executes them in such a way as they appear to be native applications
  3. Does not require a Window license

The simplest answer is to use Crossover Office to run applications that work with it, as when it works it satisfies the ideal requirements and then use Bootcamp when you absolutely have to have a true Windows machine. Crossover Office is now robust enough to run Outlook 2003 pretty well along with some older games like Counter Strike: Source though it doesn’t have support for the very latest software. There are cases that are in-between, for example you might not need full speed but you do need full compatibility, then either Parallels or VMWare Fusion ought to work for you, like when you want to run IE7 to view a website. If you are doing anything that requires high non-graphics speed and also high-compatibility like developing software, either VMWare or Bootcamp would be your best bet. VMWare allows you to use more than 1 processor and it out benchmarks Parallels. On the other hand, Parallels is a nicer application and its Coherence-mode (integrates Windows applications into the Mac desktop experience) is better than VMWare’s Unity mode. Neither VMWare nor Parallels run highly graphical twitch games to my satisfaction. Here is a little table to help you make your decision:

  • Crossover Office 6.2
    • When it works its the best option
    • Fast, well integrated, runs some games
    • Only option that doesn’t require a Windows license
    • According to CodeWeavers, will work even better under Leopard
  • Bootcamp
    • It is just Windows
    • Mac’s are really good Windows machines
  • Parallels Desktop 3.o (build 5160)
    • Easier to use than VMWare Fusion
    • Coherence is very mature
    • Does not really run games well
  • VMWare Fusion 1.0 (build 51348)
    • Faster than Parallels
    • Unity is OK
    • Lot’s of compatible VMWare appliances
    • Not much luck running games on it

Have any questions about the options? Since I have a license to all of them and use them all for various purposes I would be happy to answer any specific questions about the products.

Posted in Technology | Tagged , , , , , , , , , , | 31 Comments