The first choice you have to make against these requirements is which data definition language are you going to use? Instead of depending on the native format of the storage system I’ve decided to use Avro. Similar to Protocol Buffers and Thrift, Avro lets you define your entities using schemas and store them efficiently in a binary format. Additionally, as long as you have the original schema available you can load old data into a new schema. This will let you evolve your stored rows lazily and not have to completely update your storage system when a new field is added or a field is removed.

Whereas the data definition choice is basically commodity at this point and your choice can be somewhat arbitrary, the choice of storage technology will likely be something that has more trade-offs to consider. After looking at the features and communities of a variety of projects including HBase, Cassandra, Redis, Riak, MySQL, Membase, Terracotta, etc. I finally chose HBase for a few reasons that may not be that important to you. First I settled on a BigTable type choice based on the data model. That left HBase and Cassandra as contenders. There also a few things I think are advantages:

• multiple tables
• better ordered key support
• Zookeeper and HDFS were already in my solution
• Hive and Hadoop support against the native data format
• consistency
• compare and set
• atomic increment
• versioning

Cassandra definitely has its own advantages that didn’t out-weigh the other considerations including

• performance
• simple deployment
• always writable

You might make different tradeoffs or even use both solutions for different problems. I’m also biased somewhat as I am sharing an office with Cloudera and get top notch support at a moments notice.

Whether I chose HBase or Cassandra the implementation would have been much the same and with the HAvroBase framework you can have multiple independent storage systems. The framework does somewhat shield you from this decision, at least in your code.

When it comes to text search you really don’t get better than Lucene in open source and the features that Solr builds on top of Lucene make it even better. I don’t think there is reasonable argument for using something besides Solr at this point. Especially with their support for sharding and replication that comes with Solr Cloud. It also has the nice benefit that it supports multiple tables, like HBase, so I can efficiently separate entities and allow them to scale independently without managing an additional system.

The last choice that I made was runtime configuration. There are a few solutions including Spring, Guice and rolling my own and I finally landed on the side of Guice. I just like typed, programmatic configuration better than XML files and since that is the focus of Guice, I think it is the best solution for that.

The first thing that you need to do to use the system is to define an entity in Avro:

You can use the Avro compiler (or the Maven plugin) to convert that definition to classes (HAvroBase requires this). Then you do the typical things in code to setup your HBase connection but within a Guice module and set some of the configuration parameters for HAB:

That configuration can then be used to instantiate an instance of the HAB class that implements the connection to HBase like this:

You’ll notice that configuration is somewhat split between things that the base system needs to know and things that the HAB needs to know. For example, what table the schema is store in isn’t relevant to the base system as an implementer is free to store schemas however they like. But the assumption is that every system will have the concept of a table and a column family in order to separate entities in the underlying storage system. The Solr support is pretty elegant I think. Rather than specifying in more than one place which fields should be indexed, instead at runtime it queries the Solr core’s schema.xml that defines them and then when you put an entity in the system it automatically indexes those fields. Here is an example of using some of the APIs:

This example creates a new User, puts it into the HAvroBase, looks it up by row and then searches for it by field that is indexed and finally deletes it. In any fully implemented AvroBase system this should work identically. You’ll notice in the memcached implementation that only put/get/delete are implemented and it doesn’t support scan or search at all. I’ve considered breaking out that functionality into separate interfaces but haven’t done that yet.

If you are unhappy with the way the default Avro compiler generates code you can use my templated Avro compiler that lets you change the generated code. I have done that for my project to make them a little more developer friendly.

Things left to do:

• implement guarantees around indexing
• optimize schema keys (use an abbrev table instead of a hex sha-256 key)
• indexing nested fields and arrays in Solr

I’m sure there are more things left undone but those are some of the more obvious issues.

• Works well with HTML5/CSS3 progressive enhancement

This one is straight-forward.  Because I put no limitations on the generated text you can actually generate almost anything with mustach.java, in fact, I am using it in another project to generate Java source for Avro objects.  It is really overkill for that project but it is so easy to use that I used it anyway.

• Allows mock data within the template that is replaced at runtime
• Client-side version that leverages the mock data for shift-reload debugging

These I implemented somewhat differently from the requirement because the mustache template language is not tag based.  I have actually thought about whether it might make sense to make it tag-based and get the full requirement, but I digress.  What I did get to is a system that is easy for a frontend developer or designer to use without having the whole system running on their machine.  Included with mustache.java is a mini-server (called handlebar, thanks for the name Luke) that combines mock json data with mustache.java templates. This lets them view a fully rendered version of the page they are working on with the required templating information that will be used in production but without running a big production system. In fact, because mustache.java dynamically generates and compiles Java code, they can fire up handlebar in directory along with their mock data and just keep reloading the page to see the effect of changes they make.

• Composable components, not monolithic pages
• Very little or no business logic in the templates

The choice of mustache as a base template language ensured that this would be the case.  Including a partial within a template is easy and makes pages very easy to compose. I still miss the Django template style extends/block system and may extend mustache to support it. Having to remember to include things like the header and footer on every page doesn’t seem necessary.  As I said above, I think that mustache has the minimum reasonable logic.

• Concurrent evaluation possible

This one was very important to me from a performance perspective. My implementation makes every effort to be as low latency as possible for the end user.  At every possible point in template evaluation things are actually executed in separate threads or purely asynchronously in the case of external I/O like HTTP requests to other services.  The one last thing I am going to do here is to allow not only a straight-line evaluation of the template but the possibility of specifying a timeout so that you can return different markup if evaluation doesn’t return in time. It may be unnecessary to have this in the base template engine though, still looking at the design of the feature.

So how easy is using the template system?  I think I can answer that question by looking at the relevant portions of the implementation of handlebar:

final MustacheCompiler mc = new MustacheCompiler(new File("."));
....
Mustache mustache = mc.compile(new BufferedReader(new FileReader(filename)));
FutureWriter fw = new FutureWriter(res.getWriter());
File file = new File(mocks, base + ".json");
if (file.exists()) {
  BufferedInputStream bis = new BufferedInputStream(new FileInputStream(file));
  JsonParser parser = jf.createJsonParser(bis);
  JsonNode json = parser.readValueAsTree();
  mustache.execute(fw, new Scope(json));
} else {
  mustache.execute(fw, new Scope());
}
fw.flush();

The most interesting part of this is the FutureWriter. The future writer lets mustache write evaluated templates completed out of order in template order. The scope is essentially a sophisticated context object that knows how to pull data from Java objects of various types, including raw object fields and methods, Maps and Json objects. Conveniently, you can also store values in it directly. There are some things I would like to extend at some point, however, for all practical purposes the template system is finished.

This wouldn’t be that big a deal if Android software wasn’t already approaching the speed of the Apple iPhone even when running its applications through the interpreter (see the HTC EVO 4G running 2.1).  This is likely going to mean that 3rd party developer applications created for Android, running on the same hardware, is going to be faster than the same code written against the Objective-C libraries that Apple provides for the iPhone.  You’ll be able to get more done, have smoother user interfaces and all around build more powerful applications easier.  A better experience for the user and the developer on Android is a bad thing for Apple.

The final issue that Apple has is that far more people know and need  to know Java than Objective-C.  Except for their platform, there is no need to ever learn it.  The tool chain is much more robust, information is much easier to come by, garbage collection is available, and there are thousands more libraries written in Java than Objective-C and far more portable libraries than are written for C/C++.  You might argue that you can always drop down to C/C++ code to make up for the lost performance of using Objective-C (at a significant cost development-wise).  You can do that, except places where you need to make a call into Apple’s Objective-C runtime libraries or where you want to write callbacks from those libraries. Those are mostly Objective-C libraries and use the dynamic dispatch mechanism which I showed in 2004 was very much slower than Java.  I’d really love to see the whole computer language shootout written with Objective-C calling conventions just to show how much slower it is at that level.

Obviously performance is not the only consideration, but it is a big one.  Android has other issues like a fragmented operating system base, hardware feature base and a more complicated user experience.  All those things will conspire to hold it back but I think we can see the writing on the wall that Android is going to dominate iPhone market-share wise which will eventually make it a more attractive platform business-wise.

Update: Just so people don’t get the wrong idea, I am a rabid iPhone user and develop for the iPhone first.  This blog entry is to call out a threat that Apple should take seriously.

Update 2: Alright, you win. Don’t worry about competing with Google & Android on performance. It probably doesn’t matter that much for the user experience.  Especially don’t fix obj_msgSend() with a JIT/LLVM, that would be crazy.

The html page that he uses didn’t have to be changed that much, you can find the same demo that he uses here.  This was really quick so right now there is no way to leave out one without it freaking out.  This is more a tech demo than it is a real solution.

Come to think of it, maybe the whole thing is a little suspect. Maybe we should have been using URLs the entire time. How much value are we getting out of the layer of indirection? This is intended to be a discussion…

http://graph.facebook.com/2410418

If I could easily go to my account and specify a new location for that data, for example, maybe I would put it at:

http://graph.javarants.com/me

Which, for the sake of argument, resolves to much the same information about me, the Sam Pullara node in the graph. Similarly you could ask for metadata about what other things are available. Since it is a REST API you could mix and match those links to point to different endpoints. Some of them might point to a new Flickr Open Graph API endpoint that responds with my photos that are available to the caller. Another might point back at Facebook for my social graph. While a 3rd could point at a web service (say a new output format for Y! Pipes) that combines my posts on Facebook with my RSS feed and my Twitter feed in order to populate the ‘posts’ endpoint. Using these endpoints Facebook could construct the same set of views they have today on the graph but without being the ultimate authority for the data itself. Plus, as they would have to call your APIs with the authentication of the viewing part, privacy is under your complete control as well. For performance reasons, you could still have caching and asynchronous notification systems. Perhaps with some special sauce for websocket you could even allow them to reference the client browser rather than another server for ultimate end user control.

It would be theoretically possible to implement this now without Facebook’s involvement but without them as one of the providers I don’t think you could really get the adoption that would make this truly revolutionary.

• Works well with HTML5/CSS3 progressive enhancement
• Allows mock data within the template that is replaced at runtime
• Client-side version that leverages the mock data for shift-reload debugging
• Composable components, not monolithic pages
• Very little or no business logic in the templates
• Concurrent evaluation possible

Right now I am looking at mustache.js and its various server-side implementations as a possible solution to this. It has many of those qualities but I would likely need to make some of my own modifications for things like concurrent evaluation.

Any suggestions?

Another project that was done in someone’s free time is Zach Grave’s TableSaw that helps you generate tables from URLs. I am going to combine these two things in this post to show you how to make a new table and an environment to more easily use it. Ultimately I could see TableSaw and the YQL Storage Editor combined, to that end, I’ve released the source for this blog post on github.

I’m going to use a real example that I just recently walked through, creating an open data table for the bit.ly API. The table will be pretty simple, just allowing you to pass in a bit.ly link and it will return the long URL for that link. First thing to do is to go to the bit.ly API documentation for the expand REST API. That gives us a nice example URL that we can use in Table Saw to generate a table:

http://api.bit.ly/expand?version=2.0.1&shortUrl=http://bit.ly/31IqMl&login=bitlyapidemo&apiKey=R_0da49e0a9118ff35f52f629d2d71bf07

If you head over to Table Saw and put that in the text field and load URL you get:

Initial generated table for api.bit.ly/expand

To complete the form and update, fill in an author name and the Produces should be JSON. One current bug with Table Saw is that it doesn’t seem to properly remember the Produces field so we will have to edit that after we grab the definition. Highlight everything under “Your new YQL Table!” and copy it into the clipboard and then go to the YQL Storage Editor. On that page you will see a set of execute/select/update text fields for the various urls that you need to use a yql.storage entry. Everything is empty right now and we want to create a new one, so click the New button. That should fill in those URL fields with brand new store:// URLs. Put focus on the Bespin editor component and paste in the table we copied. If you then click save it should say that it updated the entry in YQL with this new information. You’ll notice that there is a text area with the URLs listed out, if you want to keep track of yql.storage entries you need to keep these special identifiers around so you can perform the various operations on the content. I generally have a list that I keep locally of the entries that I am using. If you paste the list into the file as a comment you can then also just remember the ‘select’ URL in order to come back later and find the other two. Lastly, when the execute key is changed, the two links on the right get updated. The first link goes to the YQL console with your execute key used as an environment and the second link goes to the YQL console with your execute key used as a table definition. These two links can help you quickly try out changes that you make to your environment or table without a lot of cutting and pasting.

Table Saw isn’t yet perfect, so I make a few edits: change the produces from XML to JSON, add a description, reformat the sample query, set the default version to 2.0.1 and make all the other keys required, this should leave you with a page that looks like:

Edited version of the bit.ly table

When you click save it should alert you that it was saved successfully by saying that it was updated. You can now test this table in the YQL console with the following query:

use 'store://8tLqFvgCm8kdOAp60lPz24' as bit.ly; select * from bit.ly where version='2.0.1' and shortUrl='http://bit.ly/31IqMl' and login='bitlyapidemo' and apiKey='R_0da49e0a9118ff35f52f629d2d71bf07'

I noticed at this point that I forgot the itemPath and then set it to “json.results” in order to cut out the error messages and some other things that I wasn’t interested in. Nagesh Susarla had a great blog post on using YQL storage for secrets. We’ll also do that by using the YQL storage editor to create a new environment for us to use that could store our secret values that bit.ly needs in order to run the API. Create another YQL Storage Editor tab, click New and then we can make an environment that uses the table, sets the login and sets the API key:

Secret bit.ly environment with their API keys

Now using that execute URL as your env value in your query URLs you can have very simple queries against this API. For example, if you use this URL for the YQL Console you can use this much simpler query that gets all its authentication implicitly:

select * from bit.ly where shortUrl='http://bit.ly/31IqMl'

This allows you to not embed those valuable keys directly in your client software or even in your source control system but hidden behind what amounts to a very secure random password.

Sam Pullara

Long-time computer programmer

Peter Seibel: How did you learn to program?

Sam Pullara: Back when I was around 10 I asked for a computer. My mother looked at the available choices and decided that the Atari 400 was just a game machine and the VIC-20 looked like I might actually learn something. She was right. I started out programming in BASIC and then later 6502 machine code on the 64 and 128. In high school I finally learned a ‘real’ computer programming language, C, by reading a 700 page volume book in a single night. One day I will take a computer class.

Peter Seibel: What is the thing that you worked on that you are most proud of?

Sam Pullara: The work that I did helping to create the WebLogic Server. Building a system that thousands of developers and in turn millions of users depended on was a great experience and very challenging. Ultimately, that work was acquired not by one company but eventually by two and is still one of the best enterprise software servers on the market in the very capable hands of Oracle.

Peter Seibel: What about books? Are there particular computer science or programming books that everyone should read?

Sam Pullara: I haven’t read a great deal of computer science books though I enjoy books about algorithms, languages and patterns rather than books about the details of programming an API or in a certain domain.

Peter Seibel: Do you still read code for fun, as opposed to reading it because you need to work with it?

Sam Pullara: I love exploring a new piece of code. Digging into something deeply and seeing the nuances and edge cases the developer discovered as they created the program is very interesting to me. I’ve had to hand my code off many times in my career and I am always pleased when someone takes the care to create code that is easily understood rather than clever.

Peter Seibel: What about organization, do you organize your code top down or bottom up? Do you refactor to keep the internal structure of the code coherent? Or do you have just very good sense at the beginning?

Sam Pullara: I try to write code that always works and refactor it into the final design. Java and its IDEs have trained me to write code backwards in many cases with the structure flowing out of an initially very procedural constructs. The invention of isomorphic transformations on code and semantic code completion have completely slaved me to languages that conform to the restrictions that make those very things possible.

Peter Seibel: How do you go about debugging?

Sam Pullara: I use a mixture of Zen, printlns and actual debuggers depending on the circumstances. For very esoteric edge conditions I prefer to figure out what must be true and work backwards from there. When code is only badly behaved rarely, printlns help you discover where you should look or think about. For complicated algorithms, I do enjoy the occasional debugging session where you interrogate every line to see if it is behaving as you expect.

Peter Seibel: How do you tackle understanding a piece of code that you didn’t write?

Sam Pullara: Figure out an interesting feature to add to it and add it. Nothing lets you really understand a piece of code like trying to change it. The tools of the trade are typically a refactoring IDE with all the modern bits of introspection that you can bring to bear. I’ve found that even obfuscated code is east to understand with modern tooling.

Peter Seibel: What about the length and intensity? I am sure you’ve done 80-, 100-, 120-hour weeks. Is that necessary? Under what circumstances it is really necessary and when it is just macho thing to do?

Sam Pullara: I have a pattern where I think and think and think, tinker, tinker, think and then finally in a like one huge breath blow the program out of my mind and into the computer. That final breath can take hours and hours straight but it is so much better than trying to force it before it is time.

Here is the form for your own answers to these questions: