The short version: I want to evaluate node.js and Twisted frameworks by building the same tool with both. The scope of the two frameworks is nearly identical. This presents the somewhat unique opportunity to create a single client user interface that interacts with both versions of the tools, with the same protocol. It should truly be an ‘apples-to-apples’ showdown.

Relevant links:

• Code: https://github.com/jjmojojjmojo/Node.js-Vs-Twisted
• Docs: http://jjmojojjmojo.github.com/Node.js-Vs-Twisted

This is the first post in a series pitting node.js squarely against Twisted, exploring what event-driven server programming can do, pushing the limits of what can be done entirely on the client side for UI (jQuery+HTML is the current plan), shedding some light on how I evaluate tools, help other folks investigating these technologies get some perspective, distribute working examples, and producing a useful application.

Wait, You’re Doing What Now?

I’m going to implement three disparate code bases: server in node, server in twisted, and the client UI. I want to go from little understanding of the server technologies involved, to two working server applications adhering to the same API.

So, why exactly?

A few things happened sort of all at once this past week:

• there’s been more talk of node.js at work.
• somebody tweeted about node.js getting more attention than Twisted.
• my team at work is changing scrum masters; I sat in on the initial ‘brain dump’ meeting.
• I recalled my past declaration against frameworks, and saw an opportunity to work at a lower level.

So it dawned on me: node.js and Twisted are similar enough to write the same server app in both. I then thought it might be fun to write an HTML-based client UI, flex my jQuery and AJAX muscles.

I had a few thoughts about alternative data stores (nosql, flat filesystem storage of JSON files, xml).

All I needed was something to implement. A problem to solve. That’s where the ‘brain dump’ meeting came into play. We have a shared spreadsheet on google docs we use for our sprint retrospective meetings. We all write up our thoughts in the same document, and use font colors to hide them from our teammates.

It works pretty well, but it’s not ideal. Listening to folks talk about it made the rough edges stick out. So I selected ‘streamlining the retrospective’ as the primary goal of my application. There are a few other meetings and procedures that could use a little web help as well. And it would be slightly more complex than a typical ‘Web 2.0′ application, since it would also involved the idea of shared state and real-time collaboration.

I know python really well, and I know javascript really well. Event-driven programming? Not so much. Raw client-server apps? Little more, but I’m no expert. This problem and these technologies presented an opportunity to approach every aspect with minimal bias.

It became obvious to me: this  is a ‘perfect storm’ of technologies, concepts, and utility. It all sort of fell together when I came to that conclusion.

And for the first time since San Francisco last November, I got really, really motivated to write some code on my own time. I’m hoping to keep the momentum going.

This series chronicles my journey. From requirements to application design and through to packaging and deployment. From no experience with node.js or Twisted to a non-trivial, working app and a good understanding of what sets the frameworks apart.

Ground Rules

Here are the general parameters that will drive the design choices I make, and how I define ‘complete’:

• the app will have good requirements, design and implementation documents.
• the app will strive for 100% test coverage.
• the app will be distributable via python eggs or npm packages.
• the app will have it’s own start-up script, installed by the package manager.
• the port and IP to listen on will be configurable at run time.
• the user interface will be served by the framework, but allow a static web server to do it instead.
• the protocol will be HTTP-based (to facilitate the HTML front-end most efficiently).
• the data interchange will be JSON.
• there must be a well-defined procedure to deploy the app for either framework.
• the servers will contain as little client-side code as possible.

Status

I’ve got the major user stories captured and some other documents established. I have a buildout that builds and configures:

• Sphinx to buld the docs
• Apache httpd (to serve the docs, optional, uses SimpleHTTPServer by default)
• Twisted
• node.js
• supervisor (used to control all of the servers)

I also have ‘hello world’ HTTP services built in each, and installable as a package.

The node.js version installs an executable in the bin directory of the buildout. Currently the Twisted-based server is set up as a plug-in for the twistd command (that was actually more complicated than setting up a console script).

Notes

There were some oddities and rough patches in putting together the buildout.

Node.js

Node compiles without any exotic dependencies, but was difficult to customize the installation: the configure flags don’t support anything but –prefix. I started installing it with –prefix=[root of the buildout], so the binaries would be in ./bin alongside ‘buildout’.

Npm comes with node these days, but it doesn’t install properly in a sandboxed environment. It assumes that you are installing to the operating system. The ‘npm’ command is actually a symlink to an executable javascript file that assumes you have node installed centrally (the first line: #!/usr/bin/env node).

I initially tried to hack around the problem by using a combination of z3c.recipe.template and cp.recipe.cmd to create a wrapper script and apply the right permissions, but that didn’t work with the compilation and template fighting over the same filename.

I contemplated using a different name for the script, but I want the end product of this project to be something that anyone familiar with node could pick up quickly. It’s best if we keep the workflow familiar, and keep the executables named as expected.

By chance I stumbled upon nodeenv. It solves the chief problems, and is getting me by for the time being, but it needs some work. For one, it assumes you have cURL installed. On my ubuntu dev box, I did not. I opted to download and compile it as part of the build, so no one using the buildout should run into that issue, but I consider that a pretty big problem, especially given how robust python’s built-in HTTP libraries are.

You can provide a list of npm modules you want installed into your new nodeenv environment, but it doesn’t seem to work like a standard ‘npm install -g’: you can’t use a directory. Also, there’s no way, when an environment is created, to ‘link’ a package (akin to a ‘develop egg’ in python), which makes it a little more difficult.

The other issues I had with nodeenv were strictly due to the fact that nodenev is modeled after virtualenv — it was never designed to be used the way I’m using it. I think I’m going to fork nodeenv and try to spruce it up and implement a zc.buildout recipe within it.

Twisted

Twisted was easier to install into the buildout, mostly because its Python based. Getting a minimal ‘hello world’  app built was a little more difficult. The documentation seems pretty good so far, but it’s written to be a comprehensive treatment, making it hard to tease out the absolute minimum code needed to solve a problem.

The first chapter of node.js’ documentation gives you the 10 or so lines you need to build a ‘Hello World’ HTTP server. I had to dig pretty deep into the Twisted docs to extract the same information, and I’m unsure if what I’ve got is truly the minimal approach. I’m also not sure if the direction I took was what people typically do.

Specifically, I wrote a plug-in for the twistd server that comes with Twisted. I used this package as the basis. I read through the documentation enough to see that plugins were an option, but the examples didn’t map well to what node.js held out as the first example.

Current Opnions

So far, I think that both tools are pretty cool. The packaging/sandboxing story for node.js is not as robust (at least at this point) as Twisted. There are some rough edges in node’s build configuration; npm makes some assumptions that make me sad. I wouldn’t discount node.js outright for these issues. I’m inclined to use the word ‘sloppy’, but at this point I think what I’m really seeing is node.js’ relative youth.

Twisted’s documentation is less accessible for someone doing this sort of comparison. I think this is an indication that Twisted is asserting itself as a general-purpose event-based network programming framework, on a platform that has a lot of really great tools for web developers. Node.js is an entire platform in itself; it’s both the framework and the toolset, and its presented as a web development tool. It’s a subtle difference, but I think the problems I’ve had so far highlight that difference.

All of that said, I think the rough edges thus far are due mostly to my own requirement of sandboxing and package-based distribution. If I just worked in a virtualenv and a nodeenv, the story so far might be less action-packed. But I think to evaluate frameworks like this, the deployment and change management stories are very important in the process.

Next Steps

The code is up at github. I’ve got a development environment that can run code in both platforms.

The next step is to pin down the application design.

To do that effectively, I need to take some time to work through some tutorials and documentation. I’m new to event-driven development, so I’m not even sure what an application like that looks like, let alone how to express it in UML.

Stay tuned!