In a typical Python web development project, you need JavaScript as well on top of Python. Here, I will show a method to satisfy your JavaScript installation and processing needs - straight from zc.buildout, the well known tool for the installation of advanced Python projects.
TL;DR
In this article, I will focus on how to do installation of server side JavaScript from bin/buildout. In upcoming posts, I will show what you can actually use server side JavaScript for, and also, how you can process client-side (browser) JavaScript for your web application.
Additional dependencies
First, we need some dependencies that buildout cannot handle itself. You need to install NodeJS and npm, the Node Package Manager. On MacOSX, this can be done in the following way, if you use MacPorts:
$ sudo port install nodejs
$ sudo port install npm
On Ubuntu, you would do the following instead:
$ sudo add-apt-repository ppa:chris-lea/node.js
$ sudo aptitude install nodejs npm
Check out the instructions for installing Node on your favorite OS here.
The good news is that while this is an extra dependency, it's the only dependency you will need. With this you can use the tools and follow the practices of the JavaScript developer community, without the need to use applications ported to Python, Java, or PHP, reinventing the wheel.
See it in action
We can use the example buildout of Substance D for demonstration. Substance D is an application development environment built using the Pyramid web framework. Substance D is used only for demonstration here, the same method works equally well for any Python, buildout based web application. That said, you can skip trying this buildout if you want, and add a similar configuration to your own project right away.
Let's check out the sources. Sdidev is the buildout for developing on Substance D. It is meant to be used in development only and the buildout is not needed in production. Note that for this demo, the ree-grunt-sprinting branch of the sdidev buildout has to be used.
$ git clone git@github.com:Pylons/sdidev.git
$ cd sdidev
$ git checkout ree-grunt-sprinting
Do a virtualenv:
$ virtualenv -p python2.7 .
And make a buildout:
$ bin/python2.7 bootstrap.py
$ bin/buildout
At the end of the buildout's log you will see some interesting things happening on the console. After building out our Python, zc.buildout starts showing colorful signs of progress with building out our - surprise! - JavaScript.
Installing node_modules.
npm http GET https://registry.npmjs.org/grunt-cli
npm http GET https://registry.npmjs.org/grunt
npm http GET https://registry.npmjs.org/yo
npm http GET https://registry.npmjs.org/bower
--- SNIP ---
NodeJS server side modules updated.
... Fun isn't it? But wait, we have not finished yet; it goes on:
Installing bower_modules.
bower copying /private/tmp/gruntdemo/sdidev/src/slickgrid
bower fetching slickgrid
bower cloning git://github.com/twitter/bootstrap.git
bower cached git://github.com/twitter/bootstrap.git
bower fetching bootstrap
bower cloning git://github.com/components/jquery.git
bower cached git://github.com/components/jquery.git
bower fetching jquery
bower checking out jquery#1.9.1
--- SNIP ---
Bower client-side modules updated.
What just happened?
Let me explain.
In our project's buildout.cfg, we are extending yeoman.cfg. Yeoman is Twitter's bundle to support processing and installation tasks for web development. It contains three tools: Grunt, Bower, and Yo. I will talk about what these do in an upcoming article. Grunt can also be installed independently from Twitter's Yeoman.
The yeoman.cfg is a buildout extension config that I created to cover all buildout tasks we deal with. You can take yeoman.cfg with you, and use it from your own buildout without the need to change it in any way. The only thing you add to your buildout.cfg is:
[buildout]
extends = yeoman.cfg
The yeoman.cfg file contains the rules for building JavaScript. It contains a few parts each doing a different installation step.
[buildout]
parts =
node_modules
bower_modules
grunt_script
bower_script
yo_script
The first part does the installation of selected Node modules via npm and will be run each time you update the buildout.
[node_modules]
recipe = collective.recipe.cmd
on_install = true
on_update = true
cmds = NODE_PATH="" npm install .; echo "\nNodeJS server side modules updated.\n"
npm is the standard tool to install server side (Node) JavaScript. "npm install ." will install the package at the current local directory, with all its dependencies. To make it work, we need to turn our buildout into a virtual Node package, thus enabling to specify its dependencies in a package.json file. Note that this file is a data configuration in JSON format, which means that generic JavaScript is not allowed in it.
{
"name": "sdidev-resources",
"version": "0.1.0",
"devDependencies": {
"grunt": ">=0.4"
}
}
You can try to issue
NODE_PATH="" npm install .
from the console, if you are in the buildout root, to see how this installation works. The same will be executed from bin/buildout, via the node_modules buildout rule.
A more complete package.json version will also install some other tools you will find useful. By adding your own package dependencies to it you can satisfy virtually all your needs. I will talk more about the usage of these tools in the upcoming parts of this post.
{
"name": "sdidev-resources",
"version": "0.1.0",
"devDependencies": {
"yo": "*",
"grunt-cli": "*",
"bower": "*",
"grunt": ">=0.4",
"grunt-contrib-jshint": "*",
"grunt-contrib-uglify": "*",
"grunt-contrib-concat": "*",
"grunt-contrib-less": "*",
"grunt-contrib-watch": "*"
}
}
How it works
The Node Package Manager provides two ways for installation: global or local. Global installation is done with the -g parameter. Global installation is often discouraged, partly because it usually requires admin privileges. In the solution I present here, we use no global installation, instead, a local installation is utilized. This has a few advantages for us.
- The installation will be local to the buildout, and separated from the other buildouts, allowing each buildout to have its own private working set of installed packages and versions.
- The NODE_PATH="" setting in the node_modules buildout rule will cause your globally installed Node packages to be ignored as well, and the usage of your local installation will be enforced. NODE_PATH has a similar role in Node as PYTHONPATH in Python.
This will cause all packages installed locally into the node_modules directory in the buildout root. You can delete node_modules and re-run bin/buildout to regenerate its contents - in the same way as it would happen if you deleted the Python eggs directory.
The only thing left to do is running executable applications. The following buildout rule takes care of that in case of grunt:
[grunt_script]
recipe = collective.recipe.template
input = inline:
#! /bin/sh
export NODE_PATH=${buildout:directory}/node_modules
${buildout:directory}/node_modules/.bin/grunt $@
output = ${buildout:bin-directory}/grunt
mode = 755
This will create a bin/grunt executable script, which will execute "./node_modules/.bin/grunt", the executable that npm has installed. This means that the JavaScript tools will be runnable from bin/* in the same way as all other buildout scripts.
$ bin/grunt --help
Want to use even shorter commands? Make use of virtualenv, if you prefer so. In this case you will not even need to put the bin/ in front of the script names. For example:
$ source bin/activate
$ python2.7 bootstrap.py
$ buildout
$ grunt --help
Summary
In this post I showed how JavaScript NodeJS installation can be integrated to zc.buildout for a Python web application, and how this integration works. Next I will show how to use your newly installed tools for your web development project.