Greenfinity: FireBase, AngularJS, Plone: Part 2

In my last article, I introduced Firebase and presented too simple demos, a poll and a chat application, integrated to a Python based Plone CMS website. To continue, I would like to show the details on how this works. This time, prepare for a longer reading, or ...

TL;DR: Firebase is a versatile real-time database service that makes it easy to develop real-time responsive applications. It works especially nice with an MVC framework like AngularJS.

Let's dive straight into the gory details.

Image may be NSFW.
Clik here to view.

How does it work?

Let's see the simplest way of binding your data from browser JavaScript to the Firebase database. To do so, you first would need to create a Firebase account and a database at the firebase.com website. There is an url that identifies the database and it has the following form:

https://<MY-DB-NAME>.firebaseio.com/  

However, Firebase also allows any demo database to be used without being created first (provided that you do not need authentication in your demo, and you can live with the chance that your database may be erased occasionally.) So you can skip visiting the FireBase site at all, and use a database URL like this:

https://<MY-DB-NAME>.firebaseio-demo.com/  

You can try this manually from your browser console. Open your favorite browser (I use Firefox with Firebug, but you could also use Chrome), open the JavaScript console and do the followings.

To start, you will need to load the JavaScript code for Firebase. Simply enter this from your browser's console:

 var script = document.createElement('script');
script.src = "https://cdn.firebase.com/v0/firebase.js";
document.body.appendChild(script);

To connect to the database, substitute <MY-DB-NAME> with the unique name you want to use.

var messagesRef =newFirebase('https://<MY-DB-NAME>.firebaseio-demo.com/messages');

This will associate messageRef to the /messages location within your database. Following this you can read and write the data via the API that FireBase provides. Let's write some data into it:

messagesRef.push({'user_id':'anonymous51','text':'I can haz my Plone beautified!'});

The push method considers /messages as a collection, and appends the data we provided above as a new element to it. Firebase has the general characteristics that it creates empty data structures on demand, which means that the empty Array or Object will be created when we first write into it.

Following this, you can read back the data you just wrote (you can try this from a different browser to demonstrate that the data is, indeed, in the cloud). In JavaScript, we have to use an event listener for this.

messagesRef.on('value',function (messagesSnapshot){var messages = messagesSnapshot.val();
  var nr = 0;
  for (m in messages) nr++; // this is needed because it's an object, not an array
  console.log('Number of messages:', nr);
});

You can now try to add various data to the /messages array, and do whatever you want to happen as a reaction to the data change on all connected clients.

Notice how I am counting the number of messages instead of using just messages.length? This is something that you will see often: instead of an array, a dictionary object with automatic, non-numeric keys is created in Firebase to represent an ordered collection of items. Don't worry: it's also possible to use numeric values as indices and handle the collections as genuine Arrays. But at the end of the day, this will not make much difference to our application.

Image may be NSFW.
Clik here to view.

Connecting the database with Angular

The usage will become even simpler if we use Firebase together with an MVC. In the demos, I am not using the interface I just described. Instead, I am using the AngularJS bindings for FireBase. This offfers a higher level API when one can simply bind $scope.myVariable to a given url in the database. Following this, one can have the local data automagically replicated in the cloud in a bi-directional way.

There are two flavors of this binding which offer different data synchronization, as explained in the documentation: explicit and implicit. Both flavors are demonstrated in the chat and poll demos.

In both modes, the changes made on the server are always populated to the clients automatically. The difference berween the two flavors is how the client's changes are propagated back to the server.

In explicit mode, we need to explicitly signal the change of the client data. This means that without calling the right API method, the database will not notice your local changes. In implicit mode however the local changes are picked up automatically, and the changes are propagated to the server (thus to all the other clients as well) by default.

angularFire($scope.firebase_url + '/choices', $scope, 'choices', []).then(function () {
    $scope.addChoice = function () {
        $scope.choices.push({
            label: '',
            count: 0    
        });
    };
    $scope.removeChoice = function (choice) {
        $scope.choices.splice($scope.choices.indexOf(choice), 1);
    };
);

While explicit more offers more control about the network traffic, implicit mode works nice and easy with Angular.

Admin interface

FireBase provides a TTW administration interface to the database, called Forge. I've shown its screenshot above.

Besides various administrative settings (security, authentication) the interface provides introspection to the database content. The changes our app makes in the data are reflected here in real-time. It is also possible to manipulate (add, edit, delete) data records directly from here.

Access rules

The authentication scheme of Firebase is powerful and versatile. In the admin interface (Forge) we can set up the security access rules for the entire database.

{
"rules": {
"chat": 
".read": "auth !== null && auth.ploneUserid != ''",
".write": "auth !== null && auth.ploneUserid != ''"
        },
"poll":
".read": "auth !== null && auth.ploneUserid != ''",
".write": "auth !== null && auth.ploneUserid != ''"
        },
        ...
    }
}

The expressions can also contain variables that originate from the agent that authenticated us. A simple example for this is a $user variable, that must match the Plone userid. Using this it is possible to set up database locations where users can only access their own data but not the data of other users. Similarly, we could further develop these rules to also handle role based permissions, where for example an admin user could access the data of all users.

{
"rules": {
"plone_interact": {
"users": {
"$user": {
".read": "auth !== null && auth.ploneUserid == $user",
".write": "auth !== null && auth.ploneUserid == $user" 
                }
            }   
        }
    }
}

Authentication

To allow security and access control, someone must authenticate the client, otherwise the database would have no way of implement access restrictions.

In the Plone demo, I used custom authentication. This means that there is a server view that knows the logged in user's Plone username, and can create an authentication token which then the client will use in the communication to Firebase. In the demos this is integrated into the portlet code, but you can easily apply it to your use case, or do the same with other Python front-ends such as Django or Pyramid.

fromfirebase_token_generatorimportcreate_token
    
firebase_secret = "***********************"

def gen_auth_token(context, request):
    portal_state=getMultiAdapter((context, request),name="plone_portal_state")
    plone_userid=portal_state.member().getId()
    custom_data={
        'ploneUserid':plone_userid,
    }    
    admin=False
    options={'admin':admin}
    token=create_token(firebase_secret,custom_data,options)
    returntoken

Firebase also offers authentication via other agents, most notably Facebook, Twitter and GitHub. It also offers email based authentication. So, you can make a single page web app that authenticates with one or all of these agents to Firebase, without the need to maintain a web server yourself.

Presence tracking

Firebase offers a way to track the presence of the users that are currently watching the page. This can be used for two main purposes.

First, one can notice when the client's network connection goes online or offline. This is essential for applications that must always be connected to the database, and being offline is considered to be an error condition that needs to be handled specially.

Second, it is possible to actually store in the database the logged in clients, which makes it possible that other clients can display which users are currently logged in. The following code server both purposes.

varonlineRef=newFirebase('https://<MY-DB-NAME>.firebaseio.com/users/');
varconnectedRef=newFirebase('https://<MY-DB-NAME>.firebaseio.com/.info/connected');

connectedRef.on('value',function(snap){

    console.log('CONN',snap.val());

    if(snap.val()===true){

        // We're connected (or reconnected)!  Set up our presence state and
        // tell the server to set a timestamp when we leave.
        
        varuserRef=onlineRef.child($scope.plone_username);
        varconnRef=userRef.child('online').push(1);
        
        // Tell the server what to do in case of disconnect.
        connRef.onDisconnect().remove();

        // Also strore the 'last seen' timestamp for this user.
        userRef.child('logout').onDisconnect().set(Firebase.ServerValue.TIMESTAMP);
    }

});

It is worth noticing that the server guarantees that the onDisconnect event executes even if the client falls out from the network without a notice, which means that the server itself will execute the desired action on going offline, which guarantees data consistency in any case. In this example, the server also stores when a user went offline, so we know when a given user was last seen online.

I am working on adding a presence portlet to the Plone demo.

Practical observations

This is very nice in theory. The question is, of course, how it will work in the practice, especially when we get to more complex use cases, and scale up the size of the application and the number of its users as well. So far, my experiences are positive, but while Firebase tries to combine the best of each concept to provide an easy to use, efficient real time database service, its decisions inevitably introduce tradeoffs. So it's important to know what these are. Here I attempt to show a glimpse of the few issues that I have experienced so far.

No database procedures

You cannot execute procedures on the database server itself. In most of the practical cases I find that this is not a big problem, as one can most of the time structure the data in a way that allows the clients to handle it directly. If this is not enough, one can deploy a daemon that connects to the database from remotely and does whatever task it needs to achieve.

Explicit vs. implicit data coupling

This is specific to the Angular support, not Firebase itself. As I mentioned above, explicit data synchronization requires more work from the programmer, but it can also be more performant in some cases. Use explicit synchronization if you want to be in better control above what gets transferred through the network.

Larger data and images

What happens if the data that has to go through the network, is large? Would that not cause performance problems?

In one application, I tried to store images in the database. In the use case, the user can upload images to the database that later get displayed and manipulated in various ways. This works easier than one would think, since it is possible to inline the images to the html itself, so we can get away without needing an additional server in addition to Firebase to serve the images. Still, larger amount of data needs to travel through the net, and I observed that this slowed down the reponsiveness of the app noticeably.

This is the moment when it is less beneficial to use the implicit synchronization of angular. Instead, one would want more control and fall back to use explicit data coupling, as I described earlier, in order to get better performance.

Limitation of the REST API and the Python bindings

Firebase supports a full grown JavaScript and Objective C API. There is also a REST API which makes it easy to write libraries to use Firebase from a web application written in other languages, such as Python.

However the REST API and thus the Python library that is based on top of it, have a serious limitation. They can read or write data, but they lack the capability to listen to real time changes from the server. Simply put, this means that while you have full access to your data, you partly loose the ability to queue management and cannot receive push notifications via the REST interface.

There are emerging libraries that would give the full Firebase functionality in Python as well which assumes the usage of network sockets instead of REST HTTP. At the time when I write this article, I have not checked the available solutions yet, so I don't know their usability status. Instead, I decided to write my simple server side applications in NodeJS instead of Python. Naturally, having a full fledged Python API would be a nice thing for anyone who is familiar with the mature libraries of this great programming language.

Limits of access rule based security and MVC

I had to implement a simple and basic use case: a queue that each user can push an item into, but that only a privileged user can read in its entirety. The following security declaration does the trick (in this use case any authenticated user counts as an admin, which means auth != null is only true for admins, but never for normal users):

{
"rules": {
"submitQ": 
".read": "auth != null",
".write": true
        }
    }
}

However when trying this I had to notice that this usecase does not work well with MVC style data synchronization. The basic problem is that the user cannot read the entire queue but still would need to push a new item into it, somehow. As an example, think about a landing web page where interested users can submit their email address, so that the owners of the landing page can later notify them. It is obvious however that users should not be allowed the entire mailing list collected so far, so it's out of the question to replicate the full array of emails to the client.

Luckily, this is no problem at all to carry out with Firebase's principal API that I presented in the beginning of my article. And one can freely combine this way of access with the AngularJS interface. So there is no practical limitation here, one just need to follow a golden rule of framework usage: use the proper tool in each use case, and don't think that just because you have a framework, you are bound to use it always.

Summary

Firebase is an interesting database service that ought to be kept on your radar. It is very easy to use it in combination with an MVC such as AngularJS. It has some tradeoffs when you want to implement more complex use cases with it, but its ease of use for developers makes it worth for you to consider it as a solution, when you need both data persistence and queue notification for your application.

In addition to introduce Firebase, I also tried to prove in this article, that MVC applications can play nice together with traditional (in our case Python based) server frameworks such as the Plone CMS.