Last night in a desperate attempt to squeeze some performance out of a plone site that was being loaded with data marshalled from a MySQL database (a new plone-based app is replacing the old generic php-introspects-the-db frontend generator-based one).
Only about 3000 objects, bad performance, lots of CPU and RAM used, etc. And it’s Plone 4. It was also eating up disk space, I’d have to pack it multiple times during the course of the import or the Data.fs would climb to 60+ GB.
I have a sinking suspicion that the bulk of the bad performance is due to some very fancy and very hastily put together roll up screens, our virtualization environment (KVM on RHEL 5), or some other factor, but I’m not sure. Profiler tools don’t seem to work well with the zope that comes with Plone 4, and what did work (CallProfiler) didn’t work with five browser views, so when optimizing we were basically flying blind.
We’ve gotten rid of any brain.getObject() calls, prudent use of memoize, plone.app.caching… we’ve done whatever we can to make it faster, and it was better, but still no bueno.
Anyway on a whim I switched my development copy of the app to use RelStorage with MySQL on the back-end (only using mysql because it’s what we’ve already got deployed). I also added memcached into the mix. The non-scientific, contrived tests I did saw a massive increase in performance (only a slight difference between using memcached and not). Almost 4000 requests per second for the default home page, vs about 900 using a stock ZEO setup on basically the same hardware (I was inadvertently testing RelStorage on 2GB of RAM vs 4 in the initial tests). Logged in. Our expensive roll up screens showed an increase from 150 requests per second to 223, with twice as much data in play (there’s an especially slow report that pulls about 1600 objects from the catalog and provides pagination.. when I did my relstorage tests I used a script to generate 900 and then copy-and-pasted them until I got over 3k).
RAM no longer climbed to the point where the VM would swap itself to death, the site remained responsive under heavy load (importing from the MySQL db via wsapi4plone) with only one client running, the database files have barely climbed over 17GB and I forgot to tweak some innodb optimization factors before I created the RelStorage tables (granted the import isn’t done yet, about 50% of the data is left to import..).
So, I’m way happy about all this. But my question now is, why doesn’t it seem like more people are using this? It’s documented very well, it’s a little weird to set up if you’re not a RDBMS person, but it was very easy. It’s fast, it’s efficient, and doesn’t require anything from the developer. You can even build it without undo if you want to make it presumably faster (I opted to not do that, but it’s a nice option). You can scale out the DB server and memcached easily (probably not easier than ZEO + FileStorage, but still very easy). You can move up to postgres or oracle if you need more heft. You can theoretically go back to ZEO if you want (I had trouble with the conversion scripts and cron4plone, but they’re fast and would probably do the trick).
RelStorage is, like, my favourite thing ever. So what’s the downside?
