We have two different performance milestones coming up at Connexions. One to improve performance of authoring by reducing the size of certain catalogs where lookup time is related to size, and one to improve viewing times by making our slowly changing content cacheable.
Getting ready for these performance milestones, we are updating our basic request architecture and we have made some nice performance improvements in the average times for viewing content and using the authoring system. And this is before starting on the real performance work.
We were using Squid to load balance between front end zeo servers and we switched to using HA_Proxy for load balancing. Take a look at these graphs of average load times. The Y-axis shows the time to service a particular request. We measure a few different requests and each shows up in a different color. The graph on the left shows performance for the last month and the one on the right shows performance for the year. You should notice a dramatic drop in service times a week and a half ago when HA_Proxy was added for load-balancing.
The graph below shows the same timings, but for actions that authors take. It shows a similar speedup (lower height lines) for authoring. In the graphs for authoring you may also notice a major performance improvement in February of this year. That improvement was thanks to increasing the size of an application object cache.
So why would changing the load balancer have such a big benefit? We were surprised at the magnitude, but not the direction of the improvement. Squid isn't specifically designed to do load balancing, and we were using the Internet Cache Protocol (ICP) to approximate load balancing. Squid would ask frontends to respond to an ICP request and then use the speed at which they responded "no" to determine which one to choose for the request. HA_Proxy is designed to do load balancing. It did take some configuration, but it is working much better than trying to contort ICP for load balancing. Our settings for HA_Proxy choose the front end with the least number of current connections and then choose among equals using round-robin.As background about our characteristics, Connexions serves about 2 million unique visitors per month. We receive between 50 and 60 requests per second, peaking at around 100 requests per second. Our performance is still spiky. Occasionally a request takes a very long time to serve, and that can be very frustrating for viewers and authors. We will continue to report on performance in the blog as we improve the infrastructure.
