esoTalk

Blog

Aggregating JavaScript & CSS

5 November 2011

Welcome to the new and not-so-improved esoTalk blog! For anyone reading who doesn't know, esoTalk is free, open-source forum software which is currently in development. I'm Toby, the person developing it. This blog is about my experiences doing so. It will also probably end up being a place where I go to vent my frustration with Internet Explorer. But I'm sure that's interesting enough, right?

At this stage, development is starting to wrap up. I'm working on some more minor "clean-up" kind of tasks which might not be viewed as essential features, but will add that extra bit of polish to esoTalk and bring it a cut above the rest. One of these features is automatic, built-in aggregation and minification of JavaScript and CSS files.

The Tale of Too Many Files

It's pretty common knowledge among the web development community that more HTTP requests equals slower page load times. Especially when CSS and JavaScript requests are page-blocking, meaning that the page will only be rendered in the browser when they've completed, it's essential for this aspect of a web site to be optimised.

With a basic web site, you could just throw all of your JavaScript and CSS in one file each and be done with it. But the task is made a bit more difficult with an application like esoTalk. Each page may require a different set of JavaScript code and CSS styles, and it would be inefficient to put every page's code in one huge global file of several hundred kilobytes. Furthermore, with assorted jQuery plugins being used selectively by each page, the number of HTTP requests can get unacceptably large.

Aggregation, the Right Way

So, we need to aggregate our resources, but we don't want to aggregate them all at once or the file will be ridiculously large. What an aggravating problem!

The solution is to aggregate files based on their usage around the different interfaces of the application: globally, and locally. This keeps the global code consistent across pages so it can be effectively cached by the user's browser, and then anything on top of that is generally just a single HTTP request.

esoTalk's API implements this functionality by allowing JavaScript and CSS files to be added to the controller in groups:

$controller->addJSFile(array("jquery.plugin.js", "conversation.js"));

...or to the global aggregation:

$controller->addJSFile("jquery.js", true);

Short and Static

An obvious way to go from here, now that we have lists of all of the files we want to aggregate together, might be to output <script> and <link> tags referencing a PHP script that will get the contents of the specified files and serve them all up together in a nice minified clump. Of course, we can implement a caching solution to get rid of the overhead of reading each individual JavaScript or CSS file and running the minifier, and we can send headers with our response to encourage the browser to cache the result.

This is the approach that Minify, a PHP app that aggregates JS/CSS files, seems to take. However, as Minify even mentions itself, this can be a bit dangerous:

Minify is designed for efficiency, but, for very high traffic sites, Minify may serve files slower than your HTTPd due to the CGI overhead of PHP.

It'd be much safer to be able to serve up a static file, scrap the CGI overhead, and let the web server handle things like caching headers and gzipping on its own. That is, after all, what web servers are good at!

This is what esoTalk does. When the page is finally outputted, esoTalk checks its cache folder for these aggregations, or if they haven't been created, it concatenates the contents of all the files in the aggregation, minifies the result, and writes it out to a static file.

These static aggregation files are then used as the source for our JavaScript and CSS, so we get something like:

<link rel='stylesheet' href='/esoTalk/esotalk/cache/css/base,styles,debug.css'>
<script src='/esoTalk/esotalk/cache/js/jquery,jquery.misc,jquery.history,jquery.scrollTo,global.js'></script>
<script src='/esoTalk/esotalk/cache/js/jquery.cookie,autocomplete,search.js'></script>

That's aggre-great!

Pitfalls

There are a couple of pitfalls with this kind of aggregation, but nothing that should prevent us from still using it.

The first is over-caching. We generate these aggregation files at one point in time, and then they remain the same until we delete the cached files. Naturally, we'd do this whenever the software is upgraded to a new version, and maybe even once every 24 hours or so, just to be safe. Regardless, this can still make development a bit of a pain—needing to delete the cached files every time you make a change to a JavaScript or CSS file is just a little bit too tedious for my liking!

My quick and dirty solution for esoTalk is to disable aggregation/caching all together if the "debug" option is turned on in esoTalk's configuration. Most plugin and skin development should be done with the debug setting turned on anyway.

The other pitfall is that use of a CDN (Content Delivery Network) would be tricky. esoTalk does offer an option to serve all explicitly-defined static resources from a different URL, but since our aggregation files are stored in the "cache" folder, and the exact scripts and stylesheets included will vary from page to user, we can't really rely on a CDN to serve these files. I don't currently have a solution for this, but I'll definitely keep on thinking about it.

Conclusion

This is by far the most efficient form of JavaScript + CSS aggregation I can think of, and it's built right into the core of the new version of esoTalk. We get a significantly reduced number of HTTP requests with consistent and cacheable aggregation, the size-reduction of minification, and optimisation and compression from the web server, all without the overhead of PHP/CGI.