5 November 2011
Welcome to the new and not-so-improved esoTalk blog! For anyone reading who doesn't know, esoTalk is free, open-source forum software which is currently in development. I'm Toby, the person developing it. This blog is about my experiences doing so. It will also probably end up being a place where I go to vent my frustration with Internet Explorer. But I'm sure that's interesting enough, right?
The Tale of Too Many Files
Aggregation, the Right Way
So, we need to aggregate our resources, but we don't want to aggregate them all at once or the file will be ridiculously large. What an aggravating problem!
The solution is to aggregate files based on their usage around the different interfaces of the application: globally, and locally. This keeps the global code consistent across pages so it can be effectively cached by the user's browser, and then anything on top of that is generally just a single HTTP request.
...or to the global aggregation:
Short and Static
An obvious way to go from here, now that we have lists of all of the files we want to aggregate together, might be to output
This is the approach that Minify, a PHP app that aggregates JS/CSS files, seems to take. However, as Minify even mentions itself, this can be a bit dangerous:
Minify is designed for efficiency, but, for very high traffic sites, Minify may serve files slower than your HTTPd due to the CGI overhead of PHP.
It'd be much safer to be able to serve up a static file, scrap the CGI overhead, and let the web server handle things like caching headers and gzipping on its own. That is, after all, what web servers are good at!
This is what esoTalk does. When the page is finally outputted, esoTalk checks its cache folder for these aggregations, or if they haven't been created, it concatenates the contents of all the files in the aggregation, minifies the result, and writes it out to a static file.
<link rel='stylesheet' href='/esoTalk/esotalk/cache/css/base,styles,debug.css'> <script src='/esoTalk/esotalk/cache/js/jquery,jquery.misc,jquery.history,jquery.scrollTo,global.js'></script> <script src='/esoTalk/esotalk/cache/js/jquery.cookie,autocomplete,search.js'></script>
There are a couple of pitfalls with this kind of aggregation, but nothing that should prevent us from still using it.
My quick and dirty solution for esoTalk is to disable aggregation/caching all together if the "debug" option is turned on in esoTalk's configuration. Most plugin and skin development should be done with the debug setting turned on anyway.
The other pitfall is that use of a CDN (Content Delivery Network) would be tricky. esoTalk does offer an option to serve all explicitly-defined static resources from a different URL, but since our aggregation files are stored in the "cache" folder, and the exact scripts and stylesheets included will vary from page to user, we can't really rely on a CDN to serve these files. I don't currently have a solution for this, but I'll definitely keep on thinking about it.