A short time later, I realised that I had made a huge mistake, and had wasted months of development on a product that would ultimately need to be rewritten—again.

Fatal Error: $coding_ego too large

So, what did I do wrong? To summarise it in a few words: I reinvented the wheel. Many times.

When I rewrote esoTalk, I wrote it from scratch, using no frameworks and no libraries. I wrote all of my own code to do things—database, MVC, session, JavaScript, etc.—when there was already code out there that was free, open-source, actively maintained, and most importantly, better.

Why did I do this? I had some kind of prejudice or bias against frameworks and code that other people wrote. I shallowly assumed that all PHP frameworks were big and clunky like CakePHP, and I didn’t want to have anything to do with them.

Stupid preconceptions aside, I started to discover the power of frameworks and libraries, and the amount of time it would save. I mean, you get all this stuff for free! Code, development, documentation—a whole bunch of stuff taken off your hands that you don’t have to worry about anymore.

Cair Paravel… er, Laravel

Then I discovered Laravel. Laravel took all my concerns about frameworks being slow, complex, and clunky, and threw them out the window. It is beautifully easy. I immediately decided that I wanted to rewrite esoTalk upon Laravel.

Since then, I made a small start on the conversion to Laravel, but it’s unfortunately quite a mammoth task at hand and I wasn’t getting very far. I decided to take a break from esoTalk and dedicate my time to other projects I’m working on, including ones that use Laravel to increase my experience with it.

Through these projects, I also discovered Twitter Bootstrap which I think will work very well for esoTalk. I’m sure I’ll discover more handy libraries now that I know that I should be looking.

Now, as I wrap these other projects up, I’m approaching the start of my university course. This quite simply means I will have very little time to work on esoTalk.

Inactive Development

I’m sorry to say this, but… Consider esoTalk on hold, indefinitely. Again.

Note, though, that I said “on hold” instead of “abandoned” or “given up.” And I said “indefinitely” instead of “forever” or “for 20 years.”

I think esoTalk is an amazing, unique piece of discussion software. It throws out all of the archaic forum conventions and provides a modern solution for discussion on the web. Rest assured that one day—perhaps a year from now, perhaps more—I will release an esoTalk that will blow all other forum software out of the water.

I will continue work on esoTalk as a hobby whenever I get the chance. I’ll still be around on the support forums. Just don’t expect any releases or updates anytime soon.

Of course, feel free to fork the project on GitHub and make your own improvements. The current version isn’t horrible, it’s just not what I want esoTalk to be in the long-term.

I’m not a big fan of dependencies. When I make software, I like to have complete control and understanding over everything I’m working with—that’s just the way I work as a programmer, I guess. Perhaps it’s a flaw. Perhaps not. But for this reason, I chose not to use an existing PHP framework.

Every framework makes assumptions about the applications that will be built upon it, and therefore you’re likely to lose a layer of control and efficiency when building upon one. With an application like esoTalk, which has a very specific purpose, I wanted to keep the code focused and concise. Instead of using a framework, I built my own pseudo-framework specifically for esoTalk. I planned everything out in a way I thought was logical, and only included components that esoTalk needs.

I’m pretty happy with the result. While it may not be as robust, powerful, or proven as existing frameworks, it can be and is tailored specifically to esoTalk’s needs—and that’s an advantage.

Since it might be a little while before proper documentation is out (although it is a high priority!), I thought I’d write a blog post about the basics of the esoTalk “framework”. Most of this can be learned by reading inline documentation (in code comments), but I thought it would be good to lay it all out together in a more understandable way.

ET

“ET” is used as a prefix for every class in esoTalk, and obviously stands for “esoTalk”.

There is a static class named “ET”. This class serves as a central means of access to various services and functions which are needed around the application, including:

• the database object (ET::$database)
• the session object (ET::$session)
• configuration reading & writing (ET::config() and ET::writeConfig())
• language translation (ET::translate(), ET::define(), ET::loadLanguage())
• lazy instantiation of various classes (ET::memberModel(), ET::formatter(), etc.)
• loaded plugins (ET::$plugins)
• the loaded skin (ET::$skin)
• event triggering (ET::trigger())
• error rendering (ET::pageNotFound() and ET::fatalError())

ETFactory

ETFactory is a static class which simply allows classes to be instantiated without having to know what the real name of the class is. For example, the database object could be created using ETFactory::make(“database”), whatever class that may refer to, be it the default ETDatabase or some class that a plugin may want to override the default with.

Anatomy of a Request

The only point of entry to esoTalk is index.php. Every request goes through this file. The process that takes place in index.php is as follows:

1. Set up the environment. Define constants, set up error handling, etc.
2. Include the configuration files. config.defaults.php is included first, and then anything in config/config.php takes precedence.
3. Register default classes with ETFactory, including helpers, models, and controllers.
4. Include plugins.
5. Instantiate the cache, session, and database objects, and set them as properties of the static ET class.
6. Include the skin and language.
7. Parse the request string (The “conversation/lock/123″ part of “http://esotalk.org/forum/conversation/lock/123″):
• The first part maps to the controller.
• The second part maps to the controller method, if it exists as a non-inherited public method of the controller class. Otherwise, we default to the “index” method.
• Any subsequent parts map to arguments which are passed to the controller method.
8. Dispatch the method and arguments to the controller. For “conversation/lock/123″, we will essentially be calling the “lock” method of the “conversation” controller, passing “123″ as a function argument.

Controllers

A controller takes user input from a request, handles it, and responds by rendering content/data.

Controllers inherit a bunch of methods to do things common to the esoTalk web interface, including methods to:

• Respond by rendering views or redirecting ($this->render(), $this->redirect())
• Set data to be passed to a view before it is rendered ($this->data())
• Add messages to be displayed when the page is rendered ($this->message())
• Add JS/CSS files to be included on the page ($this->addJSFile(), $this->addCSSFile())

Let’s take a look at what a controller method could look like:

public function index($memberId)
{
    // Insert code to get member data for the $memberId.
    // Let's assume that we get this data and set it to $member.

    // Pass along the $member data to the view.
    $this->data("member", $member);

    // Show a message in the bottom-left of the screen, because we can.
    $this->message("Hi, this is a message.", "warning");

    // Render the page, specifying what view we want to use.
    $this->render("member/profile");
}

Views

Views are simply files that output HTML code. Following along from the previous example, our “member/profile” view might look like:

<?php
$member = $data["member"];
?>
<h1><?php echo $member["username"]; ?>'s Profile</h1>

Response Types

The response type specifies the type of content that we want the controller to respond with. The response type can be specified in the request by putting a period following the method name, followed by the response type we want. For example, “conversation/lock.json/123″ would set the controller’s response type to “json” and call the “lock” method.

By default, the response type is “default”, which instructs the controller’s render() method to render the content inside the master view (the HTML wrapper.) Other response types include:

• “view”, which will render just the inner view without the HTML wrapper.
• “json”, which will output a JSON object using data collected with the $this->json() method in a controller.
• “ajax”, which is the same as “json”, but will render the inner view into the “view” key of the JSON object.

Models

Models handle data. A model contains methods to read and write data specific to a type of record. For example, some of the methods in the conversation controller are getById, getMembersAllowed, addReply, and setSticky.

Data should almost always be read and written using models. This allows models to enforce data integrity and keeps everything consistent between controllers.

Conclusion

I think that covers all of the basic parts of code that make up esoTalk. Of course, there are other helper classes and functions (to help with building SQL queries, URLs, forms, uploading files, formatting text, and rendering common assets), but I won’t go into detail about them right now. All functions and methods are documented, and I’ve made sure to write plenty of inline comments, so you can discover a lot just by looking through the source code. Don’t hesitate to ask questions on the esoTalk support forum!

Again, I’m pretty happy with the code structure of esoTalk. It’s simple, efficient, and extensible. But, of course, there’s always room for improvement—what are your thoughts, and how could esoTalk’s code structure be made better?

So, what about this “moving forward”? What happens now? In simplest terms, development continues, until we get something feature-complete and stable, and we work towards building an infrastructure that encourages active development of plugins and skins. In more detail (and with loose time goals in parenthesis)…

Feature-Complete (End of November)

The version of esoTalk released today is missing a bunch of essential features. Atom Feeds, XML Sitemaps, Emoticons, CAPTCHA, and IE7 support are the main ones. It also need a thorough security review and many additional plugin hooks. Over the next month or so, I’ll be aiming to implement all of this and have esoTalk in a relatively stable state, sans a few minor bugs and issues. Also high on my priority list will be writing extensive documentation and working on a few more plugins.

Ecosystem (Mid-December)

After esoTalk’s development is wrapping up (and hopefully the community has come on board with helping out!) I’ll begin working on developing an ecosystem for plugins, skins, and languages. This will encourage active development, enabling developers to easily distribute and update their esoTalk extensions. I also want to streamline the process for installing plugins, skins, and languages.

Stable Launch (January)

Once the above things come together, and the issues list calms down, I’ll prepare for a stable launch of the esoTalk software. I’ll do a bunch of promotion to try and spread the use of esoTalk and build up a nice, healthy community of our own. And then I’ll party!

After that, I’ll go down to my secret lair to begin work on a secret project.

At this stage, development is starting to wrap up. I’m working on some more minor “clean-up” kind of tasks which might not be viewed as essential features, but will add that extra bit of polish to esoTalk and bring it a cut above the rest. One of these features is automatic, built-in aggregation and minification of JavaScript and CSS files.

The Tale of Too Many Files

It’s pretty common knowledge among the web development community that more HTTP requests equals slower page load times. Especially when CSS and JavaScript requests are page-blocking, meaning that the page will only be rendered in the browser when they’ve completed, it’s essential for this aspect of a web site to be optimised.

With a basic web site, you could just throw all of your JavaScript and CSS in one file each and be done with it. But the task is made a bit more difficult with an application like esoTalk. Each page may require a different set of JavaScript code and CSS styles, and it would be inefficient to put every page’s code in one huge global file of several hundred kilobytes. Furthermore, with assorted jQuery plugins being used selectively by each page, the number of HTTP requests can get unacceptably large.

Aggregation, the Right Way

So, we need to aggregate our resources, but we don’t want to aggregate them all at once or the file will be ridiculously large. What an aggravating problem!

The solution is to aggregate files based on their usage around the different interfaces of the application: globally, and locally. This keeps the global code consistent across pages so it can be effectively cached by the user’s browser, and then anything on top of that is generally just a single HTTP request.

esoTalk’s API implements this functionality by allowing JavaScript and CSS files to be added to the controller in groups:

$controller->addJSFile(array("jquery.plugin.js", "conversation.js"));

…or to the global aggregation:

$controller->addJSFile("jquery.js", true);

Short and Static

An obvious way to go from here, now that we have lists of all of the files we want to aggregate together, might be to output <script> and <link> tags referencing a PHP script that will get the contents of the specified files and serve them all up together in a nice minified clump. Of course, we can implement a caching solution to get rid of the overhead of reading each individual JavaScript or CSS file and running the minifier, and we can send headers with our response to encourage the browser to cache the result.

This is the approach that Minify, a PHP app that aggregates JS/CSS files, seems to take. However, as Minify even mentions itself, this can be a bit dangerous:

Minify is designed for efficiency, but, for very high traffic sites, Minify may serve files slower than your HTTPd due to the CGI overhead of PHP.

It’d be much safer to be able to serve up a static file, scrap the CGI overhead, and let the web server handle things like caching headers and gzipping on its own. That is, after all, what web servers are good at!

This is what esoTalk does. When the page is finally outputted, esoTalk checks its cache folder for these aggregations, or if they haven’t been created, it concatenates the contents of all the files in the aggregation, minifies the result, and writes it out to a static file.

These static aggregation files are then used as the source for our JavaScript and CSS, so we get something like:

<link rel='stylesheet' href='/esoTalk/esotalk/cache/css/base,styles,debug.css'>
<script src='/esoTalk/esotalk/cache/js/jquery,jquery.misc,jquery.history,jquery.scrollTo,global.js'></script>
<script src='/esoTalk/esotalk/cache/js/jquery.cookie,autocomplete,search.js'></script>

That’s aggre-great!

Pitfalls

There are a couple of pitfalls with this kind of aggregation, but nothing that should prevent us from still using it.

The first is over-caching. We generate these aggregation files at one point in time, and then they remain the same until we delete the cached files. Naturally, we’d do this whenever the software is upgraded to a new version, and maybe even once every 24 hours or so, just to be safe. Regardless, this can still make development a bit of a pain—needing to delete the cached files every time you make a change to a JavaScript or CSS file is just a little bit too tedious for my liking!

My quick and dirty solution for esoTalk is to disable aggregation/caching all together if the “debug” option is turned on in esoTalk’s configuration. Most plugin and skin development should be done with the debug setting turned on anyway.

The other pitfall is that use of a CDN (Content Delivery Network) would be tricky. esoTalk does offer an option to serve all explicitly-defined static resources from a different URL, but since our aggregation files are stored in the “cache” folder, and the exact scripts and stylesheets included will vary from page to user, we can’t really rely on a CDN to serve these files. I don’t currently have a solution for this, but I’ll definitely keep on thinking about it.

Conclusion

This is by far the most efficient form of JavaScript + CSS aggregation I can think of, and it’s built right into the core of the new version of esoTalk. We get a significantly reduced number of HTTP requests with consistent and cacheable aggregation, the size-reduction of minification, and optimisation and compression from the web server, all without the overhead of PHP/CGI.