Hating XML is rather fashionable these days, so many API’s are appearing that are JSON representation only even when XML really is a better fit for purpose. When you’re designing a hypermedia API you’re not simply serialising an object. There are media types that add semantic meaning to JSON (application/hal+json, my own application/vnd.error+json (draft) and others) but really, XML is so much more expressive that using JSON actually makes your clients have to work harder to understand the meaning you’re trying to convey using an object serialisation format over a markup language.

XML gives you some stuff for free – ‘xml:lang’ for one. Put this on any element of your document and just the fact that your document is XML means that you can express elements in different (human) languages without having to update your media type.

Consider how this actually looks in XML vs JSON representation.

<error id="6">
    <message>Something bad happened</message>
</error>

{
    "error": {
        "id": 6,
        "message": "Something bad happened"
    }
}

This looks easy, right? Consider a future version of the api where we add support for multiple languages.

<error id="6" xml:lang="en">
    <message>Something bad happened</message>
    <message xml:lang="de">Something bad happened (in german)</message>
</error>

{
    "id": 6,
    "messages": [
        { "lang": "en", "message": "Something bad happened" }
        { "lang": "de", "message": "Something bad happened (in german)" }
    ]
}

An alternative could be to key the message objects by language;

{
    "id": 6,
    "messages": { 
        "en": "Something bad happened",
        "de": "Something bad happened (in german)"
    }
}

You can see in both scenarios that “messages” has become an different type (string to array of objects, or string to object). We’ve changed the structure of the document and introduced an almost certain backwards compatibility (BC) break in our interface – all our clients now need to be updates to follow suit. We could leave “message” as it was, and add “messages” for the multilang version of the API but then we’re duplicating data and / or making the client have to work harder.

The XML version can continue as is and the client can simply ignore the additional languages until they’re updated to support them (if they ever need to).

Then there’s just the matter of basic human readability. It’s clear to see what’s going on in XML and not so much in JSON. This is a fairly simplistic example but extrapolate that out over a fairly complex representation of an object and you can see how JSON really isn’t always the best choice for representing resources.

Imagine the scenario where we had used the second option above as our JSON representation and we wanted to add in another attribute for each of the messages. See the problem? Anything we choose to do will cause a BC break and require clients to update (or result in duplication of elements to ‘version’ the api), meaning it’s very difficult to evolve the clients and servers independently.

So i’m not saying that JSON simply ‘sucks’. I’m just saying that when considering the formats you use for representing data in an API (when it comes to selecting the media type you want to use), think hard before just going with something based on JSON by default because it’s ‘easy’. It probably isn’t.

Because of this, several people have come together and put together a proposal for the application/vnd.error media type. It has been documented over on Github and the mime type has been submitted for approval by IANA.

The intention for this media type is that it should be used when a client has made a request, however the server is unable to fulfil the request to to either a server side error (a code 5xx) or a problem with the client request (a code 4xx).

Although the client does NOT have to state that it explicitly will accept the application/vnd.error+xml or json mime type (via the ‘Accept’ header in the request), it makes a lot of sense to do so if the client can understand that response type. However, RFC2616 section 10 states:

HTTP/1.1 servers are allowed to return responses which are not acceptable according to the accept headers sent in the request. In some cases, this may even be preferable to sending a 406 response. User agents are encouraged to inspect the headers of an incoming response to determine if it is acceptable.

It is up to the implementor to decide whether the api should return a 406 (Not Acceptable) response if the client states it will not accept application/vnd.error+xml or json and an error is triggere, or if it should return the application/vnd.error+xml or json response anyway.

The specification for application/vnd.error is draft – if you have comments / suggestions or enhancements to the media type, let’s talk (comments on this blog, issues on github, twitter etc).


wget http://silex.sensiolabs.org/get/silex.phar
php -S localhost:8080


Create an index.php in the current directory (there’s an example app on the Silex home page). That’s it. Go play. :)

My problem is that I trust my developers to do the right thing. Ok, so that’s not a problem – it’s actually a really nice place to be. I see code review as a learning exercise for good developers, but I don’t see is as a blocking part of the development cycle. Too many times I have seen code reviews as a blocking step in a software release cycle end up just being a formality to get through.

With that scene set i’ve been thinking of using a karma based code review system. Developers have karma (per project) and can give out positive and negative karma to other developers. At a certain level of karma, the number of reviews required on commits starts to drop off so that eventually a commit by a developer with high karma will not generate a code review (or the percentage of commits submitted for review will be very low). As a project progresses, hopefully this leads to a place where developers who will learn more from having reviews done will have the benefit, while the developers who are well established on the project are able to get code into the project repository without a blocking code review process. Improvement can be measured by karma increasing over time.

Do you have experience with code reviews on projects? (open source vs closed source / colo teams vs non-colo / blocking vs non blocking)… Interested to hear others success and failure stories!

And soon the problems start coming up. At the end of the first sprint the client raises a few bugs with the software delivery… We didn’t account for those in the plan! And we can’t estimate them – we don’t know the size of the problem. That’s why it’s a bug… So we allocate some time in the next sprint to work on the top priority bugs and commit to developing fewer features.

The second sprint finishes, and more defects are raised in the software – some of them can be attributed to requirements not been defined clearly enough, but some of them are just code that we don’t understand how it even worked in development! Did someone actually test it before it went into the main development branch?! (how many people can honestly say they have never heard that one before…?)

Pretty soon you’re way behind the plan. The number of bugs that we cannot estimate and didn’t plan for are sucking up too much time and we don’t know how long it’s going to take to get back to a point where we know how long the project is going to take.

So what’s to be done? Bugs throughout the development life cycle are inevitable – but because we cannot know how many there will be and how long they will take to fix, we need to be able to minimise the number that crop up.

Some research done by microsoft (http://research.microsoft.com/en-us/groups/ese/nagappan_tdd.pdf) has shown that spending 15-35% extra development time on adopting test driven development resulted in a pre-release defect density decrease of 40-90%.

The ideal is when we can plan, develop and release software that contains no defects in a timescale that everyone agrees on. This wont happen any time soon – but an increase of 15-35% development time for up to a 90% decrease in the number of defects has *got* to be worth it by anybody’s standards.

Have you used TDD (or BDD) and experienced similar results? Comments encouraged!

Enjoy!

How pages are cached

The Magento enterprise page cache module observes several events that are fired during the course of dispatching your controller and sending a full page in the response. I’m going to dive into only a few of them – the main observers that actually cause your content to end up in the page cache, and then exactly how Magento gets it back out again.

The first thing we’re going to look at is how Magento caches pages, and that starts with the observation of the ‘controller_action_predispatch’ event – triggered before a request is dispatched (but after we have determined we will not be fulfilling the request from the cache). It’s here that the PageCache module will determine if the request that’s about to be dispatched is to be cached for subsequent requests. There are many things that will prevent a page from being cached and we can start by looking at these.

Examine the processPreDispatch method in Enterprise_PageCache_Model_Observer. This is the event that is fired on the controller_action_predispatch event. All the handlers first check to ensure that the page cache is enabled before doing anything else. Then we examine the no cache cookie and set a flag (memorize_disabled) in the session.

$this->_processor->canProcessRequest($request) && $this->_processor->getRequestProcessor($request)

The first of the above two condiditions does some basic validation on the request. $this->_processor will normally be an instance of ‘Enterprise_PageCache_Model_Processor’ by this point (it’s instantiated by the ‘enterprise_pagecache/processor’ model here so can be overridden), and here we will find the canProcessRequest method. First, we dip into the isAllowed method where we check that we’ve generated a requestId during the construction of this object. The requestId will become important later on in the process as it provides the key into your cache storage backend for the page content and needs to be reliably recreated based on the request parameters. We then check that the request is not over https, the no cache cookie (NO_CACHE) is not present, the no_cache GET variable is not present and that full page caching is enabled. If any of these checks fail we do not process this request any further in the full page cache module.

Assuming we pass, we move onto checking the configuration. We inspect the page depth parameter (the number of variations a page can have based on the number of variables in the query string). If we’re over the limit, the page is not cached. Then we check the cache settings for pages that are not in the default currency (system/page_cache/multicurrency = 0 and the presence of the CURRENCY cookie in the request).

The second of the two checks above is where we examine how the page is going to be cached. This is configured in ‘frontend/cache/requests’ and the defaults are found in app/code/core/Enterprise/PageCache/etc/config.xml in the following section of XML. By default all cms, catalog/category and catalog/product pages have a processor configured.

<frontend>
    ...
    <cache>
        <requests>
            <cms>enterprise_pagecache/processor_default</cms>
            <catalog>
                <category>
                    <view>enterprise_pagecache/processor_category</view>
                </category>
            </catalog>
            <catalog>
                <product>
                    <view>enterprise_pagecache/processor_product</view>
                </product>
            </catalog>
        </requests>
    </cache>
</frontend>

getRequestProcessor is where this configuration is examined and we we use the request data to attempt to construct a processor for the target page. Assuming one is configured, Magento is now ready to cache the page for the next request.

If at this point it looks as though we’re going to cache the page, the independent block cache is disabled and the first peice of setting up the cache is complete.

The next time we’re back in the PageCache code is once the request has been handled and we’re about to send the response. This is accomplished by hooking into the controller_front_send_response_before event and executing the ‘cacheResponse’ method on the observer. We very quickly end up back in the processor and in the processRequestResponse method.

The first two calls will be familiar. The same methods (canProcessRequest and getRequestProcessor) are called again – however the processor for this request already exists so the second method returns immediately. We delegate to the configured processor (configured above) at this point and call the ‘allowCache’ method with the Request object as a parameter. A number of further checks are done which could prevent the request from being cached – the presence of ‘___store’ or ‘___from_store’ in the query string (for the default processor) or the presence of the no cache flag in the session data.

Finally we’re actually ready to cache the response. Some filtering occurs to strip out blocks and replace them with placeholders (blocks can be cached independently from the page – to allow for independant cache rules – like the navigation menu which can persist across multiple pages). The placeholder configuration is stored in app/code/core/Enterprise/PageCache/etc/cache.xml and is structured like the following (this is a real example for the catalog/navigation block).

<config>
    <placeholders>
        <catalog_navigation>
            <block>catalog/navigation</block>
            <name>catalog.topnav</name>
            <placeholder>CATALOG_NAVIGATION</placeholder>
            <container>Enterprise_PageCache_Model_Container_Catalognavigation</container>
            <cache_lifetime>86400</cache_lifetime>
        </catalog_navigation>
    </placeholders>
</config>

The above peice of XML declares the catalog/navigation block to be cached separately to any page that is appears on with CATALOG_NAVIGATION as the placeholder. It specifies the container class (used during storing and extracting the block) and the lifetime in seconds for the cached block.

Blocks are marked up at the time they are rendered and you can view them in the HTML source code between comment blocks that consist of the placeholder value defined in cache.xml, and a hash that is made up by the md5 sum of the results of calling the getCacheKey. They look something like the following (again, using CATALOG_NAVIGATION as an example).

<!--{CATALOG_NAVIGATION_a61df2dc9b9e5868f17a56461177d8c4}-->
<p>some html</p>
<!--/{CATALOG_NAVIGATION_a61df2dc9b9e5868f17a56461177d8c4}-->

This functionality gives you the ability to be able to cache blocks separately to the full page – with their own rules on how long they stay in the cache for (and independently of the page that they are on). This means that blocks can be made to be cached across pages rather than within them – meaning that blocks such as catalog/navigation need only be generated once – and served up from the cache for every request after that regardless of if the page itself is being served from the cache.

How pages are served up from the cache

Rather unlike how pages end up in the cache (using observers to watch for specific points in the dispatch process for a request), the method by which Magento retrieves information from the cache is somewhat more hard coded. Before dispatching the front controller in Mage_Core_Model_App::run (one of the very first methods that gets called in fulfilling a request) a check is done to see if the current request can be served up from the pages in the cache. This happens inside Mage_Core_Model_Cache::processRequest where we immediately check to see if there is a request processor configured that we can check the request against. The request processors are configured inside app/etc/enterprise.xml – the default one is set to Enterprise_PageCache_Model_Processor (predictably the same class used to cache the results of a dispatched request).

This is quite important – if you have a reason to extend the logic in Enterprise_PageCache_Model_Processor (like adding data that forms part of the cache id for a page), you not only have to extend it and rewrite ‘enterprise_pagecache/processor’, but also make sure that it’s added to this list of request processors so that Magento has a chance of being able to retrieve your items back out of the cache again.

Magento then loops through all the configured request processors, and calls extractContent on each of them. The first thing that this function does is create the cache id based on the request parameters (this happens when the processor is constructed, when the _createRequestIds method is called – as we saw earlier when the content was being cached in the first place). We check to see if any design changes have been cached for the current page, and then check the configured cache storage to see if we can retrieve the full page content for this request. If we do not get a match at this point, the request is considered a cache miss – and we return to the run method and start the process of dispatching the request (and potentially populating the cache with the results).

Assuming that we have a cache hit, we decompress it (content will be gzipped whenever possible in the cache storage to minimise space), and the content is processed by _processContent to replace all of the placeholders with separate cached blocks (ie, the catalog/navigation block that we cached separately when putting the page into the cache in the first place).

When writing about how pages are served up from the cache I referred to the configured cache storage. Prior to Magento EE 1.11, the full page cache simply used what you configured in the section in app/etc/local.xml. From 1.11 this has changed and you must now also have a section in your local.xml configuration (it uses exactly the same options as the config).

The aim of this post is to give an overview of how page caching works in magento and I am intending on following it up with some more detailed investigations into how more areas of cacheing work. If you have a specific request for more information, please do leave a comment and it may form part of a future blog post.

Chrome / Firefox – Pretty much all that I use Safari for (sorry, apple!)

Twitter – obvious reasons

Alfred – really great app launcher (and other stuff)

Caffeine – keep your mac awake for longer

Skype – Well, more out of necessity.

Dropbox – would have been lost without this!

Virtualbox / vagrant – dev tools.

Homebrew + git + vim – more dev tools…

GitX – Nice OS X native git gui.

1Password – Conundrum – what do you do when your password file is in dropbox, and your dropbox password is in 1Password? Use the iPhone app! Brilliant.

Moom – really cool window resizing management for OS X

Growl – part of OS X but deserves a mention for v1.3 in the app store.

iTerm2 – much nicer than the default Terminal.

I’ll leave it at that for now. Will update this if I remember anything else that I have missed!

Despite all this good stuff that you get for free, xhtml, or rather the xhtml media type (application/xhtml+xml) has its problems. And one of those problems is webkit.

Mobile devices are incredibly important in modern web development, so designing for the capabilities of those devices is just part of a standard workflow. The problem however, is that webkit on these devices prefer application/xml and application/xhtml over our friend, text/html. Take a look at the following Accept header sent by two desktop browsers (Firefox and Chrome):

Accept: text/html,application/xhtml+xml,application/xml;q=0.9, */*;q=0.8


Contrast this with what is sent from the iPhone (iOS 4.3.5):

Accept: application/xml,application/xhtml+xml,text/html;q=0.9, text/plain;q=0.8,image/png,*/*;q=0.5


And further, and Android phone (2.3.5):


Accept: text/xml, text/html, application/xhtml+xml, image/png, text/plain, */*;q=0.8


And suddenly we have a problem. If we use application/xhtml+xml then we have to account for iOS and Android devices preferring it over text/html.

There’s two things we can do to counter this. The first is to put a filter in front of your application to detect the user agents for browsers that are known to prefer the xml/xhtml representation that we want to reserve for our API clients. This is an application specific solution – you have to know that the browsers that you are filtering and then rewrite the Accept header so that text/html is served up by your application. You also have to accept that requests from those browsers will never be issuing requests against your web service as those requests will be rewritten to text/html.

The second solution is to use a vendor specific media type, and serve xhtml up on that. This means that you lose a small part of the self describing nature of using application/xhtml+xml – but means that you can be absolutely certain that no existing browser will prefer your API content over what it should be seeing. Vendor media types take the format of application/vnd.. – for example application/vnd.fdrop.xhtml+xml

Opinions on which of these two options are preferred (or if you have heard of other solutions or have any suggestions on other ways of dealing with this) would be most welcome.

The abstract is as follows;

REST and HATEOAS: A Case Study

A RESTful API is only truly RESTful if it uses hypermedia to tell us
about all the actions that can be performed on the curent resource,
allowing us to traverse the API from a single entry point. This
session looks at REST and HATEOAS (Hypermedia As The Engine Of
Application State) to illustrate good service structure.

We’ll use the RESTful file sharing service fdrop.it to illustrate the
various examples of how this can be used. This session is recommended
for architects and senior developers alike and will give a good
grounding in writing excellent, self-explanatory RESTful services.

Looking forward to seeing you there!