Decentralising the Social Network

Read in

Facebooks rise to the number two spot out in the list of the most popular websites in the world is proof that there is a huge demand out there for the social network. A way of being able to share snippets of your lives with friends and family in a way that they can consume as and when they want to. The social network has replaced emails and letters. It has fundamentally changed the way that we communicate with each other.

I have always been rather uneasy about a centralised social network. My issue with it is that those snippets of my life, the announcement of when I first became a dad, my daughters first day at school, when I became a dad again, all of those little bits of information – they build up a picture. A profile. A statistic. A thing that can be sold by the social network for real money, so that the purchasers can place an advert in front of me.

I don’t care much for privacy settings. Why would corporations who sell my data to other people ever be interested in supplying a free to use and completely private system? The centralised model cannot be trusted to keep any of your information private, or restricted to being viewed only be a small group of people. Everything that I share, I share with the knowledge that it could be seen by anyone.

Over the years there has been a number of attempts to get in on the social network space. Many have tried and failed to take on Facebook as the go to social network – but none have really been successful in being anything more than another centralised content distribution channel that uses Facebook to try and drive traffic in. Today, trying to take on Facebook as another centralised social network is a bit like treading on google’s lawn in the web search space (although I am encouraged by what DuckDuckGo are doing in that area).

Diaspora is one of the first attempts at the decentralising social content. Users register with a local hub (or with the right technical know how, can host their own ‘pod’). There is no central store of information. The pods are able to communicate with each other so there’s no centralisation of your data. The barrier to entry however is that it’s just not as slick as the Facebook experience (though if you’re interested, you can sign up to the UK pod here and check it out).

Metcalfe’s law states that the value of a telecommunications network is proportional to the square of the number of connected users of the system. There’s simply not enough people on it that would make it likely that I might find my own friends and family on there. Diaspora claim to have a million users (total). Facebook has an order of magnitude more, standing tall at 1.32 billion (monthly active as of June 2014). It’s clear that the network that Facebook built has considerable value.

A UK company, ind.ie is building heartbeat. A decentralised network as part of a much larger plan to free data from ownership by large corporations. As we’re on the wrong side of the beta launch of this new network, it remains to be seen what impact it is going to have.

The largest barrier I can see to the adoption of a decentralised social network is user experience. Facebook is so compelling because it’s so easy. Fill out a form, find your group of friends and you’re done. Decentralisation necessitates that you either sign up with a centralised point of presence (a ‘pod’ in Diaspora), or you pay to host your own server for yourself or your family. The barrier to entry is automatically higher.

As a life long fan of computers (I started coding on my Commodore 64 as soon as I was able), I have fond memories of spending some time at the local university learning to use their VAX/VMS system. It was some form of BSD (I don’t remember exactly) but I do remember that the ‘social network’ really just consisted of those who happened to be logged in to the same server as I was. We could chat – using IRC, or the write command. Leave messages using email and distribute content by setting up a website in our home directory.

This is why the tildeclub experiment appealed to me. Of course it’s not a social network. It’s just a low powered linux server that a bunch of geeks / nerds / enthusiasts are messing around with and sharing content with each other.

I would love to see the day where I had a slick experience like Facebook – but I had control over my own data. Being able to provision a social network for my family (and friends) to use, and being able to link my server up to others to form a true social network of computer servers. Free from being the product. Free from advertising.

Expanding Zoom

There’s been some recent discussion on the hal-discuss mailing list around the ‘Zoom’ concept for embedding resources, which has prompted me to start exploring this area a little further with a view to potentially writing up a specification that can be applied to any media type that define a way of embedding resources. This blog post mostly serves as a narrative to document my own ideas and learnings and to collect comments and recommendations from others in order to arrive at a proposal that will work for (almost!) everyone.

At its most simple level, clients need a way of being able to take an existing resource with defined link relations and zoom (or expand) the current resource to include them in the response. This is primarily to avoid a second round trip to the server in order to retrieve the resource representation that the client is interested in seeing.

To illustrate the problem, I have provided an example (for the sake of brevity, I have only included relevant parts of the resource representation).

{
    "_links": {
        "http://.../rels/detail": {
            "href": "/example/1/detail"
        },
        "http://.../rels/misc": {
            "href": "/example/1/misc"
        },
        "self": {
            "href": "/example/1"
        }
    }
}

In the above example, imagine a scenario where the client that has requested this representation is actually interested in the resource found at the ‘http://…/rels/detail’ link relation. The client now has to make a second request to the server in order to get at the data it needs.

We can alleviate this by using a feature of the media type known as embedding – however it is impractical to embed every linked resource every time. We must provide a way of allowing the client to specify the resource it is interested in – and this is ‘Zoom’.

If we allow the ‘zoom’ (or ‘z’) parameter in the query string to be reserved for this purpose, we can request that the server embeds the link relation by performing a GET request on ‘http://…/example/1?zoom=http://…/rels/detail’. The server would then be given the instruction to embed the resource and return it all in the response.

{
    "_embedded": {
        "http://.../rels/detail": [
            {
                "_links": {
                    "self": {
                        "href": "/example/1/detail"
                    }
                },
                "firstname": "Ben",
                "lastname": "Longden"
            }
        ]
    },
    "_links": {
        "http://.../rels/detail": {
            "href": "/example/1/detail"
        },
        "http://.../rels/misc": {
            "href": "/example/1/misc"
        },
        "self": {
            "href": "/example/1"
        }
    }
}

If we wanted to embed two items, the zoom query parameter may accept multiple link relations separated by commas, i.e., http://…/example/1?zoom=http://…/rels/detail,http://…/rels/misc.

The HAL format also supports CURIE style links to shorten the length of link relations and abbreviate the link down to a ‘x:detail’ or ‘x:misc’ format. There is no reason to not support these shorter syntax link rels, if the CURIE can be resolved. Here is an example of ‘http://…/example/1/?zoom=x:detail’.

{
    "_embedded": {
        "x:detail": [
            {
                "_links": {
                    "self": {
                        "href": "/example/1/detail"
                    }
                },
                "firstname": "Ben",
                "lastname": "Longden"
            }
        ]
    },
    "_links": {
        "curies": {
            "href": "http://.../rels/{rel}",
            "name": "x",
            "templated": true
        },
        "self": {
            "href": "/example/1"
        },
        "x:detail": {
            "href": "/example/1/detail"
        },
        "x:misc": {
            "href": "/example/1/misc"
        }
    }
}

The above examples and narrative cover zoom at it’s most basic implementation – and this will fit many situations fairly well. However, this idea of partially embedding sub resources is not new – what we are seeking is a unified way of following this pattern so that a common approach can be applied to any media type that has the ability to embed resources.

Implementations of this pattern can be found ‘in the wild’ in a number of places – to varying levels of completeness. From simply being able to embed a resource in a representation through to being able to specify a DSL to allow clients to query data that’s returned (specifying a limit to the number of resources in an embedded collection, or filtering by values in an embedded collection.

I’ll write up some thoughts around querying and limiting embedded data in a future post – i’d like to keep things relatively simple for now, and dive deeper into what the possibilities are later!

Comments more than welcome below.

Hypermedia types and connection negotiation

I was fortunate enough to be able to travel to RESTFest in Greenville SC over the past week to spend time mixing with people who are highly influential in the world of REST and Hypermedia. Many interesting things where discussed there and it got me thinking about connection negotiation a little more (one of many many things I have taken away from the conference – subjects for later articles!)

Before attending the conference I had always advocated taking an existing hypermedia type (for the examples here I will use application/hal+json) and trying to represent a resource using that. More often than not, I would require some affordances that are not defined in the spec – so extending it and creating my own vender specific mime type (which defines what base type it extends) made a lot of sense to me.

Now that various media types are become ubiquitous amongst API designers (Hal and collection+json amongst others), it’s made me see the advantage of sticking to the base type whilst negotiating with the server on what content you receive. The reason? Because other clients that understand these hypermedia types are staring to emerge – much like the web browsers in which we browse the text/html hypermedia type today.

Like HTML, hypermedia types are created with the understanding that if an element exists that the client is not able to assign meaning to (an affordance that is not part of the spec, or not implemented by the client), then it can simply ignore it as if it was never there. This is quite powerful – it allows you to define optional extensions to a hypermedia type whilst maintaining compatibility with clients and intermediaries which already understand your base type.

Collection+json supports this idea quite well, and the author (Mike Amundsen) accepts additional patches to define extensions as part of the specification – extensions that one day could form part of a revised version of the specification for everyone to use, but exist as extensions that can be understood by any third party client that support them.

It is now my opinion that the creation of custom mime types (without registering them with IANA) actually harm the interoperability of the web – and close down a hypermedia API to clients which have specifically been created for your own hypermedia type.

Perhaps a missing piece of the puzzle right now is a way of being able to declare which extensions your document is actually making use of (if this is even needed once the extensions become part of the specification). Using a Link header (http://www.w3.org/wiki/LinkHeader) is potentially a good way of achieving this, though exactly how this works in practice is still being considered.

Comments encouraged!

Writing RESTful clients

There’s plenty of articles out there on how to build the perfect REST API, so I am not going to go into that. This article is all about what to think about when you’re building the perfect client for a REST API. What you’re being served is clearly important – but how you use that information is 50% of the problem – and it needs to be done correctly.

API designers can spend a lot of time ensuring that their responses to you are perfectly well formed and self describing – but unless you pay attention to what it is that those messages are telling you, all that information is simply being thrown away.

Connection Negotiation

The very first thing your HTTP client will send to the server is the Request headers. Within that request, more often than not, you will send the Accept header and a list of media types that your client can support – and the order of preference within which you would like to see them. Chrome 20.0.1132.47 sends the following Accept header.

Accept:text/html,application/xhtml+xml,application/xml;q=0.9,/;q=0.8

Obviously this is a web browser, so it’s preference is going to be for text/html or application/xhtml+xml. Failing that, we can also take application/xml. If the server still can’t handle the request – then we’ll take anything at all (*/*).

The web server will make a decision based on what your client has asked for. From that point on one of 3 things can happen. 1) You get back one of the media formats that you asked for. 2) You receive a 406 response (Not Acceptable). 3) You get back something you didn’t ask for at all (this is allowed as per RFC2616 – the server SHOULD only respond with a 406).

This means that unless you’re explicitly checking what the response is after the request comes back – you could find yourself only assuming that the server is behaving as you think it is.

The Accept header becomes very useful when you separate out a normal response from an error response. In the situation where a server error is causing a problem, you are likely to receive a 5xx response code from the server with little or no meaningful body (text/plain). Where an application error has occured you should also receive a 5xx (or a 4xx) code from the server – however having declared you are able to handle a more specific media type (application/vnd.error+xml for example), you are able to present more information to the user than you would otherwise be able to.

Take the situation where you are consuming an API which serves up application/hal+xml and text/plain (hal for the resource representation, and a plain text for an error). The server is free to be able to add support for representing errors in application/vnd.error+xml without affecting any existing client – as long as the server continues to serve text/plain unless the new media type is deemed as acceptable to the client. Once the client is updated to add support for the new media type, it can take advantage of the additional information that can be provided.

Cache

A RESTful API will contain explicit directives that declare if and for how long the response body can be cached for. This information is contained across Cache-Control / Vary headers or Expires header. If the response to your request comes back from the server and declared itself as being able to be cached (only Cache-Control: no-store is excluded from user-agent caches) then you should store it locally and serve up the cached response as long as it’s valid.

There are of course already libraries you can use in your projects that will handle cacheing for you. If you usually use PHP for this stuff (I do), then I highly recommend you take a look at Guzzle to handle this for you.

URLs

Yeah i’m going to talk about it again. Hypertext as the engine of application state. Golden rule #1, never construct links in code (unless you’re following a URI Template) – follow them from the links given to you in the resource. This means that when these links change – you don’t have to worry about your app breaking. #2, use the defined link relations of the hypermedia type to navigate through the application (ie, HTML’s ‘rel’ attribute on the link tag). That’s about it. Don’t assume anything if it’s not defined by the media type rules and the representation you receive and you can’t go wrong.

JSON Sucks

Ok, so perhaps I was a little bit OTT with the title of this blog post (but it got your attention, right?). What I actually mean is that JSON Sucks for hypermedia APIs, and I am going to tell you why. But before I do, I just want to point out that actually I think JSON is awesome for what it’s designed for – serialising an object. Objects naturally contain data structures like ints, strings, arrays and other objects – if all you need to do is represent those four things, you can’t go wrong with JSON. Seriously.

Hating XML is rather fashionable these days, so many API’s are appearing that are JSON representation only even when XML really is a better fit for purpose. When you’re designing a hypermedia API you’re not simply serialising an object. There are media types that add semantic meaning to JSON (application/hal+json, my own application/vnd.error+json (draft) and others) but really, XML is so much more expressive that using JSON actually makes your clients have to work harder to understand the meaning you’re trying to convey using an object serialisation format over a markup language.

XML gives you some stuff for free – ‘xml:lang’ for one. Put this on any element of your document and just the fact that your document is XML means that you can express elements in different (human) languages without having to update your media type.

Consider how this actually looks in XML vs JSON representation.

<error id="6">
    <message>Something bad happened</message>
</error>
{
    "error": {
        "id": 6,
        "message": "Something bad happened"
    }
}

This looks easy, right? Consider a future version of the api where we add support for multiple languages.

<error id="6" xml:lang="en">
    <message>Something bad happened</message>
    <message xml:lang="de">Etwas schlimmes ist passiert</message>
</error>
{
    "id": 6,
    "messages": [
        { "lang": "en", "message": "Something bad happened" }
        { "lang": "de", "message": "Etwas schlimmes ist passiert" }
    ]
}

An alternative could be to key the message objects by language;

{
    "id": 6,
    "messages": { 
        "en": "Something bad happened",
        "de": "Etwas schlimmes ist passiert"
    }
}

You can see in both scenarios that “messages” has become an different type (string to array of objects, or string to object). We’ve changed the structure of the document and introduced an almost certain backwards compatibility (BC) break in our interface – all our clients now need to be updates to follow suit. We could leave “message” as it was, and add “messages” for the multilang version of the API but then we’re duplicating data and / or making the client have to work harder.

The XML version can continue as is and the client can simply ignore the additional languages until they’re updated to support them (if they ever need to).

Then there’s just the matter of basic human readability. It’s clear to see what’s going on in XML and not so much in JSON. This is a fairly simplistic example but extrapolate that out over a fairly complex representation of an object and you can see how JSON really isn’t always the best choice for representing resources.

Imagine the scenario where we had used the second option above as our JSON representation and we wanted to add in another attribute for each of the messages. See the problem? Anything we choose to do will cause a BC break and require clients to update (or result in duplication of elements to ‘version’ the api), meaning it’s very difficult to evolve the clients and servers independently.

So i’m not saying that JSON simply ‘sucks’. I’m just saying that when considering the formats you use for representing data in an API (when it comes to selecting the media type you want to use), think hard before just going with something based on JSON by default because it’s ‘easy’. It probably isn’t.

The error hypermedia type

When designing a complex hypermedia API you will inevitably come across the situation where instead of returning the required resource, you need to return an error state. Currently, there is no hypermedia type registered with IANA to deal with this scenario, so it is left up to the API designer to construct and document a mime type that can be used to express an error state in the application.

Because of this, several people have come together and put together a proposal for the application/vnd.error media type. It has been documented over on Github and the mime type has been submitted for approval by IANA.

The intention for this media type is that it should be used when a client has made a request, however the server is unable to fulfil the request to to either a server side error (a code 5xx) or a problem with the client request (a code 4xx).

Although the client does NOT have to state that it explicitly will accept the application/vnd.error+xml or json mime type (via the ‘Accept’ header in the request), it makes a lot of sense to do so if the client can understand that response type. However, RFC2616 section 10 states:

HTTP/1.1 servers are allowed to return responses which are not acceptable according to the accept headers sent in the request. In some cases, this may even be preferable to sending a 406 response. User agents are encouraged to inspect the headers of an incoming response to determine if it is acceptable.

It is up to the implementor to decide whether the api should return a 406 (Not Acceptable) response if the client states it will not accept application/vnd.error+xml or json and an error is triggere, or if it should return the application/vnd.error+xml or json response anyway.

The specification for application/vnd.error is draft – if you have comments / suggestions or enhancements to the media type, let’s talk (comments on this blog, issues on github, twitter etc).

PHP 5.4 web server

For me, one of the most exciting things that we have in PHP 5.4 is its built in web server. It makes being able to jump in the deep end with new ideas much easier and avoids the tedious work in setting up and needing apache to run before you have code to deploy. I have been playing around with the Silex framework in the last few weeks to build a prototype of a REST API that I have been thinking about. This is how I got going with Silex once I had PHP 5.4 installed.


wget http://silex.sensiolabs.org/get/silex.phar
php -S localhost:8080

Create an index.php in the current directory (there’s an example app on the Silex home page). That’s it. Go play. :)

Code review Karma

Implementing a good code review system into a development process is something that I have found quite difficult to do. The idea of it is conceptually quite simple – just make sure that before code is accepted into the main development stream that it’s reviewed by someone before being allowed to pass. Tools like github compliment git really well but force you down a development path that really suits open source software development, but not necessarily a co-located team.

My problem is that I trust my developers to do the right thing. Ok, so that’s not a problem – it’s actually a really nice place to be. I see code review as a learning exercise for good developers, but I don’t see is as a blocking part of the development cycle. Too many times I have seen code reviews as a blocking step in a software release cycle end up just being a formality to get through.

With that scene set i’ve been thinking of using a karma based code review system. Developers have karma (per project) and can give out positive and negative karma to other developers. At a certain level of karma, the number of reviews required on commits starts to drop off so that eventually a commit by a developer with high karma will not generate a code review (or the percentage of commits submitted for review will be very low). As a project progresses, hopefully this leads to a place where developers who will learn more from having reviews done will have the benefit, while the developers who are well established on the project are able to get code into the project repository without a blocking code review process. Improvement can be measured by karma increasing over time.

Do you have experience with code reviews on projects? (open source vs closed source / colo teams vs non-colo / blocking vs non blocking)… Interested to hear others success and failure stories!

So you think TDD isn’t worth the cost?

Think of the full software development life cycle. We gather requirements, create user stories, break them down into tasks, estimate, fight with your client about how long it’s all going to take, renegotiate the timings (what’s 35% of the development time on unit tests? Do we need those?), submit a revised timescale with cut down requirements and no time allocated to testing. Then we start development.

And soon the problems start coming up. At the end of the first sprint the client raises a few bugs with the software delivery… We didn’t account for those in the plan! And we can’t estimate them – we don’t know the size of the problem. That’s why it’s a bug… So we allocate some time in the next sprint to work on the top priority bugs and commit to developing fewer features.

The second sprint finishes, and more defects are raised in the software – some of them can be attributed to requirements not been defined clearly enough, but some of them are just code that we don’t understand how it even worked in development! Did someone actually test it before it went into the main development branch?! (how many people can honestly say they have never heard that one before…?)

Pretty soon you’re way behind the plan. The number of bugs that we cannot estimate and didn’t plan for are sucking up too much time and we don’t know how long it’s going to take to get back to a point where we know how long the project is going to take.

So what’s to be done? Bugs throughout the development life cycle are inevitable – but because we cannot know how many there will be and how long they will take to fix, we need to be able to minimise the number that crop up.

Some research done by microsoft (http://research.microsoft.com/en-us/groups/ese/nagappan_tdd.pdf) has shown that spending 15-35% extra development time on adopting test driven development resulted in a pre-release defect density decrease of 40-90%.

The ideal is when we can plan, develop and release software that contains no defects in a timescale that everyone agrees on. This wont happen any time soon – but an increase of 15-35% development time for up to a 90% decrease in the number of defects has *got* to be worth it by anybody’s standards.

Have you used TDD (or BDD) and experienced similar results? Comments encouraged!