Whypermedia: Why use Hypermedia?

The question of whether hypermedia (sometimes referred to as the REST hypermedia constraint) is a worthwhile foundation for API design is debated quite often. One of the frequent counter-arguments is that designers and developers are not that familiar with hypermedia, and thus the costs of designing and using it outweigh the advantages. Another argument often heard is that when looking at a single API, the advantages are not that big.

This post is an attempt to provide some evidence that the potential for hypermedia designs to be better than non-hypermedia ones is substantial. The claim is that often some of the potential gets overlooked because designers only look at the benefits of producing one design, and producing it in isolation. This misses two key points of how and where most APIs exist today:

• Service Landscapes: Very few APIs exist in isolation. In fact, the current trend in API design and management, exemplified by microservices, is to plan for an API landscape with many small and well-separated services. The claim is that such a landscape provides a number of advantages over a more monolithic design with bigger individual APIs. As a result, API strategies and designs should start from the assumption that APIs and API consumers exist in settings where typically one application consumes more than just one API.
• Service Evolution: Few services are created and then remain static. In most cases, services evolve, and in a loosely coupled service landscape, ideally they should evolve in a way that does not break clients. Therefore, services should have a plan how they can evolve robustly, so that changes of the API do not break clients. This aspect may not be obvious in initial design phases, but will make a substantial difference later on; how much exactly depends on the pattern in which services and clients exist.

Combining these two aspects, the claim is that hypermedia particularly shines when it is being used in evolving service landscapes. Combining landscapes and evolution and factoring in the trend towards smaller and more frequently changing service designs, it becomes clear that a pattern that can handle these two aspects well will provide substantial benefits for application developers in such a setting. Even if they use numerous services and even if these evolve over time, there will be a better chance for applications to be robust and survive changes in the evolving service landscape.

The following sections explain a variety of advantages of hypermedia APIs, and how they specifically shine in an environment of many and continuously evolving services. Whether all or some of these issues matter (enough) for the API landscape in a particular setting is still something to decide for the architects and managers of that landscape. However, taking the perspectives of diversity and evolution into account, it should become clear that hypermedia as an API design strategy has some distinct advantages over other design approaches.

Since the Web (celebrating its 25th birthday today!) is the largest, best known, and most successful example of a hypermedia-driven service system, this post will explain the various advantages of using hypermedia with Web examples.

Self-Descriptive Control Flows

Hypermedia embeds controls within the service data, meaning that by looking at any resource received from a service, information about possible next steps is part of the representation. This makes it easy for applications to understand the control flow, because instead of having to rely on information that is external to the actual service interactions, all the information required to proceed is contained in the response to a request.

Since hypermedia links are supposed to be global in scope, not only do those controls expose possible next steps for one service, but they can equally easily represent a transition to a different service. In fact, from the client perspective this not only does not make any difference, but a client in fact is not even able to tell when it is crossing "service boundaries". From the client's perspective, it is simply following self-descriptive control flows to accomplish application goals.

Web pages contain links that guide users through designed control flows. Web designers choose which controls to include, and expose these as links that users can choose to follow. Users have the freedom to choose one of these links, or to leave the designed workflow at any time and continue elsewhere by navigating to an address of their choosing. The designed control flow of a Web page can be a link on the same site or server, or one to a different one: this is not something that users have to care about or what they can even reliably find out, even if they look at the URI and try to conclude whether this is the same or a different server/service.

Flexibility

By having self-descriptive control flows, an added advantage is that control flows can change on a per-instance basis. Since controls are part of the representation, the representation can treat them like any other part of a data model and make them mandatory, optional, or repeatable. Clients have to be prepared for this variability, and then can learn about the available control flow options on the fly, as they inspect a representation that they have received and prepare to choose a next step in the workflow.

Web pages can present links for next steps in a control flow depending on resource state. For example, a page representing an order can contain a link to cancel the order only if a particular customer has that privilege. That control may also be time-dependent, disappearing when the order has been shipped, at which point it cannot be cancelled anymore. Clients can always interact with resources at any point in time and find out the available controls at that point in time, which allows them to provide a more useful and usable interface to users as if they always presented the full set of theoretically available options, most of which may not be available at any given point in time.

Extensibility

Taking self-descriptive control flows and flexibility one step further, it is also possible for representations to provide extensibility for new kinds of interactions in the control flow. If representations are designed in a way that these extensions are themselves self-describing, then this provides a robust way of how control flows can evolve, without the need for all clients to be updated when that happens. Instead, clients unaware of these extensions will silently ignore these new options unknown to them, and will be able to continue operating based on the controls that they know about. New clients, on the other hand, will be able to take advantage of the new possibilities, even though when interacting with old services they may never encounter them. (More information about the various angles of extensibility in this post about hypermedia advantages.)

Web pages can present new options after an update to a site, but users are free to ignore them if they don't need and understand these new options. Web pages should never remove options that were available before because that may disrupt exiting users. Some sites even go as far as implementing a "classic view" where users can see a simpler view of a site as it used to be, whereas the "modern view" might present new options that classic users do not need. This difference in views is nothing but a client's view to either ignore a new set of controls that aren't used, or to take advantage of them as they might provide useful new features of the service.

Separate but Connected

Hypermedia is a concept across individual APIs, linking them as far as the supported identifier schemes allows links to go. Specifically, consumers of hypermedia services do not have to be aware of the implementation boundaries of individual services, as the links between them provide seamless interactions across various APIs. From the client side of view, the fact that accomplishing application goals takes more than one API does not even matter; the only thing that matters is that the client is able to navigate a set of interlinked resources with the application goal as the driving force.

The main user experience of the Web is that of one interconnected set of resources, with clients seamlessly navigating across all the various providers of Web pages. From the user perspective, it does not matter and is not even clear which individual sites/services are traversed when going through a Web-based workflow; all that matters is that the user's goal can be accomplished by following links that eventually will result in the user achieving their goals. This is one of the main contributing factors of the Web's access: as long as users have a starting point and can follow links to accomplish their goals, the specific sites/services they use do not matter and their composition can even change over various invocations of the same control flow.

Statelessness

The REST hypermedia constraint not just requires hypermedia links, but also requires interactions to be stateless. This means that all information required to process a request needs to be contained in the request itself. This constraint makes hypermedia much more robust, because it means that requests can be scattered across servers/services without the need for those requests to be associated through shared data on the server/service side. It is this additional constraint that allows hypermedia to be truly decentralized, because now there is no implicit assumption that all requests have to be processed by a single server/service that uses state information to handle the workflow across requests.

Having no state on the server/service side also means that clients can continue at their own pace, instead of having to make sure that it does not trigger time-outs on the server/service side. Because individual requests are disassociated from each other, servers/services have no way of telling if/when a client paused and then resumed. In addition, this design also allows clients to pass state information between each other, allowing workflows to be completed across clients if clients decide to cooperate and share client state, so that different steps in the workflow are completed by different clients.

Web pages are complete representations of client state, kept in the browser. Users can put their computer to sleep and go on vacation, and when they come back, ideally they are able to simply continue where they left off. Since all state was captured in the browser, there was no need for a server/service to remember anything, and thus no time-out was ever encountered. Web sites encoding all of the (relevant) state in a URI (often in the query string part) make it easy for users to share URIs: Pass along the URI, and another user will be able to continue the workflow from that point on. This way, it is possible for clients to share a workflow, and once again the server/service does not need to know or care, because the request/response interactions are the same, regardless of whether they are with just one client or more than one.

Comments

[Tim]

Thank you for the blog post, I'm starting to understand the benefits of hypermedia :)
I do have one question. The blog mentions that clients are unaware of service boundaries if they follow a link of a next step. But as I understand these next steps are represented by a link with rel and href.
From a micro service approach, if a response from service 1 contains a link to service 2 does this not create coupling between the two services?

[dret]

thanks for your comment, @tim. and yes, you're right that a link between two services establishes some kind of coupling. that's because when you follow the link, you now interact with the linked resource. you could classify this kind of "coupling" as "choreography", where interactions lead you step by step. that would be different from more classical "orchestration" approaches, where you have a single grand plan and that's what you follow (if you're interested in the different ways in which you can look at "coupling", http://dret.net/netdret/docs/wilde-www2009-loosely-coupled.pdf is a paper cesare pautasso and i wrote a while ago).
the neat thing about REST hypermedia is that it makes "decentralized choreography" a natural part of the design and interaction model. and because it is stateless, meaning that each request is completely self-contained, you are able to choose between steps if there is more than one available.
let's idealize a bit here: let's assume you have shopped and now you have three options to pay. the shopping service is "coupled" to these three payment services in the sense that you probably will pick one of these three, based on your preference.
in a perfect RESTful world, though, your shopping service would expose a "user x with shopping cart y needs to pay z dollars" resource, and you could even pick a non-affiliated payment service, as long as that service would be designed to work with these "i am x and have order y for z dollars to pay" representations, and your shopping service would expose this kind of resource and also provide a payment protocol for that other service to make the payment.
now that may be a bit of a stretch here, in particular because payment is such a sensitive area, but maybe it highlights a bit what the hypermedia coupling does, but also where well-designed hypermedia APIs allow clients to opt out of them, when clients don't want to use the provided hypermedia controls.

[Peter Rushforth]

Nice post, thanks! I think it's worth mentioning too the emphasis on hypermedia format that REST implies, which I (often) find sorely lacking among REST advocates' discussions. You do properly allude to it in the post (e.g. 'self-descriptive') , but it is often overlooked and sometimes misrepresented. For example I've often read that a "good" REST API does not need documentation - a statement which I take issue with, since the specification/documentation of the _format_ (s) is definitely part of the "in band" information that one needs to write a client for such an API.

Also, I am not sure that statelessness implies that the server will still be in the same state after your vacation. I think it means that one need not have performed some session initiation per the protocol used, but I would like to learn more about that aspect anyway.

All the best

[dret]

thanks for your comment, @peter! i absolutely agree that "self-describing" does not imply "no documentation". and that's not what REST means by that term. it simply means that you don't need out-of-band information to act on a message. when it comes to documentation, which one of the basic Web Concepts (http://webconcepts.info/concepts/) you are using already is a very good starting point. but then you still need to document your media types and any non-standard concepts you're using. the API documentation space still has a long way to go!
about the vacation example: your *client state* goes on vacation, too, but not the *resource state*. let's assume you browsed a library and asked about the available copies of a book. your result showed three books, and then you went on vacation. when you come back, maybe not all of those three books are still available. but your client state assumes they are. so you click on "reserve", and then the server responds with "this book is no longer is available". this is why REST clients have to be able to deal with failure *at every single step*, because this scenario can happen anytime.

[Jan]

Very valuable post. Thanks.
Currently I have to do some decisions about REST services versioning. There are many post about this, BUT I still do not know what to do.
It seems that there are at least 2 camps:
URL versioning versus HTTP Header Versioning (content negotiations, etc.).
I tend to key URL's stable over the time, BUT I am really not sure, especially in the context of large-scale integration.
Could you please elaborate on the REST versioning topic?

[dret]

thanks for your comment, @jan! me and my colleagues at the API Academy often joke that we should have a one-slide workshop on API versioning with the one slide saying: "DON'T!"
of course that's not terribly helpful, but it really is what you should be striving for. as a follow-up to http://dret.typepad.com/dretblog/2016/04/robust-extensibility.html i am planning on writing in more detail on how to design APIs this way. the key point really is to design in a way that you don't need to version; all you do is evolve, and there's no need for version labeling at all. if you do reach the point where you make non-backwards-compatible changes, you start breaking the ecosystem and there's no need to call this "versioning" anymore: you're creating a new thing and the main/only reason why you might want to keep the name is branding.
for now, i highly recommend to read mark's https://www.mnot.net/blog/2012/12/04/api-evolution which touches on all these points, and also highlights the fact how using web architecture makes you think about versioning in a very different way.

[Jan]

Hi dret
Many thanks for your reply on my comment.
In the meantime I 've read a lot of stuff (books & links) about REST versioning including those you are pointing to.

After all it looks like we should use the term 'REST versioning' in the sense of Semantic Versioning MAJOR version only, i.e. we should not talk about REST Versioning unless your API is moving to the nexr major version, right?

In this sense if you say DON'T (version REST URI) are you stating: Stay with your stable URL as long as you can, and only version URL if you have to brake the backward compatibility of the service?

In my current working context we have the following integration challenges (as many other big companies might have):

1) REST clients/consumers can be 'machine', i.e. another service, OR web browser user, OR native application (e.g. iOS) user.
In this case how do you handle e.g. mandatory input fields changes? And these changes MUST be validated in some backend layer?

2) Incremental support (providing semantically new version of a service) for e.g. pilot user group ?

3) Support for the user groups in different regions not all at the same time?

4) Major service change as incremental rollout?, i.e. no big bang deployment!

Could you please elaborate little bit on each of these topics/questions?