dretblog

Linked Data and REST: A Match Made in W3C

the W3C announced yesterday that the Linked Data Platform (LDP) Working Group has been officially launched. in case you're interested, the proposed charter is a great starting point to learn more. i am happy and proud to be listed as a proposed co-chair of the WG (together with IBM's arnaud le hors). i think this activity will be a pivotal point in the development of linked data, and interestingly, the driving force behind the working group has been the fact that linked data is probably considerably more interesting as a BI-like data integration platform than it is as an AI-like ontology/inference technology, and it's great to see that such a pragmatic view now has become the driving factor behind something as substantial as a W3C working group.

personally, what makes me most hopeful for the future of linked data on a much larger scale than so far is the very open attitude towards REST as expressed in the charter. this means that REST with its focus on encapsulating interaction semantics in media types now is readily available as a toolkit on which linked data can build. over are the days where every last bit in a linked data scenario had to be RDF, and if it were not, then things could not be linked data. this open attitude means that linked data now can grow much faster, can be more inclusive in its choice of components, and does not have to reinvent the wheel for every little piece of machinery that has to be put in place. most importantly, real-world developers will be able to ease into linked data comfortably, first finding a feed providing interesting data and useful services around it (updates, notification services, push services, ...), then happily using its entry-level metadata, and once they decide they really have found something of interest and value, they will encounter RDF when they dive into the actual payload.

linked data is a data model, and REST is an interaction model, and as such they are two nicely complementary parts. while it will be important to make sure that the data model itself is maintained at the payload level, many of the issues listed in the charter can be readily addressed by choosing existing and well-established standardized components (3/4/5/9/10/11 likely can be solved by using a combination of RFCs 4287/5005/5023 and some best practices how to use them). however, other issues may go right to the heart of RDF (for example the question of how to handle RDF granularity so that RDF documents become part of RDF's model). whatever exactly we will be able to address and achieve in the group, i am really looking forward to pragmatically combine linked data and REST, and i hope i will be part of an activity that will take linked data from being the 1% of how people handle data integration and big data, to being the 99%.

Describing Resources, the Web Way

there has been a long and often very passionate discussion around the question of how the linked data community can distinguish between identifiers that are referring to non-information resources, and those that refer to information resources. this distinction in itself is something created by the linked data community, and is a direct result of the design goal to be able to serve descriptions at the very same URIs that identify the described things. this design approach (trying to associate two different things with the same URI) has caused quite a bit of pain, and this resulted in a variety of attempts to handle it in a well-defined way, the last being the famed httpRange-14 resolution based on HTTP.

my biggest issues with both the (older) hash URI convention and the 303 redirect convention is that they unnecessarily interfere with rather basic fundamentals of web architecture, instead of simply using the mechanism that any other web-based application establishing associations between resources is using: establishing links. if there are two different things that need to be associated (the described resource and the description resource), then why not just link them, allowing clients to simply discover those links, and then act accordingly. if these are proper links, then there are three different ways how they can be used:

1. the links can be embedded in representations that support embedded links, and thus clients consuming those representations can find links from a described resource to a description and back (if both support embedded links).
2. if representations do not support embedded links, but the association is known to and/or managed by the server responding to the request, an HTTP Link header field can be used to convey the association. this is particularly convenient for representations not supporting embedded links, such as image formats.
3. since associations can also be treated as first-class citizens, it is possible to have a third-party service that associates described resources and description resources. this means of course that there also can be alternative/competing and even conflicting services claiming to provide those links, but that's just how the web is: decentralized and possibly containing conflicting information.

while there is a describedby link relation type that has been registered in the IANA link relation type registry by POWDER for quite a while, this was not even considered as a possible way to handle the linked data problem. so maybe the lack of the inverse described was the problem, because it certainly would be more useful to be able to represent both directions of the same association? for this reason, there now is an I-D for registering The 'describes' Link Relation Type. here's the abstract:

This specification defines the 'describes' link relation type that allows resource representations to indicate that they are describing another resource. In contexts where applications want to associate described resources and description resources, and want to build services based on these associations, the 'describes' link relation type provides the opposite direction of the 'describedby' link relation type, which already is a registered link relation type.

apart from the fact that this approach is more in line with existing web architecture, simply registering a new link relation, an added benefit is that it allows to layer descriptions, so that descriptions can in turn be described by other resources.

any feedback about this new link relation type is greatly appreciated, in particular any feedback about which linked data scenarios may not be appropriately supported by such an approach.

PATCHing AtomPub

the Atom Publishing Protocol (AtomPub) predates HTTP PATCH, and therefore there is no way how it could explicitly allow updates to be made via HTTP PATCH instead of PUT. but instead of allowing any update method, both the edit and the edit-media link relation type hardcode the fact that updates are made by using PUT. this means that conforming implementations have to support PUT (and maybe they can support PATCH; supporting other methods is neither explicitly allowed nor disallowed by the spec).

one of the advantages of PATCH is that updates can use any kind of diff format that is deemed appropriate for the application, and thus (in case or larger resources, for examples) it may be desirable to use the general patterns established by AtomPub, but to allow updates by PATCH, maybe in addition to PUT, or as the only allowed method for updates. but what would it take to PATCH-enable AtomPub? there are (at least) three possible answers, and i am curious to see whether there is some community consensus for the best one.

1. do nothing: maybe the interpretation that updates must be made by using PUT is overly narrow and the fact that neither edit nor edit-media mention the required method allows us to stretch the interpretation of AtomPub a little and assume it is ok to also or only allow PATCH? (i guess my quotation marks make it clear that i don't think this would be the proper way to do it.)
2. add new links: it would be possible to define patch and patch-media link relations, which would be explicitly defined to use be used with PATCH. it would be a bit unfortunate to have two links instead of just having one edit link that could be used with OPTIONS to learn about the available methods, but one could argue that the fault lies with AtomPub's hardcoded PUT, and thus this has to be done to not mess with AtomPub semantics.
3. create an AtomPub profile: another approach would be to allow OPTIONS on edit and edit-media and thus allow clients to explore the available methods. this could be signaled by a profile link in the feed or entry, and clients supporting that profile would know that they could use OPTIONS. clients not knowing this profile would simply have to deal with the fact that some edit/edit-media links might return 405 (Method Not Allowed) errors, but REST clients should be able to deal with this kind of failure scenario anyway.

any opinions or comments about these three options are highly welcome, and in particular any pointers to implementations that use AtomPub and already support PATCH. if there is some community consensus, it would be interesting to see whether it would be strong enough to move forward with option 2 or 3 and turn it into a formal spec (and if everybody is fine with option 1, then no action is required anyway).

REST Layers and REST Applications

there has been a rather long history of people trying to describe REST applications, and an equally long history of responses saying that it cannot be done, should not be done, and that REST is allow about following your nose across a web of interlinked resources. many approaches have used some kind of RDF-based model, and have either tried to model/describe an application in terms of SemWeb principles, or in some cases the standards (such as HTTP) themselves. however, so far there seems to be no consensus on whether it could/should be done, and given the fact that we expect RESTful APIs to become more popular, it would be good if the REST community could provide some guidance on how those APIs should be described/documented.

in an attempt to better understand what and why people are trying to accomplish when they try to describe REST, here is a first approach to more cleanly layer the issues, see what's already around, and try to figure out what value could provided by some format/technology for each layer. we are actively working in this area because we simply need something that will allow service providers to describe (and by that we mostly mean document) their services. our platform allow customers to define and expose REST services, and when they do this and want to point, for example, external partners at these services asking them to develop clients, instructing them to tell your external partners to follow their nose just doesn't quite cut it.

in this first attempt at a layering model, there are five layers:

1. architectural style: this is where REST itself is defined, independent of any technology. this is where the architectural principles of the style are defined, and fielding' thesis by definition is the one and only authoritative source in this layer.
2. technology framework: on top of REST's principles, web architecture encompasses a set of enabling mechanisms, most importantly URI for identification, HTTP for interaction, and media types for labeling representations (and it is interesting to notice that the core technologies on this layer are all defined by IETF instead of the W3C). while this layer is essential for the fabric of the web, in itself it does not yet define a working system, because it lacks concrete representations that can be used as transfer formats.
3. core technologies: for the human web, HTML is the most important media type (but without more media types such as images, the web never have succeeded), and for machines XML and RDF have become important enabling technologies. it is important to notice that all of these media type are application-agnostic and not application-specific formats. clients need to support those media types to work properly, but without additional guidance (provided by human operators or additional information), they don't know what they are doing and blindly interact with a set of semantically shallow resources. core technologies allow clients to work in a RESTful way, but they may not allow servers to expose the richness of the services they are providing.
4. added semantics: (almost) all of the web's core technologies have some mechanism to add semantics. sometimes these mechanisms have formalized representations (such as XML's DTDs or XSD), sometimes not (HTML profiles are not described in any standardized way). some added semantics are metamodels (XML schema languages allow authors to define their own schemas), some are fixed models (podcasts define a fixed set of extensions that can be found in feeds). most importantly, the core technologies provide ways how these added semantics can be signaled to clients, for XML through namespaces and/or associated schemas, and for HTML through profile parameters (and the proposed 'profile' link relation type might be able to unify this mechanism across media types). added semantics allow servers and clients to support semantic overlays that allow richer and more semantics-driven forms of interaction.
5. applications: based on core technologies and possibly added semantics, developers build applications, which expose the resources of a given service in some URI space. these applications can use any number of media types and any set of added semantics on top of these, and in some cases developers will define their own media types, whereas in others they will just reuse existing ones. in both cases, the application is seamlessly embedded into the bigger context of the RESTful system, and while that may be not interesting at all for some clients, and might as well be interesting for others. for the HTML web, google's sitemaps format has become an established representation in this area because it allows servers and certain clients (crawlers) to interact in ways that are advantageous for both sides.

based on some of the discussions that happened around describing REST, it seems that at times, there was a mismatch in where in these layers people wanted to be. when it comes to layers 3 and 4, documentation/description should be the media type or semantic extension itself. for these cases, all that a client should need to know is the processing and operational model of the resource, and then it can act. however, when it comes to layer 5, i am a firm believer that there are scenarios where a description format would be very desirable, and the sitemaps format is one (very widely deployed) example.

in our work on the Resource Linking Language (ReLL) we have explored this area and made some design choices that i would now make differently. but generally speaking, giving application providers the possibility to describe the extent of their application and doing that in a way which itself is a self-describing resource seems like a very RESTful approach. if a client is interested in this information, there is a discovery mechanism (for sitemaps a slightly awkward robots.txt-based rule that hardcodes an URI-based approach) and then the client can happily use that information, and could for example inform the user that he is now leaving the application (which may or may not be indicated by a change in the some URI prefix). if clients are not interested in the notion of an application at all (like regular users happily surfing the web without ever experiencing application boundaries), they simply ignore this information.

starting from this idea of REST layers, i'd be curious to get some feedback about the perceived usefulness of a description language that specifically just resides in layer 5. the main goal is to establish a format that could be seen as sitemaps for REST, and definitely should be less chatty than exhaustively listing all URIs. this would allow service providers to describe the universe of resources they are exposing, and would give clients a snapshot view of the current expectations they can have about exposed resources, and the assumptions about where the scope of the applications ends. in a follow-up to this post, i will outline our scenarios, and write more about the things we would like to include and exclude in such a REST Application Description Language.

Resource Profiles

one of the not-so-well-known features of HTML is the ability to specify a profile attribute for a web page. it goes right in the head element like this: <head profile=" some URI ">. here is what the HTML spec says about this attribute:

This attribute specifies the location of one or more meta data profiles, separated by white space. For future extensions, user agents should consider the value to be a list even though this specification only considers the first URI to be significant. Profiles are discussed below in the section on meta data.

The profile attribute of the HEAD specifies the location of a meta data profile. The value of the profile attribute is a URI. User agents may use this URI in two ways:

• As a globally unique name. User agents may be able to recognize the name (without actually retrieving the profile) and perform some activity based on known conventions for that profile. For instance, search engines could provide an interface for searching through catalogs of HTML documents, where these documents all use the same profile for representing catalog entries.
• As a link. User agents may dereference the URI and perform some activity based on the actual definitions within the profile (e.g., authorize the usage of the profile within the current HTML document). This specification does not define formats for profiles.

the HTML profile attribute is defined so that it may contain white-space separated URIs, each of them identifying one profile. this is a bit unconventional, but the overall design goal is clear and reasonable: why restrict a web page to only conform to one profile? in practice, though, the profile attribute is hardly ever used, and if people know of any deployments where it is actually being used by clients, please make sure to point to this in a comment!

in our work on RESTful services, we have repeatedly run into the situation where we want to have a runtime mechanism how a resource can represent the fact that it follows additional constraints and/or rules. creating media types at runtime is not such a great idea, and it would also not allow us to represent the fact that a resource is still using a representation based on some media type, but also can be treated according to additional constraints and/or rules. a profile mechanism would be a good way of doing this, and thus we would like it to become a recognized link relation type.

seeing that profiles are a good idea, but should not be restricted to HTML and thus be represented in a more generic way, microformats.org published a proposal for a profile relation type a while ago. this is a good initiative, but there are two downsides to the current microformat proposal:

• it is not intended to register the profile link relation type in the IANA link relation type registry as defined by RFC5988. it would be useful to have this link relation type standardized at the web level.
• the proposal makes assumptions about the linked resources and the processing model for them. instead, it should be left to media types (the ones using/supporting profiles and possibly profile media types themselves) to makes these assumptions, so that the link relation is decoupled from media type specifics.

this means that it might be a good idea to push forward with an I-D proposal to standardize a profile link relation type. This relation type would state that such a link serves as an identifier that a resource representation follows additional constraints or rules, so that clients processing it could take those into account. whether the link is dereferencable at all and if so, what kind of resource it would point to, would be left open. this blog post is soliciting feedback about this idea, and any yes, this would be a good idea, please make it happen! or no, this is a bad idea and here is why comment is welcome. depending on the feedback, i'll continue with my work to publish an initial draft, which should be out in the next week or two.

[[ update 2012-04-15: there now is a draft out, draft-wilde-profile-link is available through the IETF site (and the IETF datatracker), and as of today has been updated to version 01. ]]