Link Discovery for XML

one of the main principles of REST is what unfortunately most often is called Hypermedia as the Engine of Application State (HATEOAS), which must be the worst acronym ever invented. regardless of that, this is one of the central REST constraints, and also one that is often ignored, with services using no discoverable links at all, or using URI templates and assuming that this already is RESTful. it can be, but it often isn't, because even if there was a stable standard for URI templates (and there isn't), it still would be impossible to know how to substitute the template variables to get URIs of available resources. which is why RESTful services should include links in resources, so that applications can simply follow those links.

in web architecture, REST's links are found by knowing a resource's media type, and knowing how to find the links in that format. the most popular linked formats on the web probably are (X)HTML and Atom, both of them define links and link semantics in their data format. however, since XML is such an popular syntax for many different content types, i have been toying with the idea of defining a generic XML mechanism for discovering links in XML (the only generic descriptions for XML available today are the built-in xml:lang, and the separate xml:id specification). notice that this is different from datatype mechanisms such as XSD, which are able to type content as a URI, but it is still not clear whether this URI is a link, and if so, what its semantics are (which i will call role from now on). as a starting point, here is a brief list of requirements:

• link discovery should work inline (information embedded in XML instances) and out-of-line (as an external description of how to find links in some XML).
• link discovery can be a perspective of a media type, i.e. different applications might have different ideas of how to find links in a given media type, or a subset of documents of a given media type.
• links can have roles, and are allowed to have more than one role. role names can be strings, qualified names, or URIs.
• the mechanism should rely on the smallest possible set of technologies, so that it can be easily implemented using widely deployed tools and technologies.
• there should be a simple syntax for fully describing the link discovery information, so that it can be serialized and exchanged.

starting from this simple and very likely incomplete set of requirements, here is my first stab at a mechanism for Link Discovery in XML (LDX). the main idea is that links are identified by XPath-based selectors (XSLT's pattern subset might be a good match here), that selected links are associated with roles, and that this mechanism can be applied to any XML-based media type. what would be the advantage of such a mechanism? a simple way to describe the links and link semantics of XML media types, the ability to build generic tools that can extract links from any XML-based media type, and the ability to better separate general REST functionality (what links can i follow in this resource?) from content-specific functionality (what does the content mean, and what does it mean to follow the links?). generally, such a mechanism could become part of a REST toolbox, and there are advantages for both service providers and consumers.

now on to the specifics. it's a bit harder to come up with a good inline format (HLink still might be remembered by some), so my first pre-alpha idea focuses on the out-of-line format. it's a simple set of selectors associated with 0-n roles. each selector selects an attribute or element content (interpreted as the string value), and this content is then recognized as a link with the associated roles. selectors probably should be able to be grouped and contextualized, so that for example some minimal LDX for XHTML might look like this:

<links xmlns:html="http://www.w3.org/1999/xhtml">
 <context match="html:html/html:head">
  <link match="html:link[@rel='stylesheet']/@href" role="stylesheet"/>
 </context>
 <context match="html:html/html:body">
  <link match=".//html:img/@src" role="image"/>
 </context>
</links>

this would allow to automatically extract all links to stylesheets and images from XHTML documents. the role names in this case are just strings, and there definitely has to be some better handling of non-string role names, in particular differentiating qualified names and URIs (which could, at least in theory, use CURIE syntax) might become a bit of a challenge.

namespaces (in the described media types) are a headache as usual, the eternal question being what to do with unprefixed names in selectors. use the LDX document's default namespace (à la XSD), or have a special attribute for setting the default namespace (à la XSLT)? i prefer the latter, but that's more syntax juggling than anything else.

the fun part is that because of the selectors, applications can even try to be smart and create LDX that works for the specific content they're concerned with. let's imagine a RESTful application using XHTML which contains links, and always contains a link to a product in the first column of each row of a specific table. using a selector such as table[@id='products']/tr/td[1]/a/@href, it would be possible to associate these links with a specific role, and with LDX support in a REST toolbox, finding all those links would be trivial. and the LDX description of the assumptions made about the content would be concise and declarative, so that it would be easy to understand the linking assumptions of the application.

most of LDX currently exists in my head only, and right now i am mostly wondering what others think of that idea. does this make sense from a RESTful point of view? would it be a worthwhile addition to a REST toolbox? maybe even to the REST landscape in general? and, assuming it does make some sense and i am tempted to start writing down a bit more of my ideas, should i start with a W3C or with an IETF template?

Comments

[Stefan Tilkov]

I see why you introduce this extra level of indirection, but I wonder whether it would not be easier to define a standardized link element syntax that can be used everywhere, and then map other vocabularies onto this as needed.

[dret]

@stilkov, thanks for the comment. there are three main reasons why i think a discovery/description mechanism might be a better way to go than some reusable schema module:

- for media type designers: it's hard to predict what designers want to do. , for example, has three links (@src, @usemap, and @longdesc), and designing a schema module that would allow this kind of thing is possible (for example using XSD types), but might make things more complicated than intended.

- for media type publishers: links are one of the major constraints of REST, and such a language would provide a declarative and easy-to-use way of how to document that facet of a RESTful scenario.

- for media type users: decoupling the media type and the way how to discover links in it allows users to have different "perspectives" of a media types. for example, for the above example, you can ignore @longdesc if you're not interested. another example: you can augment the basic LDX (i have to think more about how that should work exactly, so that you could combine more than one LDX for discovering links in a document): extending HTML's basic LDX, you could select html:link[@rel='DCTERMS.creator']/@href and assign it the role "DC.creator" (ignoring the fact that there can be more than one role in @rel). the main point: by having a separate format, discovering links becomes a separate process that can be extended or changed by media type consumers.

i hope this answers your question. i am still working on some of the specifics (such as how to extend/combine LDX descriptions), but in general, LDX being separate from a media type's schema should be considered a feature and not a bug; at least that is where i started from (items 1 and 2 of my tiny "requirements list").

[Felix Sasaki]

Hello Erik,
maybe this (ITS 1.0)
http://www.w3.org/TR/its/#selection-precedence
provides some answers on how to create what you call inline, and how to relate it to out-of-line. The application area is totally different than yours, but the problems to solve are similar, e.g.: what is the precedence between inline (in the ITS terminology called "local" and out-of-line (in the ITS terminology called "global"), or can there be several out of lines and what happens in case of conflicts (e.g. a node is selected by XPath expressions with conflicting link information) etc.

Best,
Felix

[dret]

thanks a lot for the pointer, @felix! this indeed looks like a very useful document to look at. i guess the problems are, at a certain level of abstraction, the same:

- how to have both inline and out-of-line notations for the same information.

- how to make both notations match the intended data model (in my case, link discovery and associating roles with them) without any restrictions.

- how to have well-defined rules on what's happening for all possible combinations of the available notations.

you pointed to the problem of combining inline and out-of-line information, and that's important. than there also can be more then one out-of-line information (for example one created by the media type creator, and another one created by a consumer). or there can be extensions of out-of-line information (a consumer extending the information created by the media type creator). finding some examples where some of these problems already have been tackled (such as in the ITS) definitely will help to come up with good solutions.

[Felix Sasaki]

Hi Erik again,
I agree, on a certain level of abstraction these are similiar problems. An additional pointer for the generalization of such problems is a paper I presented at Extreme Markup Languages 2006, see
http://fabday.fh-potsdam.de/~sasaki/pubs/archforms-css-rdf.pdf
esp sec. 5, fig 7.
Best,
Felix

[Michael Hausenblas]

Very useful stuff, indeed. Can you please clarify if or how this relates to LRDD [1] - maybe it can be aligned or integrated there?

I'd not go for the W3C path in this case, an IETF RFC is more promising in this area, IMHO. Let me know, happy to support you there ;)

Cheers,
Michael

[1] http://tools.ietf.org/html/draft-hammer-discovery-03

[dret]

@michael, thanks for the comment and the LRDD pointer, i did not know of that draft so far. so my first impression of it is probably a bit superficial. i am not quite sure what LRDD is trying to achieve. it uses POWDER's "describedby" relation type (which would be a "role" in my terminology). it talks about , but probably means (lowercase) and even then it's not quite sure how such a link would be identified across media types. just by the local name matching "link" (case-insensitively)? or would it have to be in a special namespace (in which case XHTML and Atom do not qualify). there is no magic to element names, and this is exactly why i am proposing to have XPath selectors: they work on the Infoset (or XDM, based on which XPath version it will be), and thus allow you to work cleanly with XML's naming concept. as i said, my impression definitely is based on a first pass of LRDD, but it seems to me that LRDD simply enumerates a number of possibilities how in a dualistic web of resources/descriptions, implementations might find the description, given a resource URI and HTTP interactions. there seems to be a lot of dependencies on fluid specs (and the draft seems to define its own URI template method). my impression is that the draft mainly registers POWDER's describedby relation type, and a new field for the host-meta registry, and both of these registries currently do not exist. am i missing something?

so, to compare LRDD and LDX (based on my limited understanding of LRDD): LRDD seems to be concerned with how to find a resource description given a resource URI, and describes a number of ways how to do this description discovery. it also defines special registered values for two of the methods described. LDX only talks about how to identify links in XML-based media types, and is only concerned with discovering links (any links relevant for RESTful services, which probably are many more than just those linking to descriptions) and associating roles with them. LRDD's "describedby" relation type could be one such role, so theoretically speaking, LDX could be listed as one additional way of finding a resource's description in LRDD's list of methods, but only if the resource is XML-based and there is an LDX description for it.

[Michael Hausenblas]

Erik,

Thanks for your answer, I think I see a bit clearer now.

So, yes, LRDD basically proposes three methods for discovery, namely HTTP Link: header, a well-known location and the element (XHTML, Atom), and assigns a sort of uniform interface/semantics to them. As it seems, no single method covers the broad range of use cases, this looks sensible to me. Please have also a look at [1], where Eran puts LRDD into context.

As I said, I dunno if it makes sense to align or integrate LDX and LRDD. We all operate on moving targets, right? ;)

Another thing that came to mind when I read your statement 'any links relevant for RESTful services, which probably are many more than just those linking to descriptions' - how about WADL? Frankly, as you know, I'm no REST expert, however wondering how LDX and WADL are related, or overlapping, respectively.

Cheers,
Michael

[1] http://www.hueniverse.com/hueniverse/2009/03/the-discovery-protocol-stack.html

[dret]

@michael, you're bringing up all the right pointers! so, what about WADL and LDX? WADL focuses on what it calls "grammars" (the schemas for representations) and "resources" (which are a combination of HTTP methods and URI templates). as far as i know, WADL has no provisions to identify links with representations, it depends on the URI templates given in the WADL description (and i think this URI template focus is one of the main criticisms of WADL from a RESTful perspective). so in a way, LDX could augment WADL by adding link discovery to WADL's "grammar" concept. on the other hand, this would conflict a bit with WADL's approach of using URI templates for describing resource links, so i think there is some conceptual difference between LDX and WADL in terms of how they assume that links are being discovered. WADL is statically depending on WADL descriptions, LDX should be more dynamic/adaptive by being bound to media types, at least that's the idea.