dretblog

The Linked Data™ Police

this morning i was contacted by the Linked Data™ Police. i found the following email in my inbox, from a pretty well-established researcher at a pretty well-established institution:

I came across this: http://repositories.cdlib.org/ischool/2009-035/

So you consciously redefine term linked data to mean something different. Look, the big point of linked data is that it means you are interoperable with a large number of existing data sources available in the same data model and protocol, like Freebase, DBpedia, Geonames, and so forth. I had to explain to colleagues that no, there is nobody working on providing linked data for recovery.gov, and the guys in Berkeley who claim to be working on it are actually just abusing the term as a buzzword in order to get free attention, but actually have no intention of delivering anything that's interoperable with the linked data standards.

There are perfectly descriptive terms for the technology you are using, REST and ROA.

You do a disservice to yourself. What you do is intellectually dishonest and unnecessarily antagonizes those who have worked hard to establish a set of interoperable practices around RDF on the web.

Please stop this practice.

so instead of just discussing the question of whether Linked Data requires RDF, you now get a cease and desist letter when your work is about publishing data that is linked and you dare to call it Linked Data. i think the letter mostly speaks for itself, but what really scares me is the upside-down attitude expressed by it. instead of looking at the web at something that is good and helpful and should be improved by adding more semantics, and then trying to figure out the most effective way of doing this, given the constraints of the scenario considered, it starts with a set of technologies you have and want to use, and then claim that whatever you want to do, you have to do it using those technologies.

generally, the approach to use generic problem names to refer to specific technologies is something that only helps to confuse a lot of people, and mostly is an attempt to position these technologies in a way which makes competition harder. my favorite examples for bad labels for specific technologies are Linked Data, Semantic Web, Web Services, and XML Schema. if you build technologies, stick to calling them technologies, don't start calling them solutions. a solution is a technology that is applied to a given problem within the limits given by the problem scenario's constraints, and no technology can predict the problem.

oh, and, just for the record: if you look at the approach we're taking beyond just checking of whether we propose to use RDF directly, RDF is just a stylesheet away. publish your data in a RESTful way that's accessible to all feed users and readers out there, and then writing some GRDDL that will fill a triple store becomes trivial. which kind of demonstrates the point that RDF is just an implementation issue, it should not be the litmus test of whether you follow the True Path Of Linked Data. and the report quoted above actually explicitly explains exactly what we do, and how it compares to the narrower definition of Linked Data:

Recently, the idea of openly accessible data has been promoted under the term of linked data, with recent recommendations being centered around a very specific choice of technologies and data models (all centered around Semantic Web approaches focusing on RDF for data representation and centralized data storage). While it is possible to use these approaches for building Web applications, our recommendation is to use better established and more widely supported technologies, thereby lowering the barrier-to-entry and choosing a simpler toolset for achieving the same goals as with the more sophisticated technologies envisioned for the Semantic Web. For the remainder of this report, we use the term linked data as the general concept of publishing interlinked data representations, without referring to the one specific way of implementing it that is often associated with that term as well.

i think this is about as clear as you can get. and i think a pretty large set of substantial  problems have yet to be tackled in the RDF approach to linked data; identity, provenance, dynamic services, granularity, and updates come to mind. so rather than arguing about who is entitled to call linked data Linked Data™, it might be more interesting and enlightening to compare different approaches as solutions for specific problems, and see how well they work.

Links in HTML

looking at how HTML exactly works from a RESTful perspective requires to look at all the different links that HTML defines. these are actually quite a bit more than just <A> links, and based on the HTML 4.01 transitional DTD, here is the complete list of all links that can be specified in HTML. one interesting observation is that some of these links are not supported in today's browsers, such as the cite attribute on <Q> and <BLOCKQUOTE>.

Element	Attribute	Method	Response Resource	Semantics
BODY	background	GET	image/*	texture tile for document background
A	href	GET	*/*	URI for linked resource
AREA	href	GET	*/*	URI for linked resource
LINK	href	GET	*/* (based on rel and/or type, more specific types may be expected)	URI for linked resource
IMG	src	GET	image/*	URI of image to embed
	longdesc	GET	*/*	link to long description (complements alt)
	usemap	GET	text/html (fragment must be MAP element)	use client-side image map (often local fragment identifier)
OBJECT	classid	GET (relative to codebase)	application/*	identifies an implementation
	codebase	-	-	base URI for classid, data, archive
	archive	GET (multiple URIs, relative to codebase)	*/*	space-separated list of URIs
	data	GET (relative to codebase)	*/*	reference to object's data
	usemap	GET	text/html (fragment must be MAP element)	use client-side image map (often local fragment identifier)
APPLET	codebase	-	-	base URI for classid, data, archive
	archive	GET (relative to codebase)	*/*	comma-separated archive list
	code	GET (relative to codebase)	*/*	applet class file
	object	GET (relative to codebase)	*/*	serialized applet file
Q, BLOCKQUOTE	cite	GET	*/*	URI for source document or msg
INS, DEL	cite	GET	*/*	info on reason for change
FORM	action	GET/POST (depending on method)	POST Request: application/x-www-form-urlencoded or multipart/form-data (depending on enctype) Response: */*	server-side form handler
INPUT	src	GET	image/*	for fields with images
	usemap	GET	text/html (fragment must be MAP element)	use client-side image map (often local fragment identifier)
FRAME, IFRAME	longdesc	GET	*/*	link to long description (complements title)
	src	GET	*/*	source of frame content
HEAD	profile	GET (often treated as a name only)	*/*	named dictionary of meta info
BASE	href	-	-	URI that acts as base URI
SCRIPT	src	GET	application/*	URI for an external script
	for	-	-	reserved for possible future use

Good Lists are Hard

being a web plumber, one of pages i visit almost daily is the technical reports (TR) page of the W3C at http://www.w3.org/TR/. when this page was updated last week (as part of the launch of the new W3C site), it was amazing to see how many things you can break at the same time, and it also was an interesting lesson in how simple and smart design can be much more usable and useful than just adding functionality.

the old TR was a good web page in a variety of ways. it was lightweight, had a completely noise-free design, had all the recent updates on top as shortcuts to the complete listings further down, and in short was a well-designed and self-contained representation of the TR work done by the W3C. you could go there, quickly see up the latest additions with your browser clearly highlighting the pages you had not yet visited, and do everything else with the browser's find function. it was one of the pages i always showed to people when explaining why simple and smart is often better than just adding fluff without properly considering usability and utility.

the new TR page managed to break almost everything that was useful, and it does so in a variety of ways. it removed the previously smart quick/complete combined view, it added a lot of visual noise, and it breaks the previously correct usage of the W3C's spec URIs. here is where the new design fails:

• quick view plus complete info. one of the useful things of the old page was the fact that it had a short recent specs section at the beginning, and complete listings further done. it was a very elegant way to both have a quick view of what's new, and a complete view of everything, all available via simple browser-based search. now there are more options, but none of those is as useful as the old view, which is not available anymore.
• useless layout noise. the layout of the old page was simple yet effective, putting as much information in one browser window as possible. the new layout contains a lot of useless layout noise such as horizontal lines between each line, plus a lot of spacing surrounding the text and the lines. it probably doubles the space required per item, while also listing less information. using a table costs even more space without adding any value to the presentation.
• proper URI handling. W3C specs have versioned as well as canonical URIs, and the old TR page linked to the versioned URIs, which is the right thing to do, because the TR page contains a date-based of specs, and not a list of canonical URIs of the latest versions of all specs. the new page links to the canonical URIs, essentially linking to something other than the resource it is actually listing on the TR list. this even breaks basic principles of REST, where the pages users are GETting don't properly link to the resources they are referring to; instead they are referring to a canonical URI that does not represent the actual document that has been published. the very real consequence of this particular breakage is that the browser cannot reliably color the links based on whether the URIs of the particular versions are in the history or not.

in summary, it would be interesting to know whether the new TR has undergone any internal usability and utility testing before being released. it is broken in a variety of ways, which is kind of interesting, but from the perspective of a regular user, it really is sad to see that a page as useful and well-designed as the old TR page is being replaced by a much more noisy and much less useful version. i will certainly miss the old TR, and i guess so will others who were closely following the W3C's TR work.

if you think the old TR was better, please leave a brief +1 comment here, and maybe it will at least come back as a classic view for those who like web pages that are simple and effective.

REST and Link Computation

the idea of defining a Formal Hypermedia Context definitely has some appeal, taking the rather basic HATEOAS principle one step further. thinking about Media Types and Resource Types basically moves in the exact same direction: figuring out what it takes to describe and then use a RESTful service. deciding on a transition in a RESTful service not only depends on the links found in representations; it also critically depends on understanding (a) where to find those links in the first place (link syntax), (b) what those links mean in the context of the RESTful service (link semantics), and (c) which link to actually follow to achieve the application goals (link pragmatics).

we are currently working on a description language for RESTful services which is trying to make as much explicit about a RESTful service as possible. it is a bit similar to the model recently presented on InfoQ, but extends it in a crucial way. the InfoQ model has link relation types and XML schemas, but since there is no existing XML schema language that explicitly supports a link type, there is no way how a link can be found in any given XML. we change this by adding selectors that allow to select links in XML using XPath. XPaths may select URIs, but they can also just select IDs which can then be transformed into URIs by applying link-specific rules, such as adding a prefix.

this idea of computing links based on the hypermedia context could be as simple as prefix/postfix, more sophisticated such as using URI templates, even more sophisticated in the case of a even more expressive language for computing URIs. taken to the extreme, such a of link computation could even include variables carried over from previous navigational actions, in which case we would lose some of the nice context-free properties of one-step-at-a-time hypermedia links. we are still experimenting with how expressive these languages should be (so that more real-world services can be described, even if their RESTfulness is less than perfect), and how constrained they should be so that they're reasonably easy to understand and use. for now, we might just go the simple prefix/postfix route, but exploring the idea of a more formal model certainly would be an interesting exercise, and also help to understand at which point a more expressive model starts to compromise some core principles of REST. my guess would be that local link computations are acceptable, but that starting to carry over state from previous link traversals would too clearly collide with the HATEOAS principle.

Media Types and Resource Types

one of the central points of REST is that it is resource-centric, and that learning about new resources should be done by following hyperlinks found in previous resources. how to discover these links of course depends on the types of resources a client is dealing with, which means that the ability to understand resources plays a central role in REST.

what we've recently been running up against is the fact that understanding actually happens on multiple levels, and based on these levels, more or less sophisticated ways to use a service are possible. some examples:

• site navigation links on web sites often would allow very sophisticated understanding of the major resources the site provides, but since HTML has no good way of exposing that information, it is not possible to extract the full richness of the links contained in the typical navigation menu. the best case in this scenario would be to have something akin to sitemap in machine-readable sitemap format, but there is no such thing.
• for specific types of pages (let's consider product pages on some major online retailer) it would be possible to get more specific links regarding a product if the layout (i.e., the type of the page) would be taken into account. this of course breaks as soon as the layout is changed, but given that only few web sites are carefully managing their @class and @id, this at least works well as long as the layout remains stable withing these limits. in this case, the layout of the page provides important semantic hints.
• looking at machine-readable formats, it is interesting to look at Atom. because it is based on XML, it could be served as application/xml, but because it is well-known, it should better be served as application/atom+xml, so that clients understand it's Atom and not just XML, and so that clients can specifically request Atom. however, if it contains additional information such as GeoRSS or feed paging, this cannot be signaled by the media type, but instead it needs to be found out by inspection on the client side. and since there is no media type for these formats, there is no RESTful way to request a GeoRSS or a paged feed.

all of this means that while the theory of only looking at media types and requesting media types looks well in theory, in practice (in particular when embedding something into an existing set of resources and media types instead of starting from scratch) it often is necessary to make assumptions about resource types based on media types and additional hints. these additional hints can be resource inspection, or they can be URI pattern matching. in particular the latter makes many REST folks see red, but i am really wondering whether this isn't something to consider as well when working with existing services or simply describing them as good as possible.

in the end, we want to build an application that starts with a URI, interacts with the URI through a given protocol, processes the resource received as a result of that interaction, continues by discovering links in that resource, chooses the appropriate link type and link instance for the next interaction, and eventually continues this cycle. if the problem of better understanding a resource's semantics can be improved by inspecting the resource itself or its URI, while the primary method of designing interactions still is based on finding links and following them, then things still should be pretty RESTful, even though not in the most ideal way imaginable.

i guess the main issue here is when to promote a pattern or a property of a class of resources to mediatypeness. if some class of resources exposed by a service has useful links in them, and they are not exposed as a specific media type, then there has to be additional information that can be used to identify a resource as a member of that class, so that you can build richer interactions with it. why this may stray from the purist path of REST, for our purpose of describing RESTful services (or almost RESTful services), this is the way that seems to make most sense, pragmatically speaking.

Web of Things 2010 (WoT 2010): Call for Papers

First International Workshop on the Web of Things (WoT 2010)

In conjunction with the 8th IEEE International Conference on Pervasive Computing and Communications (PerCom 2010), Mannheim, Germany

Web site: http://www.webofthings.com/wot/2010

Paper submission deadline: October 18, 2009 November 15, 2009 (Extended)
Notification of acceptance: December 21, 2009
Camera-ready papers due: January 29, 2010
Workshop Date: March 29 or April 2, 2010

The world of embedded devices has experienced radical changes over the past few years as real-world objects can now easily connect to the Internet. This convergence of physical computing devices (wireless sensor networks, mobile phones, embedded computers, etc.) and the Internet provides new design opportunities and challenges. The Internet of Things has mainly focused on establishing connectivity in a variety of challenging and constrained networking environments, and the next logical step in the evolution in pervasive computing builds on top of network connectivity by focusing on the application layer: how to develop ubiquitous computing applications on top of heterogeneous devices? The Web of Things is the vision that brings embedded devices into the Web by using Web standards as application protocol to interact with things. This workshop aims at bringing together the pervasive computing and Web communities to explore the possibilities for using the core principles and technologies of modern Web architecture (e.g., HTTP, REST, Atom, JSON) for seamless integration of things into the Web and developing applications on top of web-enabled devices (physical mashups). Research contributions are solicited in the following areas:

• Web-based interaction patters for embedded devices (streaming, eventing, etc.)
  
• Applications, deployments, and evaluation of Web of things systems
  
• Human-things interaction models and paradigms (mobile interfaces, etc.)
  
• User-oriented, context-aware discovery and dynamic search for the real world
  
• Web composition and macro-programming models (e.g. mashups)
  
• Semantic technologies for description of devices and services
  
• Optimization methods for embedded Web servers and applications
  
• Security, access control, and physical sharing of physical things on the Web

Since this workshop is the first in this area, the goal is to develop a community in this area and provide an interactive forum for researchers to learn and discuss about existing efforts to enable cross-fertilization. Hence, we expect all attendees to read all workshop papers in advance and to prepare questions for each. Presentations shall specifically highlight and address similarities and differences with other accepted papers. This will help to provide a forum to foster future collaboration, beyond the mere presentation of research results. In order to ensure a high-quality technical session, submissions must cover one of the topics above and should not exceed six (6) IEEE conference format pages. Research papers must be original prior unpublished work and not under review elsewhere as they will be published on the IEEE Digital Library. All submissions will be peer-reviewed and selected based on their originality, merit, and relevance to the workshop. Submission requires at least one author to fully register for PerCom 2010 and to present the paper on-site.

Organizers:

• Dominique Guinard, ETH Zurich and SAP Research Zurich
• Erik Wilde, School of Information, UC Berkeley
• Vlad Trifa, ETH Zurich and SAP Research Zurich

Program Committee:

• Gregor Broll, DOCOMO Communications, Germany
• Adam Dunkels, SICS, Sweden
• Martin Gaedke, University of Chemnitz, Germany
• Stamatis Karnouskos, SAP Research, Germany
• Gerd Kortuem, University of Lancaster, UK
• Marc Langheinrich, University of Lugano, Switzerland
• Thomas Luckenbach, Fraunhofer Institute FOKUS, Germany
• Friedemann Mattern, ETH Zurich, Switzerland
• Alexandros Marinos, University of Surrey, UK
• Rene Mayrhofer, University of Vienna, Austria
• Tatsuo Nakajima, Waseda University, Japan
• Benedikt Ostermaier, ETH Zurich, Switzerland
• Cesare Pautasso, University of Lugano, Switzerland
• Dave Raggett, W3C, USA
• Michael Smith, W3C, Japan
• Vlad Stirbu, NOKIA, Finland
• Inaki Vazquez, University of Deusto, Spain
• Agnes Voisard, Fraunhofer ISST, Germany

(( Update 10/8: Extended paper submission deadline from 10/18 to 11/15. ))

Distribution and the Semantic Web

RPC and SemWeb have the same implicit goal of hiding distribution that somehow goes against the grain of RESTful loose coupling was an observation i recently voiced on twitter, and that might need a bit of explanation. for now, i'll focus on the semantic web part of that, but it seems to me that this general approach of trying to hide distribution is amazingly similar in both the WS-* world and the semantic web. in both cases, this makes it hard to think of how to get them better aligned with basic web principles. to stay a bit more focused, i will concentrate on RDF for the purpose of this post.

RDF is great at making statements about URI-identified things, because this is what it is all about. build graphs of things that attach, by reference, to anything that can be identified by URI. how to identify RDF triples and graphs themselves is an entirely different issue, though. RDF's abstract model leaves them dangling in the void, and while SPARQL introduces the concept of named graphs (effectively turning triples into quadruples), these names are basically namespace URIs, which means they identify named graphs, but there is no standardized way of how to interact with them (they are never dereferenced). so this means that in a typical RDF store, triples are not identified at all, and even if they are (let's say somebody manages their RDF store in a way which guarantees that every triple is associated with exactly one named graph URI), there is no well-defined interaction with those triples.

this all starts with the basic approach of many semantic web projects: people want to have and use information about things on the web, or they start with the assumption that others have information about things on the web, and then they can harvest that information. this terminology is quite interesting, and if we want to get all metaphorical, then this is what harvesting is all about: you cut stuff, separating it from its roots, and you end up with a lot of useful harvested stuff, but it's sort of dead and you will never be able to reconnect it with where it came from. it may give you useful seeds for new things as well, but again, those things will only be very loosely connected to where they came from.

so, on the semantic web, it's all about getting many RDF triples, dropping them into one big silo (which is why you see all these statements about we have converted such-and-such into x billion triples!), and then working with that silo (maybe providing one tiny hole in that silo through which SPARQL queries can be funneled). this approach makes it easy to abstract from the fact that all the information started out from different sources (and probably still lives there and may be exposed via live services), but it also makes it hard to manage things such as provenance and authenticity (which are critically important for distributed and federated systems). as patrick murray-john pointed out in his comment on my recent post about Data, Models, Metamodels, Cosmologies, RDF's biggest strength is its excellent capability to mix data together. however, this seamless blending of data from different sources gets problematic when you look at REST, which is built around the general idea of coarse-grained interactions with resources identified by URI.

so when you are treating semantic web work as the process of harvesting data, and then using it locally in some form that allows read access only and loses track of the identity and the boundaries of the original data sources, then you're not working with a web of data, but you are just working with data you harvested from the web. this assumes that the harvested data is static, and that any updates to the harvested data will not have to be propagated to where it originally came from. essentially, such an approach treats the web, in an idealized world view, as one huge RDF store, assuming that the process of collecting all the data is taken care of, and that there is no channel of how to interact with the original data sources. and this idealized world view is reflected in many projects around the semantic web, where it is assumed that problems of distribution are taken care of, and can be regarded as solved. as a side note, i think this is why there recently has been such a huge uptake of semantic web work in the government data space, because the government can be conveniently viewed as one homogeneous data source, providing one big collection of RDF data that can be accessed through centralized services. many of the problems of federated data sources, conflicting data, changing data, data authenticity, data integrity, and heterogenous data sources and services can be ignored in such a centralized scenario.

the current work around SPARQL Update does not look all that different. it looks at how to allow the full spectrum of CRUD operations for SPARQL implementations (in the same way as XQuery progressed from a read-only language to one with updates), but it still follows the same approach of looking at RDF data as living in one big silo, and standardizing a language for how to interact with that silo. it's important that such a language exists and i think SPARQL and the Update extension to it are very important specs, but this language has as much to do with the web as SQL or XQuery: not all that much. it's back-end stuff, and for me as a client of a web service, all i want is a way of how i can interact with that service through HTTP so that i can use that service effectively; whether the back-end of that service uses SQL or XQuery or SPARQL or a column store does not really interest me all that much.

so what i am currently doing (collaborating with linked data expert michael hausenblas) is trying to figure out a way how to make SPARQL more REST-friendly, so that SPARQL can be easily used as a back-end to provide RESTful services on the web. i am looking forward to our work of making SPARQL more REST-friendly, but it is interesting to me how that turns into an exercise of ignoring the subject URIs of triples as actionable URIs, and instead using the named graph URIs to give triples an identity on the web, so that they can be exposed in a RESTful way. in such a RESTful SPARQL world, it will be possible to GET a bunch of triples from somewhere, and then to PUT the exact same bunch of triples to a URI that identifies a named graph that needs to be updated. it seems to me that this kind of RESTful approach will make it much easier to expose meaningful resources on the semantic web, and to allow RESTful interactions with them.

Retrofitting RETRO

ICWE 2009 was an interesting conference, and one of the good parts of it was the fact that even though it still has a substantial model-driven engineering bias, there are signs that other communities start recognizing the conference as well. i hope this trend will continue, and it definitely will be interesting to compare the programs of the 2009 and the 2010 conferences (btw, ICWE 2010 will be held in vienna, hosted by TU Vienna).

one of the people at ICWE was alexandros marinos, one of the creators of RETRO. RETRO recently caused some discussions in the REST communities because it tries to provide a RESTful approach to transactions, which often are more perceived as being in the realm of RPC/middleware oriented approaches to web services. i guess, if you want to use REST and transactions, there are three basic approaches:

• leave the transaction semantics in the application, so that clients just need to know what is going on. this is not loosely coupled, and makes it impossible to provide tools and support for transactions.
• introduce new methods. this is what WebDAV did by extending HTTP with new methods. this is a very clean approach, but also one which makes adoption rather hard.
• introduce some overlay of links and/or resources that represent the transactions. the magic .../lock resource is one such thing, but as long as there is no way how it can be discovered, this also falls into the first category. otherwise, there have to be ways of how links and/or resources can expose their transaction semantics.

the RETRO approach falls squarely into the last category, and personally, that is the category that i prefer. there are three issues i have with the (admittedly very early) draft:

• it seems to me that not all of the resources introduced in the model are strictly necessary, and least not as publicly available resources. it would be interesting to look at the minimal core, and define all other resources as optional.
• RETRO introduces media types and suggests to include a certain XML fragment in lockable resources. it would probably be more consistent and more extensible to just require certain vocabularies be present in the resource representations, so that each service could (if it wanted) use its own schema, importing the RETRO schema. RETRO could still provide default schema to use for those services that do not want to define their own schemas, but link discovery should be based on namespaces and imported schemas, not on media types.
• changing URIs from transaction-contained resources to the eventual URI is a pretty major operation. at least, there should (or maybe even MUST) be an xml:base in the resource that always sets its base URI to the final URI. and generally, i am asking myself whether this identity switch is not a potential source of confusion and/or errors. would there be other ways of doing this? and is it even necessary to have these temporary resources, or are they also optional and not part of the core model?

but even though i think the model could be improved, i really like the direction, and i am particularly interested to investigate potential synergies with LDX, which currently mainly is an idea and a first sketch, but will get more attention soon. i still think that REST's idea of placing hyperlinks in the core of the interaction model demands something like LDX: a way of how to make statements about links and their semantics (in the sense of link roles). combining RETRO sans media types and using LDX as the way to specify RETRO's interconnected transaction resources could be a nicely flexible way of allowing services to use their own media types (as long as they're XML), without sacrificing the discoverability of the RESTful transaction links.

Link Discovery for XML

one of the main principles of REST is what unfortunately most often is called Hypermedia as the Engine of Application State (HATEOAS), which must be the worst acronym ever invented. regardless of that, this is one of the central REST constraints, and also one that is often ignored, with services using no discoverable links at all, or using URI templates and assuming that this already is RESTful. it can be, but it often isn't, because even if there was a stable standard for URI templates (and there isn't), it still would be impossible to know how to substitute the template variables to get URIs of available resources. which is why RESTful services should include links in resources, so that applications can simply follow those links.

in web architecture, REST's links are found by knowing a resource's media type, and knowing how to find the links in that format. the most popular linked formats on the web probably are (X)HTML and Atom, both of them define links and link semantics in their data format. however, since XML is such an popular syntax for many different content types, i have been toying with the idea of defining a generic XML mechanism for discovering links in XML (the only generic descriptions for XML available today are the built-in xml:lang, and the separate xml:id specification). notice that this is different from datatype mechanisms such as XSD, which are able to type content as a URI, but it is still not clear whether this URI is a link, and if so, what its semantics are (which i will call role from now on). as a starting point, here is a brief list of requirements:

• link discovery should work inline (information embedded in XML instances) and out-of-line (as an external description of how to find links in some XML).
• link discovery can be a perspective of a media type, i.e. different applications might have different ideas of how to find links in a given media type, or a subset of documents of a given media type.
• links can have roles, and are allowed to have more than one role. role names can be strings, qualified names, or URIs.
• the mechanism should rely on the smallest possible set of technologies, so that it can be easily implemented using widely deployed tools and technologies.
• there should be a simple syntax for fully describing the link discovery information, so that it can be serialized and exchanged.

starting from this simple and very likely incomplete set of requirements, here is my first stab at a mechanism for Link Discovery in XML (LDX). the main idea is that links are identified by XPath-based selectors (XSLT's pattern subset might be a good match here), that selected links are associated with roles, and that this mechanism can be applied to any XML-based media type. what would be the advantage of such a mechanism? a simple way to describe the links and link semantics of XML media types, the ability to build generic tools that can extract links from any XML-based media type, and the ability to better separate general REST functionality (what links can i follow in this resource?) from content-specific functionality (what does the content mean, and what does it mean to follow the links?). generally, such a mechanism could become part of a REST toolbox, and there are advantages for both service providers and consumers.

now on to the specifics. it's a bit harder to come up with a good inline format (HLink still might be remembered by some), so my first pre-alpha idea focuses on the out-of-line format. it's a simple set of selectors associated with 0-n roles. each selector selects an attribute or element content (interpreted as the string value), and this content is then recognized as a link with the associated roles. selectors probably should be able to be grouped and contextualized, so that for example some minimal LDX for XHTML might look like this:

<links xmlns:html="http://www.w3.org/1999/xhtml">
 <context match="html:html/html:head">
  <link match="html:link[@rel='stylesheet']/@href" role="stylesheet"/>
 </context>
 <context match="html:html/html:body">
  <link match=".//html:img/@src" role="image"/>
 </context>
</links>

this would allow to automatically extract all links to stylesheets and images from XHTML documents. the role names in this case are just strings, and there definitely has to be some better handling of non-string role names, in particular differentiating qualified names and URIs (which could, at least in theory, use CURIE syntax) might become a bit of a challenge.

namespaces (in the described media types) are a headache as usual, the eternal question being what to do with unprefixed names in selectors. use the LDX document's default namespace (à la XSD), or have a special attribute for setting the default namespace (à la XSLT)? i prefer the latter, but that's more syntax juggling than anything else.

the fun part is that because of the selectors, applications can even try to be smart and create LDX that works for the specific content they're concerned with. let's imagine a RESTful application using XHTML which contains links, and always contains a link to a product in the first column of each row of a specific table. using a selector such as table[@id='products']/tr/td[1]/a/@href, it would be possible to associate these links with a specific role, and with LDX support in a REST toolbox, finding all those links would be trivial. and the LDX description of the assumptions made about the content would be concise and declarative, so that it would be easy to understand the linking assumptions of the application.

most of LDX currently exists in my head only, and right now i am mostly wondering what others think of that idea. does this make sense from a RESTful point of view? would it be a worthwhile addition to a REST toolbox? maybe even to the REST landscape in general? and, assuming it does make some sense and i am tempted to start writing down a bit more of my ideas, should i start with a W3C or with an IETF template?

REST Programming Toolbox Requirements

REST is all the rage, and everybody claims they're doing REST. REST is a style of designing web services, and the question is how this style can be best supported by a programming environment. there are two possible perspectives of this problem. one is the more high-level view of how to best structure code that is built around the assumption of a uniform interface. a popular example for this is JAX-RS, released as JSR-311 last year, and implemented by the Restlet framework. this is not what i want to discuss here.

a different perspective is that of basic tools that might be useful for implementing RESTful services (in that case interpreting REST in its narrower sense of web architecture and its use of HTTP and URIs). in a recent blog post, marc hadley reports on Integrating Jersey and Abdera. the interesting thing is how his approach combines a high-level framework (JAX-RS), with more low-level tools for handling HTTP and content (in that case, feeds).

so looking at the low-level support one would like to see when programming RESTful services, what is required or useful? i came up with this list, which is probably incomplete, so any feedback about what else should go in here would be highly welcome!

• HTTP support: HTTP must be fully accessible, which means that all methods, headers, and status codes must be accessible to the application. if required, certain behavior might be enabled on request (such as following redirects), but this should be fully under the control of the application.
• HTTP encoding: since content in HTTP can be transmitted using chunked encoding or some transfer-encoding, it must be possible to either explicitly deal with these encodings, or to have this handled by the HTTP API.
• HTTP authentication: in many cases, HTTP authentication is required for gaining authenticated access to access-controlled resources. the HTTP API should support the predefined HTTP authentication methods (basic/digest), and should also have a plug-in mechanism so that other authentication methods can be used as well.
• caching: caching is one of the core features of how REST can be optimized. caching can be based on modification dates and/or on ETags. client-side caches can be shared or private caches. there must be built-in support for caching, API controls for handling shared and private caches, and there should be integration with a client-side shared cache for the client as a whole (imagine how useful it would be if all browsers on your computer were smart enough to share the same cache).
• URI parser: URI parsing is not trivial, and URIs are used everywhere. there should be URI prcessing support for parsing and resolving URIs. and depending on where URI templates are going (they are still under construction), support for them  might be very useful as well.
• language tag processing: since language tags are part of HTTP and can be used in HTTP mechanisms such as content negotiation, handling language tags is a integral part of HTTP. since language tags have a surprisingly complex logic behind them, support for parsing and matching languages is very useful.
• MIME type processing: similarly to language tags, MIME types (officially called media types) have a structure, are used in content negotiation, and often should be compared or matched. support for these operations should be part of a toolbox as well.
• XML tools: XML is the most widely used syntax for machine-readable data, so there should be a parser, validation support for maybe various schema dialects (DTD, XSD RELAX NG), selection via XPath, hopefully XPath 2.0 as the more powerful language, and finally more advanced XML processing might be supported by XML-centric languages such as XSLT or XQuery.
• JSON tools: many RESTful services support JSON, and sometimes only JSON. support for parsing JSON therefore also should be part of the toolbox.
• Atom tools: one of the most widely used formats for machine-readable data is Atom, and thus Atom support should be part of the toolbox, also supporting additional features which are part of the Atom landscape such as feed paging.
• AtomPub: AtomPub is one way of implementing a RESTful service providing write access to Atom-oriented services, and thus support for AtomPub (at least for the client side) might be useful as well.
• fault-tolerant parsing: many RESTful services access resources from a variety of sources. support for fault-tolerant parsing of HTML and feeds (in any of the RSS variants and in Atom) makes it possible to implement more robust applications.

while i am sure that this list is incomplete, i am really interested to better figure out what it really takes to provide a useful REST programming toolbox. currently, i am mostly interested to figure out what would be of general interest to most/many RESTful service developers.