dretblog

REST Layers and REST Applications

there has been a rather long history of people trying to describe REST applications, and an equally long history of responses saying that it cannot be done, should not be done, and that REST is allow about following your nose across a web of interlinked resources. many approaches have used some kind of RDF-based model, and have either tried to model/describe an application in terms of SemWeb principles, or in some cases the standards (such as HTTP) themselves. however, so far there seems to be no consensus on whether it could/should be done, and given the fact that we expect RESTful APIs to become more popular, it would be good if the REST community could provide some guidance on how those APIs should be described/documented.

in an attempt to better understand what and why people are trying to accomplish when they try to describe REST, here is a first approach to more cleanly layer the issues, see what's already around, and try to figure out what value could provided by some format/technology for each layer. we are actively working in this area because we simply need something that will allow service providers to describe (and by that we mostly mean document) their services. our platform allow customers to define and expose REST services, and when they do this and want to point, for example, external partners at these services asking them to develop clients, instructing them to tell your external partners to follow their nose just doesn't quite cut it.

in this first attempt at a layering model, there are five layers:

1. architectural style: this is where REST itself is defined, independent of any technology. this is where the architectural principles of the style are defined, and fielding' thesis by definition is the one and only authoritative source in this layer.
2. technology framework: on top of REST's principles, web architecture encompasses a set of enabling mechanisms, most importantly URI for identification, HTTP for interaction, and media types for labeling representations (and it is interesting to notice that the core technologies on this layer are all defined by IETF instead of the W3C). while this layer is essential for the fabric of the web, in itself it does not yet define a working system, because it lacks concrete representations that can be used as transfer formats.
3. core technologies: for the human web, HTML is the most important media type (but without more media types such as images, the web never have succeeded), and for machines XML and RDF have become important enabling technologies. it is important to notice that all of these media type are application-agnostic and not application-specific formats. clients need to support those media types to work properly, but without additional guidance (provided by human operators or additional information), they don't know what they are doing and blindly interact with a set of semantically shallow resources. core technologies allow clients to work in a RESTful way, but they may not allow servers to expose the richness of the services they are providing.
4. added semantics: (almost) all of the web's core technologies have some mechanism to add semantics. sometimes these mechanisms have formalized representations (such as XML's DTDs or XSD), sometimes not (HTML profiles are not described in any standardized way). some added semantics are metamodels (XML schema languages allow authors to define their own schemas), some are fixed models (podcasts define a fixed set of extensions that can be found in feeds). most importantly, the core technologies provide ways how these added semantics can be signaled to clients, for XML through namespaces and/or associated schemas, and for HTML through profile parameters (and the proposed 'profile' link relation type might be able to unify this mechanism across media types). added semantics allow servers and clients to support semantic overlays that allow richer and more semantics-driven forms of interaction.
5. applications: based on core technologies and possibly added semantics, developers build applications, which expose the resources of a given service in some URI space. these applications can use any number of media types and any set of added semantics on top of these, and in some cases developers will define their own media types, whereas in others they will just reuse existing ones. in both cases, the application is seamlessly embedded into the bigger context of the RESTful system, and while that may be not interesting at all for some clients, and might as well be interesting for others. for the HTML web, google's sitemaps format has become an established representation in this area because it allows servers and certain clients (crawlers) to interact in ways that are advantageous for both sides.

based on some of the discussions that happened around describing REST, it seems that at times, there was a mismatch in where in these layers people wanted to be. when it comes to layers 3 and 4, documentation/description should be the media type or semantic extension itself. for these cases, all that a client should need to know is the processing and operational model of the resource, and then it can act. however, when it comes to layer 5, i am a firm believer that there are scenarios where a description format would be very desirable, and the sitemaps format is one (very widely deployed) example.

in our work on the Resource Linking Language (ReLL) we have explored this area and made some design choices that i would now make differently. but generally speaking, giving application providers the possibility to describe the extent of their application and doing that in a way which itself is a self-describing resource seems like a very RESTful approach. if a client is interested in this information, there is a discovery mechanism (for sitemaps a slightly awkward robots.txt-based rule that hardcodes an URI-based approach) and then the client can happily use that information, and could for example inform the user that he is now leaving the application (which may or may not be indicated by a change in the some URI prefix). if clients are not interested in the notion of an application at all (like regular users happily surfing the web without ever experiencing application boundaries), they simply ignore this information.

starting from this idea of REST layers, i'd be curious to get some feedback about the perceived usefulness of a description language that specifically just resides in layer 5. the main goal is to establish a format that could be seen as sitemaps for REST, and definitely should be less chatty than exhaustively listing all URIs. this would allow service providers to describe the universe of resources they are exposing, and would give clients a snapshot view of the current expectations they can have about exposed resources, and the assumptions about where the scope of the applications ends. in a follow-up to this post, i will outline our scenarios, and write more about the things we would like to include and exclude in such a REST Application Description Language.

Atom Content Negotiation

media types are an important part of web architecture, and they even have (some) structure; the very obvious type/subtype syntax, and the maybe not as obvious "+" convention in types such as application/atom+xml. however, while that allows you to expose the fact that something is Atom and that Atom is based on XML, it does not allow you to talk about what the published entries are.

let's assume you have a collection of items that is exposed via feeds and managed in whatever back-end storage works well for you. you want to be a good web citizen and publish it as HTML, XML, JSON, and RDF. you also want to be a good web service citizen and publish any updates via feeds, so that consumers are notified whenever the collection changes. what is the best web pattern to do that? currently, there are two main approaches:

• publish different feeds. in this case, each content type is published in a different feed. one problem is that these feeds are not interlinked, so it is not possible to switch feeds easily. another problem is that there is no way how to advertise these feeds in a way that marks them as being content type variants of the same content.
• link content variants. in our suggestions for Web Services for Recovery.gov, we had just one feed that would embed reports in HTML, and an XML version would be linked in each entry as an alternate version. this clearly makes HTML primary and XML secondary, and if all you're interested in is XML, it forces you to GET all HTML, only to ignore it and then GET the XML.

both approaches have advantages and disadvantages, and both are not very pretty. in addition, when adding more advanced feed-based services such as updates and push, well-defined handling of content types across the whole landscape becomes more important.

i am wondering what the more popular approach is, and why. i am also wondering whether it might be worth the effort to have a small extension that would make this more reliable in the sense that there would be a link relation for interlinking feeds that carry the same content, but using different content types. it could be as simple as having such a link on the feed level:

<link rel="alternate-content" href="..." content-type="application/xml"/>

is that something that would be worth doing? it could become more sophisticated (and complicated) by allowing other feed parameters to be negotiated, such as feed formats (here's an RSS version of me), feed variants (here is the feed with breaking news only, here's the feed with abstracts only), and probably a variety of other things. currently, i would be tempted to do the simple version only, but i am really curious to find out in which cases these problems already have been solved, in which way they have been solved, and whether standardizing this might provide some value.

[[ a pointer to this entry has been posted to the atom-syntax mailing list. ]]

Web Linking

one of the main components of REST is linking (modeling applications as hypermedia, i.e. interconnected resources), and more specifically, typed linking, where a link not only specifies where it links to (in web-oriented systems this is always done using URIs), but what it links to, so that clients can choose which links to follow. in many cases, these link types are application-specific, but there also are link types which are fairly universal, and which should be reused across applications instead of being reinvented many times.

so far, link relation types were defined within the confines of applications and/or representations; both HTML and Atom/AtomPub, for example, define their own link relation types, and those are only expected to be used for these specific representation types.

the recently published RFC 5988 (Web Linking) specifies relation types for web links more generally, and defines a registry for them. it also defines the use of such links in HTTP headers using the Link header field. this means that from now on, links for RESTful application do not (necessarily) have to be embedded in representations anymore. instead, if they pertain to the requested resource, they can be communicated in a Link header field, which is specifically mentioned to be semantically equivalent to HTML's <link> element and Atom's feed-level <link> element.

the immediate advantage of this approach is that links can now be used for resources where the representation does not define a place to put them, and that resources do not have to be parsed to find links connecting them to other resources. in fact, by using HEAD requests, resources do not even have to be retrieved to find the links that link them to other resources, which means that Link-aware clients can traverse RESTful services in a much more efficient way, as long as all they need are resource-level links.

adding to my REST Programming Toolbox Requirements, i would thus expect a good REST programming toolbox to start supporting the Link header field, and to allow applications to more or less ignore the fact whether a link has been found in resource, or in a Link header field. ideally, a toolbox might even be smart enough to optimize requests, so that HEAD requests are used whenever only resource-level links are required in cases where the server supports the Link header field. this, however, requires knowledge of whether the server will reliably serialize all links as Link header fields, and this is not something that can be determined automatically; a programmer would have to configure that.

in addition to the initial link types defined by RFC 5988, RFCs 5829 (Link Relation Types for Simple Version Navigation between Web Resources) and 5989 (A SIP Event Package for Subscribing to Changes to an HTTP Resource) have added values to the registry so far, and over time the registry should grow as a place where RESTful applications can look up, reuse, and add link relation types for hypermedia-driven applications.

since RFC 5988 extends the way in which link types should be handled in Atom and can be exposed by HTTP servers publishing Atom, i have restructured the Atom Landscape Overview to start listing any RFCs (or drafts) that might add link relation types to the link relation type registry. as usual, any feedback to further improve the overview is greatly appreciated.

Data, Models, Metamodels, Cosmologies

when i recently tweeted about the fact that the semantic web or linked data (choose your preferred brand name) had this troubling attitude of one metamodel to rule them all, i got some feedback saying that this was wrong. it's not, and i guess the reason for the confusion in that space is that data, models, metamodels, and cosmologies are somehow hard to grasp, so here is my brief attempt at displaying all of them in one handy overview (and the selection of items in that overview is definitely not complete and simply provided for illustrative reasons).

Cosmology	Key/Value Pairs	Unordered Tables		Ordered Trees		Generalized Graphs
Metamodel	?	Entity-Relationship (ER)	Relational (SQL)	SGML	XML Infoset XDM	RDF
Model	?	ER Graph	SQL DDL	DTD	DTD XSD RELAX-NG	RDFS OWL
Data	CSV Bigtable SimpleDB	?	RDBMS	SGML	XML EXI	RDF/XML N3 Turtle

what does this mean, in particular the weird cosmology part? it just starts with the observation that people usually start with an application domain, and certain recurrent features/patterns in that domain they want to see supported in their models and the tools they work with. which is the reason why the hierarchical databases of the early days were replaced by the more appropriate tabular structures provided by ER and SQL. for the same reason, document processing always had its own world, because working with documents cannot be done effectively in unordered tabular structures, so this is where the trees come in. they exist in various flavors, but the main idea is that you have a tree and that the tree is ordered, which maps well to document structures.

as long as you stay within the same cosmology, mappings usually work reasonably well, and working with the data can be done in at least comparable ways. there may be differences in features and tools and languages, but the big picture of how the data is structured is the same. sometimes there are special features of metamodels, for example whether a model is even required: SQL data always needs a database to live in, whereas XML and RDF can happily live without a specific model (i.e., schema) and can still be used.

things become much more complicated if you want to take data from one cosmology, and map it to another cosmology. this is always possible, but for non-trivial data it often results in a severe mismatch between the application model and the underlying assumptions of the metamodel. it also often means that the tools and languages in the new environment are not appropriate anymore. imagine representing a complex XML document in RDF (i am sure somewhere there is some RDFS for representing XML). it can be done, sure, but compared to the tools provided with XSD and XQuery and XSLT, working with that data will become much more complicated and very ineffective (and also probably very inefficient in terms of processing times when considering large document collections), because the inherent ordered tree structure of the data is not supported by the underlying metamodel anymore.

some people may think that because it is always possible to define mappings between any metamodels, that implies that there is one overarching metamodel that spans all of the above cosmologies (as a layer on top of all cosmologies), and since RDF is the metamodel with the fewest built-in constraints, the claim is often made by the semantic web community that RDF can be that one metamodel. however, this ignores the fact that there is a reason why there are different cosmologies and metamodels and model languages and tools: these have evolved over time to deal with certain classes of data, and they usually do a good job in their domains. RDF has had success because it, too, deals with a certain class of data, it is typically being used for metadata. metadata, even though it's only loosely defined, typically is not data with complex structures, and thus using a simple model such as RDF works well.

the motivation for this post is simply to show that RDF is not the metamodel to end all metamodels. it is one metamodel, and simply a new addition to the existing multitude of cosmologies, metamodels, and models that have been around for a long time, and it has found an application area for which it is well-suited. claiming that its simplicity (i.e., the ability to map other cosmologies to the RDF cosmology) means that all data out there can be appropriately represented and handled as RDF ignores the fact that in the end, it's not important that it's possible to do something, but only how effective you can be when you're doing it: if your job is processing large collections of documents, what models, languages and tools make you most productive at getting your job done.

Retrofitting RETRO

ICWE 2009 was an interesting conference, and one of the good parts of it was the fact that even though it still has a substantial model-driven engineering bias, there are signs that other communities start recognizing the conference as well. i hope this trend will continue, and it definitely will be interesting to compare the programs of the 2009 and the 2010 conferences (btw, ICWE 2010 will be held in vienna, hosted by TU Vienna).

one of the people at ICWE was alexandros marinos, one of the creators of RETRO. RETRO recently caused some discussions in the REST communities because it tries to provide a RESTful approach to transactions, which often are more perceived as being in the realm of RPC/middleware oriented approaches to web services. i guess, if you want to use REST and transactions, there are three basic approaches:

• leave the transaction semantics in the application, so that clients just need to know what is going on. this is not loosely coupled, and makes it impossible to provide tools and support for transactions.
• introduce new methods. this is what WebDAV did by extending HTTP with new methods. this is a very clean approach, but also one which makes adoption rather hard.
• introduce some overlay of links and/or resources that represent the transactions. the magic .../lock resource is one such thing, but as long as there is no way how it can be discovered, this also falls into the first category. otherwise, there have to be ways of how links and/or resources can expose their transaction semantics.

the RETRO approach falls squarely into the last category, and personally, that is the category that i prefer. there are three issues i have with the (admittedly very early) draft:

• it seems to me that not all of the resources introduced in the model are strictly necessary, and least not as publicly available resources. it would be interesting to look at the minimal core, and define all other resources as optional.
• RETRO introduces media types and suggests to include a certain XML fragment in lockable resources. it would probably be more consistent and more extensible to just require certain vocabularies be present in the resource representations, so that each service could (if it wanted) use its own schema, importing the RETRO schema. RETRO could still provide default schema to use for those services that do not want to define their own schemas, but link discovery should be based on namespaces and imported schemas, not on media types.
• changing URIs from transaction-contained resources to the eventual URI is a pretty major operation. at least, there should (or maybe even MUST) be an xml:base in the resource that always sets its base URI to the final URI. and generally, i am asking myself whether this identity switch is not a potential source of confusion and/or errors. would there be other ways of doing this? and is it even necessary to have these temporary resources, or are they also optional and not part of the core model?

but even though i think the model could be improved, i really like the direction, and i am particularly interested to investigate potential synergies with LDX, which currently mainly is an idea and a first sketch, but will get more attention soon. i still think that REST's idea of placing hyperlinks in the core of the interaction model demands something like LDX: a way of how to make statements about links and their semantics (in the sense of link roles). combining RETRO sans media types and using LDX as the way to specify RETRO's interconnected transaction resources could be a nicely flexible way of allowing services to use their own media types (as long as they're XML), without sacrificing the discoverability of the RESTful transaction links.

Link Discovery for XML

one of the main principles of REST is what unfortunately most often is called Hypermedia as the Engine of Application State (HATEOAS), which must be the worst acronym ever invented. regardless of that, this is one of the central REST constraints, and also one that is often ignored, with services using no discoverable links at all, or using URI templates and assuming that this already is RESTful. it can be, but it often isn't, because even if there was a stable standard for URI templates (and there isn't), it still would be impossible to know how to substitute the template variables to get URIs of available resources. which is why RESTful services should include links in resources, so that applications can simply follow those links.

in web architecture, REST's links are found by knowing a resource's media type, and knowing how to find the links in that format. the most popular linked formats on the web probably are (X)HTML and Atom, both of them define links and link semantics in their data format. however, since XML is such an popular syntax for many different content types, i have been toying with the idea of defining a generic XML mechanism for discovering links in XML (the only generic descriptions for XML available today are the built-in xml:lang, and the separate xml:id specification). notice that this is different from datatype mechanisms such as XSD, which are able to type content as a URI, but it is still not clear whether this URI is a link, and if so, what its semantics are (which i will call role from now on). as a starting point, here is a brief list of requirements:

• link discovery should work inline (information embedded in XML instances) and out-of-line (as an external description of how to find links in some XML).
• link discovery can be a perspective of a media type, i.e. different applications might have different ideas of how to find links in a given media type, or a subset of documents of a given media type.
• links can have roles, and are allowed to have more than one role. role names can be strings, qualified names, or URIs.
• the mechanism should rely on the smallest possible set of technologies, so that it can be easily implemented using widely deployed tools and technologies.
• there should be a simple syntax for fully describing the link discovery information, so that it can be serialized and exchanged.

starting from this simple and very likely incomplete set of requirements, here is my first stab at a mechanism for Link Discovery in XML (LDX). the main idea is that links are identified by XPath-based selectors (XSLT's pattern subset might be a good match here), that selected links are associated with roles, and that this mechanism can be applied to any XML-based media type. what would be the advantage of such a mechanism? a simple way to describe the links and link semantics of XML media types, the ability to build generic tools that can extract links from any XML-based media type, and the ability to better separate general REST functionality (what links can i follow in this resource?) from content-specific functionality (what does the content mean, and what does it mean to follow the links?). generally, such a mechanism could become part of a REST toolbox, and there are advantages for both service providers and consumers.

now on to the specifics. it's a bit harder to come up with a good inline format (HLink still might be remembered by some), so my first pre-alpha idea focuses on the out-of-line format. it's a simple set of selectors associated with 0-n roles. each selector selects an attribute or element content (interpreted as the string value), and this content is then recognized as a link with the associated roles. selectors probably should be able to be grouped and contextualized, so that for example some minimal LDX for XHTML might look like this:

<links xmlns:html="http://www.w3.org/1999/xhtml">
 <context match="html:html/html:head">
  <link match="html:link[@rel='stylesheet']/@href" role="stylesheet"/>
 </context>
 <context match="html:html/html:body">
  <link match=".//html:img/@src" role="image"/>
 </context>
</links>

this would allow to automatically extract all links to stylesheets and images from XHTML documents. the role names in this case are just strings, and there definitely has to be some better handling of non-string role names, in particular differentiating qualified names and URIs (which could, at least in theory, use CURIE syntax) might become a bit of a challenge.

namespaces (in the described media types) are a headache as usual, the eternal question being what to do with unprefixed names in selectors. use the LDX document's default namespace (à la XSD), or have a special attribute for setting the default namespace (à la XSLT)? i prefer the latter, but that's more syntax juggling than anything else.

the fun part is that because of the selectors, applications can even try to be smart and create LDX that works for the specific content they're concerned with. let's imagine a RESTful application using XHTML which contains links, and always contains a link to a product in the first column of each row of a specific table. using a selector such as table[@id='products']/tr/td[1]/a/@href, it would be possible to associate these links with a specific role, and with LDX support in a REST toolbox, finding all those links would be trivial. and the LDX description of the assumptions made about the content would be concise and declarative, so that it would be easy to understand the linking assumptions of the application.

most of LDX currently exists in my head only, and right now i am mostly wondering what others think of that idea. does this make sense from a RESTful point of view? would it be a worthwhile addition to a REST toolbox? maybe even to the REST landscape in general? and, assuming it does make some sense and i am tempted to start writing down a bit more of my ideas, should i start with a W3C or with an IETF template?

REST Programming Toolbox Requirements

REST is all the rage, and everybody claims they're doing REST. REST is a style of designing web services, and the question is how this style can be best supported by a programming environment. there are two possible perspectives of this problem. one is the more high-level view of how to best structure code that is built around the assumption of a uniform interface. a popular example for this is JAX-RS, released as JSR-311 last year, and implemented by the Restlet framework. this is not what i want to discuss here.

a different perspective is that of basic tools that might be useful for implementing RESTful services (in that case interpreting REST in its narrower sense of web architecture and its use of HTTP and URIs). in a recent blog post, marc hadley reports on Integrating Jersey and Abdera. the interesting thing is how his approach combines a high-level framework (JAX-RS), with more low-level tools for handling HTTP and content (in that case, feeds).

so looking at the low-level support one would like to see when programming RESTful services, what is required or useful? i came up with this list, which is probably incomplete, so any feedback about what else should go in here would be highly welcome!

• HTTP support: HTTP must be fully accessible, which means that all methods, headers, and status codes must be accessible to the application. if required, certain behavior might be enabled on request (such as following redirects), but this should be fully under the control of the application.
• HTTP encoding: since content in HTTP can be transmitted using chunked encoding or some transfer-encoding, it must be possible to either explicitly deal with these encodings, or to have this handled by the HTTP API.
• HTTP authentication: in many cases, HTTP authentication is required for gaining authenticated access to access-controlled resources. the HTTP API should support the predefined HTTP authentication methods (basic/digest), and should also have a plug-in mechanism so that other authentication methods can be used as well.
• caching: caching is one of the core features of how REST can be optimized. caching can be based on modification dates and/or on ETags. client-side caches can be shared or private caches. there must be built-in support for caching, API controls for handling shared and private caches, and there should be integration with a client-side shared cache for the client as a whole (imagine how useful it would be if all browsers on your computer were smart enough to share the same cache).
• URI parser: URI parsing is not trivial, and URIs are used everywhere. there should be URI prcessing support for parsing and resolving URIs. and depending on where URI templates are going (they are still under construction), support for them  might be very useful as well.
• language tag processing: since language tags are part of HTTP and can be used in HTTP mechanisms such as content negotiation, handling language tags is a integral part of HTTP. since language tags have a surprisingly complex logic behind them, support for parsing and matching languages is very useful.
• MIME type processing: similarly to language tags, MIME types (officially called media types) have a structure, are used in content negotiation, and often should be compared or matched. support for these operations should be part of a toolbox as well.
• XML tools: XML is the most widely used syntax for machine-readable data, so there should be a parser, validation support for maybe various schema dialects (DTD, XSD RELAX NG), selection via XPath, hopefully XPath 2.0 as the more powerful language, and finally more advanced XML processing might be supported by XML-centric languages such as XSLT or XQuery.
• JSON tools: many RESTful services support JSON, and sometimes only JSON. support for parsing JSON therefore also should be part of the toolbox.
• Atom tools: one of the most widely used formats for machine-readable data is Atom, and thus Atom support should be part of the toolbox, also supporting additional features which are part of the Atom landscape such as feed paging.
• AtomPub: AtomPub is one way of implementing a RESTful service providing write access to Atom-oriented services, and thus support for AtomPub (at least for the client side) might be useful as well.
• fault-tolerant parsing: many RESTful services access resources from a variety of sources. support for fault-tolerant parsing of HTML and feeds (in any of the RSS variants and in Atom) makes it possible to implement more robust applications.

while i am sure that this list is incomplete, i am really interested to better figure out what it really takes to provide a useful REST programming toolbox. currently, i am mostly interested to figure out what would be of general interest to most/many RESTful service developers.

REST and RDF Granularity

bob ducharme's recent post about Semantic web technology and humanities research and our recent WWW2009 tutorial talking about the route from SOA to REST (to be repeated in june at ICWE 2009) made me think about the fundamental tension between RDF-based semantic web approaches, and RESTful architectures.

it seems to me that the biggest problem of RESTful RDF is the lack of a well-defined notion of a document. one of the strengths of RDF is that you harvest triples from wherever you find them, take the graph, compute the closure of this graph under whatever schemas or ontologies you might have, and then process the resulting graph. on the semantic level this means you take whatever you can get as data, whatever you can get as reasoning mechanisms, and extend the data using inference. it is this general model of raw data and derived information that makes RDF so powerful.

the price you pay, however, is the loss of provenance and document boundaries. of course this can be overlaid using various mechanisms, but it is not a fundamental part of the RDF model, and it's easy to forget it or get it wrong.

this leads to problems when it comes to the rather document-centric model of REST, which talks about representations that need to be exchanged for enabling interactions. how do you use RDF, when the data you're working with for your application purpose is a mix of retrieved RDF and inferred information?

XML, on the other hand, has a rather static idea of a document. well, that's actually not entirely true, because XSD introduced the concept of the Post-Schema Validation Infoset (PSVI), which in a way is the same mix of source data (the original XML document) and inferred data (validation information, type annotations, and default values). type annotations can be regarded as a bonus that can make data processing much more robust and reliable. ironically, default values, however, are widely frowned upon in XSD best practices, because they tightly bind document processing to document validation, and that is largely perceived to be a Bad Idea.

and this is where it gets interesting: XML's very idea is that of enabling applications to GET XML and blindly process it by simply processing well-formed markup. RDF, on the other hand, is more rooted in the idea that the RDF you find blends into an existing set of other triples and schemas or ontologies. this latter model makes working with RDF more seamless and more convenient in an RDF-only world, because essentially, applications can ignore the fact that data had different origins, and they can work on what sometimes has been referred to as the giant global graph. it seems to me that this replicates the pattern of trying to hide distribution that underlies many of the middleware-inspired approaches to web services in a rather interesting way.

XML, on the other hand, makes it almost impossible to get this seamless view of the world, and requires applications to handle one XML document at a time, often steered by overlaid mechanisms such as XInclude at a very low level, or by following hyperlinks when it comes to RESTful data formats. what this means is that XML and this RESTful picture force you to work on the document level, and this also forces developers to explicitly deal with distribution, which means dealing with failures at the network or representation level.

so it seems as if RDF's virtues, the ability to transcend document boundaries and to seamlessly include data from all kinds of sources, become problematic when moving towards RESTful services, where coarse granularity and well-defined document concepts are an essential part of the architectural style.

this does not imply that RDF cannot be used for RESTful services and document-oriented scenarios; it simply means that XML makes it more natural (or you might say: less avoidable) for developers to deal with strict document boundaries, whereas in RDF, this requires special attention. i am really curious to find out whether somebody already came up with a set of best practices and design guidelines for building RESTful RDF applications, and this becomes particularly interesting when looking at issues such as SPARQL (and its basic model of accessing an RDF triple store through just one URI) and updates of RDF data.

it seems to me that the road towards a RESTful semantic web (if we take the RDF-centric view of that concept) is still not fully clear, and i am really curious to explore in more depth the fundamental tension between REST's things are diverse and distributed and you have to deal with it, and RDF's in the end it's all just triples world views.

Atom's Future as a General-Purpose Format

subbu allamaraju recently raised the question of Atom as a General-Purpose Format, and showed one example (Google Code Search Data API) that looked like a not-so-great way of using Atom. in my comment on his blog, i agreed to his observation that this did not look all that great as far as Atom goes, but i also argued that it might still make sense to use Atom for such a scenario.

comparing Atom with HTML (not as formats, but as methods for information dissemination) can teach us a lot, in particular when looking at possible future developments for Atom. the Atom Landscape already shows some similarities to HTML, thanks to Atom's extension semantics. there are microformats (licenses and paging, for example), there is a lot of custom code only picked up by custom clients (such as subbu's Code Search Data API example), and some of the Atom on the Web is so badly crippled that it is utterly useless for plain Atom clients. that's exactly like the good old best viewed in Internet Explorer web pages.

however, while this might not be what we want to see, it may very well be what we need to see to make Atom as exciting as i think it will be. the web is what it is exactly because HTML is so bad. HTML is bad on many different levels and not good at anything in particular. but it's sufficient as a container for an astonishing number of things, and it is simple enough to be learned and used by many. people still use PDF when they care about display fidelity and Flash when they care about multimedia features, but all of these formats are nothing compared to HTML.

REST still is on its way up and i am concerned about the fact that it has been officially downgraded into a buzzword and will suffer the hype cycle backlash at some point in time. but regardless of that, REST is what makes the web tick, and for many people, REST is still hard to grasp (our WWW2009 From SOA to REST tutorial last week was great, but also showed us that there is quite some way to go to spread the message). having best practices and design patterns and frameworks and tools will help a lot, and i expect all of these to appear around Atom.

and we will see all the things that are inevitable when something gets picked up by many. we'll see under-specified extensions (such as GeoRSS), overlapping formats (such as Apple's Podcasts and Yahoo's Media RSS), toolkits (for example supporting feed paging and archiving), and all the other things (some of them invasive and/or malicious) we have seen in HTML. and as inconvenient as that may be from time to time, it also and most importantly is the sure sign of a healthy and active ecosystem.

so what does all of that mean for Atom-oriented developers? similar things as for HTML-oriented web designers, plus some extra care that's required because your goal is to be understood by machines, not by humans. always validate what you're producing, test with other than your preferred clients (such as a plain Atom feed reader), try to reuse established standards in the Atom Landscape or promising candidates for new formats, and most importantly: build your system in a way so that you can change and adapt. if new standards or formats emerge, you should be able to support them, because you have implemented a clear and clean separation between your model (whatever it may be) and the representation of it (Atom plus some extensions).

Feeds as Query Result Serializations

Many Web-based data sources and services are available as feeds, a model that provides consumers with a loosely coupled way of interacting with providers. The current feed model is limited in its capabilities, however. Though it is simple to implement and scales well, it cannot be transferred to a wider range of application scenarios. This paper conceptualizes feeds as a way to serialize query results, describes the current hardcoded query semantics of such a perspective, and surveys the ways in which extensions of this hardcoded model have been proposed or implemented. Our generalized view of feeds as query result serializations has implications for the applicability of feeds as a generic Web service for any collection that is providing access to individual information items. As one interesting and compelling class of applications, we describe a simple way in which a query-based approach to feeds can be used to support location-based services.

Feeds as Query Result Serializations is a technical report published today. it describes what we think will be the next step in the evolution of lightweight web services. in the same way as containerization has revolutionized the global trade of physical goods, feeds are already essential as a lightweight and universal way of sharing machine-readable information, and this trend will continue.

one of the major missing pieces in the atom landscape today is the lack of well-defined query features, and we argue that such a capability would make atom feeds much more flexible in a wide variety of scenarios. this can be as simple as querying feeds by category (something already possible with GData), or as sophisticated as spatial queries when considering geo-referenced collections. more specifically, the step from the Internet of Things to the Web of Things, i.e. making the step from enabling connectivity to a web architecture for networked things, probably will be made possible by smart ways of extending and combining feed technologies. after all, many of the data streams produced by sensors or other pervasive technologies can be readily mapped to feeds, and exposing stream processing capabilities as queryable feeds would bridge the gap between web-based information streams, and information streams originating in the physical world.