dretblog

Media Types and Resource Types

one of the central points of REST is that it is resource-centric, and that learning about new resources should be done by following hyperlinks found in previous resources. how to discover these links of course depends on the types of resources a client is dealing with, which means that the ability to understand resources plays a central role in REST.

what we've recently been running up against is the fact that understanding actually happens on multiple levels, and based on these levels, more or less sophisticated ways to use a service are possible. some examples:

• site navigation links on web sites often would allow very sophisticated understanding of the major resources the site provides, but since HTML has no good way of exposing that information, it is not possible to extract the full richness of the links contained in the typical navigation menu. the best case in this scenario would be to have something akin to sitemap in machine-readable sitemap format, but there is no such thing.
• for specific types of pages (let's consider product pages on some major online retailer) it would be possible to get more specific links regarding a product if the layout (i.e., the type of the page) would be taken into account. this of course breaks as soon as the layout is changed, but given that only few web sites are carefully managing their @class and @id, this at least works well as long as the layout remains stable withing these limits. in this case, the layout of the page provides important semantic hints.
• looking at machine-readable formats, it is interesting to look at Atom. because it is based on XML, it could be served as application/xml, but because it is well-known, it should better be served as application/atom+xml, so that clients understand it's Atom and not just XML, and so that clients can specifically request Atom. however, if it contains additional information such as GeoRSS or feed paging, this cannot be signaled by the media type, but instead it needs to be found out by inspection on the client side. and since there is no media type for these formats, there is no RESTful way to request a GeoRSS or a paged feed.

all of this means that while the theory of only looking at media types and requesting media types looks well in theory, in practice (in particular when embedding something into an existing set of resources and media types instead of starting from scratch) it often is necessary to make assumptions about resource types based on media types and additional hints. these additional hints can be resource inspection, or they can be URI pattern matching. in particular the latter makes many REST folks see red, but i am really wondering whether this isn't something to consider as well when working with existing services or simply describing them as good as possible.

in the end, we want to build an application that starts with a URI, interacts with the URI through a given protocol, processes the resource received as a result of that interaction, continues by discovering links in that resource, chooses the appropriate link type and link instance for the next interaction, and eventually continues this cycle. if the problem of better understanding a resource's semantics can be improved by inspecting the resource itself or its URI, while the primary method of designing interactions still is based on finding links and following them, then things still should be pretty RESTful, even though not in the most ideal way imaginable.

i guess the main issue here is when to promote a pattern or a property of a class of resources to mediatypeness. if some class of resources exposed by a service has useful links in them, and they are not exposed as a specific media type, then there has to be additional information that can be used to identify a resource as a member of that class, so that you can build richer interactions with it. why this may stray from the purist path of REST, for our purpose of describing RESTful services (or almost RESTful services), this is the way that seems to make most sense, pragmatically speaking.

Web of Things 2010 (WoT 2010): Call for Papers

First International Workshop on the Web of Things (WoT 2010)

In conjunction with the 8th IEEE International Conference on Pervasive Computing and Communications (PerCom 2010), Mannheim, Germany

Web site: http://www.webofthings.com/wot/2010

Paper submission deadline: October 18, 2009 November 15, 2009 (Extended)
Notification of acceptance: December 21, 2009
Camera-ready papers due: January 29, 2010
Workshop Date: March 29 or April 2, 2010

The world of embedded devices has experienced radical changes over the past few years as real-world objects can now easily connect to the Internet. This convergence of physical computing devices (wireless sensor networks, mobile phones, embedded computers, etc.) and the Internet provides new design opportunities and challenges. The Internet of Things has mainly focused on establishing connectivity in a variety of challenging and constrained networking environments, and the next logical step in the evolution in pervasive computing builds on top of network connectivity by focusing on the application layer: how to develop ubiquitous computing applications on top of heterogeneous devices? The Web of Things is the vision that brings embedded devices into the Web by using Web standards as application protocol to interact with things. This workshop aims at bringing together the pervasive computing and Web communities to explore the possibilities for using the core principles and technologies of modern Web architecture (e.g., HTTP, REST, Atom, JSON) for seamless integration of things into the Web and developing applications on top of web-enabled devices (physical mashups). Research contributions are solicited in the following areas:

• Web-based interaction patters for embedded devices (streaming, eventing, etc.)
  
• Applications, deployments, and evaluation of Web of things systems
  
• Human-things interaction models and paradigms (mobile interfaces, etc.)
  
• User-oriented, context-aware discovery and dynamic search for the real world
  
• Web composition and macro-programming models (e.g. mashups)
  
• Semantic technologies for description of devices and services
  
• Optimization methods for embedded Web servers and applications
  
• Security, access control, and physical sharing of physical things on the Web

Since this workshop is the first in this area, the goal is to develop a community in this area and provide an interactive forum for researchers to learn and discuss about existing efforts to enable cross-fertilization. Hence, we expect all attendees to read all workshop papers in advance and to prepare questions for each. Presentations shall specifically highlight and address similarities and differences with other accepted papers. This will help to provide a forum to foster future collaboration, beyond the mere presentation of research results. In order to ensure a high-quality technical session, submissions must cover one of the topics above and should not exceed six (6) IEEE conference format pages. Research papers must be original prior unpublished work and not under review elsewhere as they will be published on the IEEE Digital Library. All submissions will be peer-reviewed and selected based on their originality, merit, and relevance to the workshop. Submission requires at least one author to fully register for PerCom 2010 and to present the paper on-site.

Organizers:

• Dominique Guinard, ETH Zurich and SAP Research Zurich
• Erik Wilde, School of Information, UC Berkeley
• Vlad Trifa, ETH Zurich and SAP Research Zurich

Program Committee:

• Gregor Broll, DOCOMO Communications, Germany
• Adam Dunkels, SICS, Sweden
• Martin Gaedke, University of Chemnitz, Germany
• Stamatis Karnouskos, SAP Research, Germany
• Gerd Kortuem, University of Lancaster, UK
• Marc Langheinrich, University of Lugano, Switzerland
• Thomas Luckenbach, Fraunhofer Institute FOKUS, Germany
• Friedemann Mattern, ETH Zurich, Switzerland
• Alexandros Marinos, University of Surrey, UK
• Rene Mayrhofer, University of Vienna, Austria
• Tatsuo Nakajima, Waseda University, Japan
• Benedikt Ostermaier, ETH Zurich, Switzerland
• Cesare Pautasso, University of Lugano, Switzerland
• Dave Raggett, W3C, USA
• Michael Smith, W3C, Japan
• Vlad Stirbu, NOKIA, Finland
• Inaki Vazquez, University of Deusto, Spain
• Agnes Voisard, Fraunhofer ISST, Germany

(( Update 10/8: Extended paper submission deadline from 10/18 to 11/15. ))

Version Links in Atom

Link Relations for Simple Version Navigation (draft-brown-versioning-link-relations) has just been updated; it defines Atom link relations for navigation between a resource and its versions. since i missed the first version, i have now added it to the Atom Landscape Overview as a new draft regarding Atom extensions.

in theory, this draft can be regarded as a standalone proposal to add some more possible link relationships to Atom. in practice, it seems that the draft mainly targets Atom applications in WebDAV and JCR environments. a final version of that draft should also add the newly defined link relationships to the proposed IANA registry for link relationships, but since this one is also still under development, it remains to be seenwhich one is moving along faster.

Sampling Feeds

now that it becomes increasingly apparent that Atom will succeed as a general-purpose format, it also becomes more interesting to think about the world of feeds. there is a very rich body of research around web metrics and web characterization. this is messy on the web, because sampling and categorizing web pages is hard enough. but at least web pages are typically well-connected (not as well as hypertext enthusiasts might wish, but still good enough to be able to crawl the web by simply following links), even though choosing a good seed set of pages to start with and good conditions to stop crawling because of link spam can be a challenge.

doing the same things for feeds would be very interesting, but more challenging. because feeds are different. feeds usually have a rather lonely life, because they are not linked in the web page sense of the word. and they may be discoverable, but many feeds are not made available by autodiscovery but only through regular links. and many feeds are not linked to at all, you simply have to know their URI to find them. and for feeds which are results of query services, a feed is a looking glass into the deep web, but probably one that only shows a very limited part of the underlying data. furthermore, feeds change a lot, so you definitely don't want just a snapshot of a feed, but some analysis over time.

even if there was a large of feeds URIs available, it would be difficult to decide what a representative sample is. for example, large services such as flickr or twitter routinely publish feeds, so these platforms alone are hosting millions of feeds. but all of these feeds look identical (apart from the content, of course), because they are all generated by the same application. the interesting thing, however, would be to find out what typical feeds on the web look like (and please don't ask me to define typical). questions that would be interesting to answer would be statistics for the following things:

• feed formats (RSS versions and Atom)
• use of feed extensions (podcasts, GeoRSS, paging and archiving, threading, licensing)
• feed size (# entries, size of entries, size of feed)
• usage of feed/entry features (statistics about element/attribute usage and values)
• feed update statistics (new entries per hour/day/week, distribution of updates)

all of this is probably a lot to ask for, but i think many people would be very interested in this data. maybe such a study has been done and i am not aware of it? another question i am asking myself: 80legs provides a pretty interesting programming environment for web analytics, but do they also store feeds? and if they do, are those feeds updated? probably not, but 80legs might at least by a pretty useful starting point to find those feeds that are somehow discoverable from web pages.

Atom In-lining

another Atom Landscape update, this time concerning draft-mehta-atom-inline: In-lining Extensions for Atom, which defines mechanisms for in-lining representations of linked resources in Atom documents.

there is an ongoing discussion whether this is Right or Wrong, because it extends Atom in a way which is not clearly allowed by the Atom spec, but it also is not disallowed. Atom's extension model leaves something to be desired, and because it is underspecified, it is impossible to predict how implementations will behave when encountering certain extensions. the basic question is which of these two policies are followed by an implementation:

• everything that is not allowed is forbidden.
• everything that is not forbidden is allowed.

both are not unreasonable assumptions to make (feel free to make your own fun associations with religions here), but in the absence of detailed instructions for every possible syntax variation, they are mutually exclusive and lead to different implementations. thus it is a bit unfortunate that Atom is not entirely clear about its general approach to this problem.

the draft as it currently stands has two problematic extensions: the first extension is that it introduces a child element for atom's <link> element, which usually is an empty element; this looks strange, but may not derail most implementations because they probably only access the <link> elements' attributes anyway. the second extension suggests that the extension element contained in the <link> element may contain atom markup (a complete <feed>); this will probably derail more implementations, because they may assume that atom markup only appears in the feed/entry structure defined in the atom spec, and not in other places as well.

all things considered, this draft seems to address a niche (optimizing pre-fetching of linked resources) and does so in a problematic way. in the end, it is a matter of taste whether people like it or not, so it remains to be seen how well this draft does. but whatever the outcome for that draft will be, it has sparked some useful discussions around the general question of Atom extensibility and best practices around that, and this might eventually lead to a new IETF Atom WG looking into these issues.

Atom Landscape Update

the Atom landscape has been moving quite a bit, and there even is talk about a new IETF working group to better coordinate the development of Atom extensions. the exciting part is that all of this means that more and more people are using Atom for things other than the blogging scenario from which it emerged. this means that Atom's Future as a General-Purpose Format looks more promising than ever, but it also means that it will become more challenging to make all Atom users happy.

these recent activities mean that the Atom Landscape Overview needs to be updated, and as usual, please let me know if i am missing something. here are the latest changes:

• draft-snell-atompub-tombstones: Atom Syndication Format Tombstones has become active again.
• draft-sneddon-atom-export: Atom Export Format as a way how to export a blog (with comments) using Atom.
• draft-divilly-atompub-discovery: AtomPub Guidelines for Collection Discovery as a new attempt to make AtomPub services discoverable.
• draft-divilly-atom-hierarchy: Hierarchy Extensions for Atom replaces draft-divilly-atompub-hierarchy.

i hope this correctly summarizes the recent developments, the complete picture can be found in the Atom Landscape Overview.

Recovery.gov Overhaul

Recovery.gov gears up for a makeover, and while the official announcement has not yet been released, this initial announcement looks a bit troubling. In our stimulus feed guidelines, we have made the point that the recovery architecture (here our first reaction, more detailed feedback, and our comments about the updated guidance) has to look at the bigger picture, and not just at the web site.

The main issue with the overhaul is that it seems to focus just on the site, probably in an effort to provide open access to stimulus data. However, if transparency is also a goal of this initiative, then the whole recovery reporting architecture must be designed and implemented in sync. In particular, there must be open and accessible data flows from the reporting entities into the recovery site. The feed approach described in the initial and updated guidance (which as of today is still not enforced, not supported by any tools or tests, and not used by the recovery site itself) would be the best way to ensure open access and transparency. The announcement reads as if recovery.gov will become another silo that is fed by opaque processes, so that it essentially is the very opposite of transparent.

For a truly transparent and forward-thinking architecture, a federal feed cloud would be the best approach. It would provide a hosted infrastructure for agencies to manage and publish their reporting (if they are not hosting their own feeds), provide tools and tests to make sure data is being published according to the guidelines, and would be a completely transparent way of how everybody could access stimulus reporting data. Recovery.gov would just be one site using this data, and others could build their own sites, if they wanted to.

In theory it is possible that the overhaul could be used to create such a transparent reporting architecture. In practice, however, the challenging deadline and the absence of specifically enforcing a feed-centered architecture make it likely that the new site will have the same intransparent data flows as the current site (which is still being fed by reports being emailed to the OMB, not by the few feeds that are actually available).

Our hope is that the overhaul process will clearly and cleanly separate the reporting architecture, which should be feed-based and fully transparent, and the publication of that data on recovery.gov. The new Recovery.gov should not be based on an intransparent submission process, but instead should finally implement the forward-looking feed architecture mandated in the initial and the updated guidance documents.

It is still not too late to design and implement such an architecture, and it could set a very important precedent for how formerly internal and opaque data flows inside of the government become accessible and transparent. Devaney's office [ the Recovery Act Transparency and Accountability Board ] received $84 million in stimulus funds, the majority of which will be used for the site maintenance and redesign. It would be an excellent use of stimulus money to design and implement an architecture that not only realized the immediate goal of providing transparent stimulus reporting, but also served as a demonstration of how openness and transparency can be implemented based on simple and accessible standards, leading the way for other e-government initiatives. Spending the money on implementing a silo with a polished interface would not only miss and important opportunity, it would also ignore the requirements set forth in the forward-looking guidance documents. It will be very interesting to read the full set of requirements for the recovery.gov overhaul when they are published.

REST Programming Toolbox Requirements

REST is all the rage, and everybody claims they're doing REST. REST is a style of designing web services, and the question is how this style can be best supported by a programming environment. there are two possible perspectives of this problem. one is the more high-level view of how to best structure code that is built around the assumption of a uniform interface. a popular example for this is JAX-RS, released as JSR-311 last year, and implemented by the Restlet framework. this is not what i want to discuss here.

a different perspective is that of basic tools that might be useful for implementing RESTful services (in that case interpreting REST in its narrower sense of web architecture and its use of HTTP and URIs). in a recent blog post, marc hadley reports on Integrating Jersey and Abdera. the interesting thing is how his approach combines a high-level framework (JAX-RS), with more low-level tools for handling HTTP and content (in that case, feeds).

so looking at the low-level support one would like to see when programming RESTful services, what is required or useful? i came up with this list, which is probably incomplete, so any feedback about what else should go in here would be highly welcome!

• HTTP support: HTTP must be fully accessible, which means that all methods, headers, and status codes must be accessible to the application. if required, certain behavior might be enabled on request (such as following redirects), but this should be fully under the control of the application.
• HTTP encoding: since content in HTTP can be transmitted using chunked encoding or some transfer-encoding, it must be possible to either explicitly deal with these encodings, or to have this handled by the HTTP API.
• HTTP authentication: in many cases, HTTP authentication is required for gaining authenticated access to access-controlled resources. the HTTP API should support the predefined HTTP authentication methods (basic/digest), and should also have a plug-in mechanism so that other authentication methods can be used as well.
• caching: caching is one of the core features of how REST can be optimized. caching can be based on modification dates and/or on ETags. client-side caches can be shared or private caches. there must be built-in support for caching, API controls for handling shared and private caches, and there should be integration with a client-side shared cache for the client as a whole (imagine how useful it would be if all browsers on your computer were smart enough to share the same cache).
• URI parser: URI parsing is not trivial, and URIs are used everywhere. there should be URI prcessing support for parsing and resolving URIs. and depending on where URI templates are going (they are still under construction), support for them  might be very useful as well.
• language tag processing: since language tags are part of HTTP and can be used in HTTP mechanisms such as content negotiation, handling language tags is a integral part of HTTP. since language tags have a surprisingly complex logic behind them, support for parsing and matching languages is very useful.
• MIME type processing: similarly to language tags, MIME types (officially called media types) have a structure, are used in content negotiation, and often should be compared or matched. support for these operations should be part of a toolbox as well.
• XML tools: XML is the most widely used syntax for machine-readable data, so there should be a parser, validation support for maybe various schema dialects (DTD, XSD RELAX NG), selection via XPath, hopefully XPath 2.0 as the more powerful language, and finally more advanced XML processing might be supported by XML-centric languages such as XSLT or XQuery.
• JSON tools: many RESTful services support JSON, and sometimes only JSON. support for parsing JSON therefore also should be part of the toolbox.
• Atom tools: one of the most widely used formats for machine-readable data is Atom, and thus Atom support should be part of the toolbox, also supporting additional features which are part of the Atom landscape such as feed paging.
• AtomPub: AtomPub is one way of implementing a RESTful service providing write access to Atom-oriented services, and thus support for AtomPub (at least for the client side) might be useful as well.
• fault-tolerant parsing: many RESTful services access resources from a variety of sources. support for fault-tolerant parsing of HTML and feeds (in any of the RSS variants and in Atom) makes it possible to implement more robust applications.

while i am sure that this list is incomplete, i am really interested to better figure out what it really takes to provide a useful REST programming toolbox. currently, i am mostly interested to figure out what would be of general interest to most/many RESTful service developers.

REST and RDF Granularity

bob ducharme's recent post about Semantic web technology and humanities research and our recent WWW2009 tutorial talking about the route from SOA to REST (to be repeated in june at ICWE 2009) made me think about the fundamental tension between RDF-based semantic web approaches, and RESTful architectures.

it seems to me that the biggest problem of RESTful RDF is the lack of a well-defined notion of a document. one of the strengths of RDF is that you harvest triples from wherever you find them, take the graph, compute the closure of this graph under whatever schemas or ontologies you might have, and then process the resulting graph. on the semantic level this means you take whatever you can get as data, whatever you can get as reasoning mechanisms, and extend the data using inference. it is this general model of raw data and derived information that makes RDF so powerful.

the price you pay, however, is the loss of provenance and document boundaries. of course this can be overlaid using various mechanisms, but it is not a fundamental part of the RDF model, and it's easy to forget it or get it wrong.

this leads to problems when it comes to the rather document-centric model of REST, which talks about representations that need to be exchanged for enabling interactions. how do you use RDF, when the data you're working with for your application purpose is a mix of retrieved RDF and inferred information?

XML, on the other hand, has a rather static idea of a document. well, that's actually not entirely true, because XSD introduced the concept of the Post-Schema Validation Infoset (PSVI), which in a way is the same mix of source data (the original XML document) and inferred data (validation information, type annotations, and default values). type annotations can be regarded as a bonus that can make data processing much more robust and reliable. ironically, default values, however, are widely frowned upon in XSD best practices, because they tightly bind document processing to document validation, and that is largely perceived to be a Bad Idea.

and this is where it gets interesting: XML's very idea is that of enabling applications to GET XML and blindly process it by simply processing well-formed markup. RDF, on the other hand, is more rooted in the idea that the RDF you find blends into an existing set of other triples and schemas or ontologies. this latter model makes working with RDF more seamless and more convenient in an RDF-only world, because essentially, applications can ignore the fact that data had different origins, and they can work on what sometimes has been referred to as the giant global graph. it seems to me that this replicates the pattern of trying to hide distribution that underlies many of the middleware-inspired approaches to web services in a rather interesting way.

XML, on the other hand, makes it almost impossible to get this seamless view of the world, and requires applications to handle one XML document at a time, often steered by overlaid mechanisms such as XInclude at a very low level, or by following hyperlinks when it comes to RESTful data formats. what this means is that XML and this RESTful picture force you to work on the document level, and this also forces developers to explicitly deal with distribution, which means dealing with failures at the network or representation level.

so it seems as if RDF's virtues, the ability to transcend document boundaries and to seamlessly include data from all kinds of sources, become problematic when moving towards RESTful services, where coarse granularity and well-defined document concepts are an essential part of the architectural style.

this does not imply that RDF cannot be used for RESTful services and document-oriented scenarios; it simply means that XML makes it more natural (or you might say: less avoidable) for developers to deal with strict document boundaries, whereas in RDF, this requires special attention. i am really curious to find out whether somebody already came up with a set of best practices and design guidelines for building RESTful RDF applications, and this becomes particularly interesting when looking at issues such as SPARQL (and its basic model of accessing an RDF triple store through just one URI) and updates of RDF data.

it seems to me that the road towards a RESTful semantic web (if we take the RDF-centric view of that concept) is still not fully clear, and i am really curious to explore in more depth the fundamental tension between REST's things are diverse and distributed and you have to deal with it, and RDF's in the end it's all just triples world views.

Atom's Future as a General-Purpose Format

subbu allamaraju recently raised the question of Atom as a General-Purpose Format, and showed one example (Google Code Search Data API) that looked like a not-so-great way of using Atom. in my comment on his blog, i agreed to his observation that this did not look all that great as far as Atom goes, but i also argued that it might still make sense to use Atom for such a scenario.

comparing Atom with HTML (not as formats, but as methods for information dissemination) can teach us a lot, in particular when looking at possible future developments for Atom. the Atom Landscape already shows some similarities to HTML, thanks to Atom's extension semantics. there are microformats (licenses and paging, for example), there is a lot of custom code only picked up by custom clients (such as subbu's Code Search Data API example), and some of the Atom on the Web is so badly crippled that it is utterly useless for plain Atom clients. that's exactly like the good old best viewed in Internet Explorer web pages.

however, while this might not be what we want to see, it may very well be what we need to see to make Atom as exciting as i think it will be. the web is what it is exactly because HTML is so bad. HTML is bad on many different levels and not good at anything in particular. but it's sufficient as a container for an astonishing number of things, and it is simple enough to be learned and used by many. people still use PDF when they care about display fidelity and Flash when they care about multimedia features, but all of these formats are nothing compared to HTML.

REST still is on its way up and i am concerned about the fact that it has been officially downgraded into a buzzword and will suffer the hype cycle backlash at some point in time. but regardless of that, REST is what makes the web tick, and for many people, REST is still hard to grasp (our WWW2009 From SOA to REST tutorial last week was great, but also showed us that there is quite some way to go to spread the message). having best practices and design patterns and frameworks and tools will help a lot, and i expect all of these to appear around Atom.

and we will see all the things that are inevitable when something gets picked up by many. we'll see under-specified extensions (such as GeoRSS), overlapping formats (such as Apple's Podcasts and Yahoo's Media RSS), toolkits (for example supporting feed paging and archiving), and all the other things (some of them invasive and/or malicious) we have seen in HTML. and as inconvenient as that may be from time to time, it also and most importantly is the sure sign of a healthy and active ecosystem.

so what does all of that mean for Atom-oriented developers? similar things as for HTML-oriented web designers, plus some extra care that's required because your goal is to be understood by machines, not by humans. always validate what you're producing, test with other than your preferred clients (such as a plain Atom feed reader), try to reuse established standards in the Atom Landscape or promising candidates for new formats, and most importantly: build your system in a way so that you can change and adapt. if new standards or formats emerge, you should be able to support them, because you have implemented a clear and clean separation between your model (whatever it may be) and the representation of it (Atom plus some extensions).