dretblog

XML PATCH

the PATCH Method for HTTP is a rather recent addition to HTTP, supporting partial resource modification. while it is similar to PUT in the sense that it is used to update an existing resource, it is similar to POST in the sense that it is neither safe nor idempotent (because applying the same patch two times can modify a resource two times). for a good overview of why PATCH may be good for your web services, mark nottingham's blog post is a great starting point.

one of the defining characteristics of PATCH is that PATCH requests are supposed to carry representations of how to patch a resource, meaning that such a request fully describes what the client is attempting to do. this means that you either have to add such a "patch model" to your media type (i.e., the design of your web service), or maybe you can reuse a patch model that has been defined elsewhere (and then you can simply use this media type for your PATCH interactions).

the recently published JSON Patch does exactly that: it defines a generic patch model for JSON, so that services that don't want to design their own patch model can simply use the application/json-patch+json media type for PATCH interactions (and the Accept-Patch header even allows servers to advertise the supported patch media types at runtime). JSON Patch defines its own patch model as part of the specification.

in the similar way, the current draft for XML Patch proposes a model and a media type for a generic XML Patch model. unlike JSON Patch, though, it depends on an existing XML Patch framework that is based on XPath and simple operators. in the process of defining the XML Patch media type, however, we discovered that RFC 5261 has a couple of problems, which are corrected in an appendix of the XML Patch specification. these problems were mainly caused by confusing namespace declarations and namespace nodes, and some misunderstandings of XPath. many thanks to my colleague bas de bakker for bringing up these issues when we were looking at how to implement XML Patch in our native XML database xDB!

XML Patch is at a point now where we have one implementation, and we believe that other XML services (in particular generic XML services such as XML databases) might be interested to adopt the model, instead of having to define their own patch model for partially modifying interactions with XML resources. of course, service-specific patch models (defined in the context of specific XML-based media types) always can be better optimized to the data model of the service, but even if services do expose their own patch models, the Accept-Patch header allows services to advertise support for more than just one patch model.

while the draft so far has been an individual effort (i.e., not driven by any working group), earlier this month an attempt was made to have it adopted by the IETF Applications Area Working Group (appsawg). however, there was not enough interest/support within the working group, which means that from the working group perspective, the document is on hold for now. however, if you are supporting XML-based services in generic way, and would like to support PATCH, then you still might want to consider using the model proposed in XML Patch. and if there is some interest in the XML community to turn this media type into a standard, then i'd like to encourage you to get in touch and provide some feedback, and to make your voice heard on the appsawg mailing list.

On Profiles

now that The profile Link Relation Type has been published as RFC 6906 (and is properly registered in IANA's link relation registry), this is a good moment to explain what profiles can be used for, how they should be used (and shouldn't be used), and which kinds of scenarios might be most interesting to look at for using the mechanism.

Bundling Functionality

one way of describing profiles is to look at them as capability bundles (i am just making this term up here), which are supposed to identify a specific way in which an extensible and open data format is used to accomplish a more specialized goal. specifically, the idea of a profile is not to just identify a single extension element or attribute.

instead, consider podcasts used as one example in the RFC. or, you could also think of OPDS, another feed-based content distribution architecture. in both cases, the specifications are using feeds as their foundation, i.e. you can read podcasts and OPDS feeds with a generic feed reader. this satisfies one of the requirements for scenarios that are a good fit for profiles: it should be possible to have meaningful applications using the services without knowing/using the profile. specifically, for this reason it would not be a proper use of profiles to say that podcasts or OPDS are profiles of RDF or XML, because you cannot meaningfully consume podcasts or OPDS just based on on parsing RDF or XML. you need knowledge of the concept of feeds to have something that minimally works (but of course a generic feed reader will not support any of the features that are specific to podcasts or OPDS).

the second requirement for using profiles in a useful way is that profiles should identify meaningful and self-contained capability bundles. both is true for podcasts and OPDS, which start from plain feeds and then add features and capabilities to them in order to address a specific set of use cases. don't give in to the temptation to make things too fine-granular and turn every single property/feature into a profile. this would lead to an inflation of profiles, and make working with them awkward and unnecessarily complicated.

The OO View

apart from the capability bundles concept, another useful way of thinking of them is in terms of the OO concept of a subclass. instances of subclasses can be handled in exactly the same way as instances of their superclass, but they have additional constraints, and/or have additional behavior. the same can be said of resources using profiles: they can be used according to their profile (i.e., as an instance of the subclass), or they can be used according to their media type (i.e., as an instance of their superclass), ignoring the fact that they follow a profile.

taking this view even further, you could say that in that OO view each superclass can have multiple subclasses (i.e., it is possible for a resource to specify more than one profile), but subclasses only have one superclass (because there onle is one media type in which a profile is used).

however, since the notion of a profile is fairly open, there is nothing that keeps profile designers from saying that their profile can be used with different media types. one obvious example could be RDF, which is an abstract model, and has a number of possible serializations (all identified by different media types). a profile could be based on RDF's abstract model, i.e. not be specific for any specific serialization, and then could be identified in different RDF media types through the same profile URI, making sure that the profile can be recognized in a way that is indepedent from the specific serialization.

Profile URIs

as defined by the RFC, profile URIs are purely identifiers: clients are expected to recognize them when they are used with profile-typed links or as profile media type parameters, and then deduce from that that a resource follows a certain profile. this means that profile URIs never have to be dereferenced, and thus could also use non-referencable URI schemes such as URNs.

however, as with other identifier-style URIs (XML namespace URIs come to mind), it is a good idea to use referencable HTTP URIs, and provide some documentation (or other helpful resources) at that URI. also, when choosing the profile URI, it might be a good idea to stick to the naming advice from the XML Namespaces section on Comparing URI References: Because of the risk of confusion between URIs that would be equivalent if dereferenced, the use of %-escaped characters in [...] names is strongly discouraged.

providing documentation at a profile URI is a good idea, but there is no standard what to provide there. it probably makes sense to follow the pattern of XML namespace URI pages such as the XSLT namespace, which links to a variety of helpful resources, so that developers accessing the URI (hopefully) find what they are looking for.

What Now?

now that profiles can officially be used in both content and HTTP, it would be great to see services taking advantage of them. media types are not such a great design for what we're doing on the web today and one day could be redesigned to better fit today's needs. but they are one of the most fundamental aspects of not only the web, but many other internet technologies (such as email) as well, so it's unlikely that they will change radically anytime soon.

the idea of profiles is simple and fits into the architecture and limitations of today's standards and tooling. the hope is that by using profiles, resources on the web will become a little more self-describing, and it will be easier to properly design scenarios where services are extending others, but still want to maintain a working relationship with the service ecosystem they are building on.

personally, adding profile support to atom is the next step in the grand plan, but other than that, hopefully other representations and media types will start supporting profiles as well. it's very simple: mint a URI, ideally provide helpful information at that URI, and then start using this profile URI in your representations and maybe in media types as well. if you have any questions about how to best do this for a specific representation and/or media type, please feel free to comment or send me an email.

Github Fork Etiquette

this is a brief post about how to contribute to a github repository through the fork and pull request method. while there are a lot of short intros and how-to pages out there, none had the right mix of commands and workflow for me, so i am writing this in part simply as my own cheat sheet.

keep in mind that this is just one way of doing this, and there are many other methods and workflows that work equally well (and may work better for your use case).

the main idea is to never ever commit to the master branch in your fork, and only use it as a reference for the upstream changes. all changes are made in topic branches, then you're rebasing those onto the master branch, and then you're creating pull requests. if you are working on multiple issues, simply create multiple topic branches off the master branch.

as a starting point, fork the repository you want to start contributing to. after forking, you need to clone to your local machine:

git clone https://github.com/yourname/repo.git

now your cloned fork is ready to work on, but it is disconnected from the original. the convention is to add a remote called upstream, which you can later use to refer to the original repository:

cd repo
git remote add upstream https://github.com/original/repo.git

you can easily find the original repository's URI by visiting its home page on github. now that your fork is connected to the original repository, you should periodically pull the latest changes from the original:

git pull --ff-only upstream master

the --ff-only option ensures that in case you accidentally did make any changes to your fork's master branch, the pull will not work, and you first have to clean up. after a successful pull from upstream, make sure to push the changes:

git push origin master

the pull/push sequence is something you should repeat fairly frequently, in particular when you start working on a new issue. it makes sure you're up to date with what's going on in the original repository.

when you start working on an issue, create a branch off the master branch, and maybe give it a name that's helpful to the maintainer of the original repository:

git checkout -b yourname/issuename

a word of caution: when you are creating a new issue branch, make sure you are creating it from the master branch, i.e. make sure to first checkout the master branch, and then create your new branch.

all your changes now should be committed to that branch, which now nicely isolates your changes. however, should the master branch move forward, your branch now may start getting out of sync. this is something you need to address before creating a pull request.

when you want to submit your changes, first pull the latest changes from the upstream master, like shown above. now your fork's master branch reflects the upstream master branch. now you need to rebase your branch, so that it is based off this newer version of the master branch (make sure you are in the right branch when doing this):

git rebase master

now you are at the point where your branch looks as if you have made all the changes against the latest version of the upstream master. now you can create a pull request and review the changes before you actually submit it. make sure to create a pull request with the base being the original repository's master, and the head being your fork's topic branch.

once the pull request has been accepted, the changes become part of the original's master branch, which means the next time you pull from upstream, you should be getting your own changes (now part of the original master branch). this also means that the branch you've created does not need to exist anymore, and you can safely remove it:

git branch -D yourname/issuename

that's it. what's not covered here are two issues that also may be relevant:

• rebasing may lead to conflicts, which have to be resolved. git will lead you through that process, but essentially you are paying the price to be able to create one neatly self-contained pull request.
• when you have created many commits for your branch, you could reduce the number of commit by choosing --interactive for the rebase command. this will allow you to squash commits before creating the pull request.

maybe these are good topics for a second installment, but for now, this basic workflow already works well enough. corrections and comments are of course very welcome!

Open Spatial Services

is location a web concept (yet)? not really so far, even though of course there are many location-based services (LBS) out there. but many are trapped in specific services such as google maps or foursquare, often for business reasons, sometimes also for technical reasons. there still needs work to be done to turn location and LBS into first-class citizens on the web. as an illustration, this week brought two interesting developments:

garmin is one of the biggest makers of GPS devices. in the consumer market, they have been plagued by what i call the sony syndrome: really great hardware, but both the software and the service offerings backing it are mediocre at best, or just pathetic (other companies afflicted by this syndrome are suunto and motorola). case in point are the new online capable edge GPS devices. i have an edge 800 for my bike and always thought how cool it would be to pair it with strava and get alerts of upcoming segments. the new edge devices are now online capable by pairing them via bluetooth with your smartphone, but get this: the only site on the web they can talk to is garmin's own connect service. yes, that's apparently what garmin thinks online-capable means.

seeing something like this always is interesting to analyze: is it for business reasons, or because of a lack of vision? in this case it must be the latter, because garmin is not making money off of connect (which also happens to be one of the not-so-stellar services for sharing GPS-based activity data out there).

exhibit 2: the fabulous raceshape.com service recently launched a strava-based heatmap service, where users can create heatmaps of all of their activities. it's fascinating, because you can see hundreds of rides and runs on one simple map. the map image shown here is a screenshot of a full interactive version where raceshape has generated thousands of tiles which summarize three years of GPS track data.

now imagine a GPS device you can take with you, that at each and every turn can show you where you went before, and where you might see new things (or simply switch to the global heatmap to see where people in general go and don't go). by pulling the heatmap into a GPS device, you could do this and whatever else people come up with as ideas. you would have an open ecosystem for ideas around location, many people could build specialized LBS sites, and since garmin is not making any money with connect itself, they wouldn't lose money if people ignore connect's mediocre and limited services, and instead would subscribe to their favorite services. recent bird sightings. geocaching. places of historical interest. botanical guides. you name it, just let somebody build it, plug it into the ecosystem, and there's one more incentive to buy a garmin for a previously indifferent group of people.

what would it take to make this happen, apart from more forward-looking software and service strategies one the part of hardware specialists? an open architecture to build LBS that allow devices to dynamically subscribe to and pull data in from LBS in a standardized way. in our tiled feeds work, we have developed such an architecture, which interestingly is based on the model that raceshape is using now: tiles for various zoom levels to represent LBS features. check out our demo app with a couple of (hardcoded) sample LBS providers, and all of this is completely decentralized. the image shown here is using my iPhone locations (pulled from the phone before apple fixed this particular privacy snafu).

imagine being able to simple configure a couple of URIs as LBS providers, powering up the GPS, pairing it with the smartphone, and then the information you are interested in is shown right in front of you. you could even cache this when you know where you're going to be, because it's all REST-based and thus LBS tiles can be requested and cached on the device, if you're not going to be online all of the time.

my newest hope for a company getting it is leikr, the OSM-based GPS watch just funded through kickstarter. maybe if they have enough strategic vision to identify a competitive edge, they will recognize this opportunity that is still being left wide open by the incumbents. currently, they seem to plan to be pretty tightly coupled to endomondo, which to some extent is understandable (for the activity management part). however, enabling LBS to be freely pulled into the watch could be the great differentiator, and it remains to be seen which company in this space will first understand that services matter, ecosystems matter, and openness matters.

Structuring URI Templates

after musing about how to better combine URI Template Descriptions, mike amundsen responded rather fittingly with getting fancy, are we? while it may not be strictly required to make URI Templates composable via the description mechanism, it probably would be a nice capability to have.

as a scenario, take the example description from the current draft: the idea is to define some more advanced paging mechanism than just RFC5005-style links, and this idea very likely should be consistently used in services that are exposing various feeds. but in many cases, these feeds might have additional variables to specify, and then the question is how to best model, manage, and expose this.

the interesting question is whether this should and/or could be supported by Template Descriptions. let's assume that the paging model of feeds is as described in the example description (using pagesize and page variables).

one rather simple way to go would be the syntactic route. if somebody wanted to make this capability reusable, the template description for this would need to specify something along the lines of {pagesize,page}. unfortunately, URI Template expressions cannot be nested, which means that we cannot easily use this in a Template Description, add some linking concept across Template Descriptions, and then let a another URI Template say {?paging,color}, with the paging variable referring to the Template Description containing {pagesize,page}.

this problem could be avoided by using {?pagesize,page} in the paging template, and then using {paging}{&color} for the example given above, but this breaks down if we want to reuse two templates in another one: they would need to be defined in advance with either ? and &, which would make things rather brittle.

one possible solution to this problem would be to stick with {pagesize,page} as the reusable paging template, and add a processing rule that would say that if templates are referenced, and the included template is an expression (i.e., delimited by { and } with no other braces are in it), then the inclusion process strips those braces.

in this case, we could use {pagesize,page} as the paging template, and the example from above would use {?paging,page}, with paging being defined as an include variable. eliminating all XML fluff, we would end up with something like this for the paging template (let's call this paging.xml:

<td hreft="{pagesize,page}">
   <variable name="pagesize" concept-uri="http://example.com/feedpaging/pagesize"/>
   <variable name="page" concept-uri="http://example.com/feedpaging/page"/>
</td>

and then a template using this would look similar to this:

<td hreft="http://example.com/colorfeed{?paging,color}">
   <variable name="paging" template="paging.xml"/>
   <variable name="color" concept-uri="http://example.com/color"/>
</td>

there probably are many different ways in which this could be achieved, and one rather different approach would be to always reuse variables, instead of whole templates. that would change the dynamics of the mechanism, though, because in the above example, paging.xml could be updated with a third variable, and then all referencing descriptions would expose that additional variable without any need to change them.

the design presented here is mostly driven by the use cases we have for our internal designs so far, which revolve around an annoying number of variations of feeds, with a lot of concept reuse across them, and currently no ability to manage this without copious copy/paste, which always is a bad sign. i'd be very interested to hear about other use cases in services that use URI Templates, and whether they have invented their own reusability concepts, or might be able to use the design presented here (or a variation of it).

Templates Descriptions are a -00 draft and there are several open issues, and the problem described here has only made it into the open issues section for -01. anybody using URI Templates, please share your experience with how you're managing this currently, and whether something similar to the design presented here might be able to make this management problem easier. thanks!