dretblog

Stimulus Feed Details

after blogging about Stimulus Feeds Done Right, i received a couple of comments saying that the post was mostly complaining about the Initial Implementing Guidance for the American Recovery and Reinvestment Act guidelines, but not really saying what to improve specifically. well said, i had to admit, so here is a more concrete approach at how to do stimulus feeds right.

• Finding Stimulus Feeds is important, and instead of setting up a central registry (which still might be a good idea), it should be possible to discover stimulus feeds in a predictable way. page 58 requires all agencies to set up recovery web pages at http://agency.gov/recovery, and there are a number of requirements for these pages. feed autodiscovery should be added as a requirement for this web page, so that going to a agency's http://agency.gov/recovery page and searching for a linked feed is a reliable method for discovering an agency's stimulus feed.
• the guidelines seem to require three feeds, the major communications feed, the formula block grant allocation feed, and the weekly report feed. it is not quite clear, though, whether agencies are free to have just one feed carrying three types of entries, or whether three separate feeds are required. from an information dissemination point of view, it would probably be better to require three separate feeds (and maybe provide a fourth one aggregating all three). in that case, feed discovery would have to be more specific and make sure that all three feeds can be discovered from an agency's recovery web page.
• a feed can contain the information (in what format? more on that later) or can point to a file containing the information (in what format? more on that later). if the feed points to a file, how is it supposed to do this? it could do so via atom:link/@src, atom:content/@src, atom:content[@type='xhtml']//html:a/@href, or, god forbid, atom:content[@type='html']//html:a/@href. ideally, links should not be allowed at all, but if they are, they should be required to use atom:content/@src.
• agencies are allowed to provide no feeds at all, they can just publish the files via predefined URI structures. however, since it is unclear if and how a web server will make the directory of such a set of files available, it will be impossible to reliably discover and retrieve files with such an approach. at the very least, agencies incapable of producing feeds should be required to provide machine-readable directories on all levels of all posted files at the suggested URIs. there should be a required and machine-readable format for these directories.
• the guidelines specify the feed format as preferred: Atom 1.0, acceptable: RSS, and do not even mention RSS versions. this means that feeds can use 10 different formats, Atom plus the 9 different RSS variants. while it would be best to only allow Atom, it would be good to at least limit RSS to specific versions, such as RSS 2.0 (which probably should be specified to be RSS 2.01 rev 2, according to mark pilgrim's versioning).
• since the feeds are allowed to contain only links instead of the actual data (essentially turning them into a notification mechanism), there also is a file format for how to publish the information. unfortunately, the templates for this file format are not publicly available. it would be important for this file format to be easily accessible with generally available tools, which limits the choices to plain text and XML. if XML is used, it should be plain XML and not some complex format. if more complex formats such as XBRL are required, XSLT transforms for up- and down-translation between these more complex formats and a plain XML format should be provided.
• starting on page 55, the guidelines mention data elements the feeds should include. it is unclear how these should be included (if the feed points to a file, the file format probably contains those data elements, but if the feed is supposed to contain those data elements, there must be a feed-oriented syntax for them). in addition to that, the datatypes seem to be copied straight from a SQL database scheme. there should be one required syntax for these data elements, and it should be based on XSD datatypes. implementation variants for this syntax are plain XML or microformats; if microformats are chosen, RDFa should be used.

this is not all there is to it (and without seeing the templates for the files it is hard to make specific recommendations for that part), but it should point in the right direction. i really hope that the current suboptimal guidelines will not make people believe that feeds necessarily result in a rather unpredictable hodgepodge of how information dissemination is organized. it actually does not take that much effort to come up with guidelines that create a robust and predictable landscape, and providing test data, test tools, and validators would help agencies to conform to those guidelines.

the good news is that the current guidelines (published 2/18/2009) specifically mention that more detailed guidance will be published within 30-60 days. which means that some of the above issues may still show up in the final guidance. this also means that it makes sense for agencies to wait until the more detailed guidelines become available.

Atom Feed Autodiscovery

even though most people expect to find feeds made accessible in a browser's address bar, this method is still not used by all feed providers. and surprisingly, even though there were two attempts to standardize how this feed autodiscovery should work (draft-ietf-atompub-autodiscovery which was replaced by draft-snell-atompub-autodiscovery), neither of these attempts was finalized as an RFC.

HTML5 defines a feed relationship for the link element, but HTML5 probably is a long way from being finished, and does not retroactively define anything for earlier HTML versions. so it seems as if there currently is no standard for how to do feed autodiscovery; but of course there are many web pages (for example here, here, here, and here) describing the established method that seems to be well supported by most browsers.

since feed autodiscovery would be an important part of the way how feeds should be used on the web, i have added the expired draft to the Atom Landscape Overview, and i am curious to hear what the Atom community has to say about the current state of affairs.

[[ march 1, 2009: there is an active IETF draft for a link relation registry, which i missed when i wrote that post. ]]

Finding Stimulus Feeds

while it is interesting to think about what it would take to implement the stimulus feed architecture in a more useful way, it is also interesting to think about where to actually find these feeds once agencies start publishing them. the guidelines require agencies to also send email updates to recoveryupdates@gsa.gov, so they probably count on emails as the information flow. but where do stimulus mashup builders go to find the stimulus feeds?

we thought about this problem a while ago, when we thought about the general problem of how to get subscriptions out of and into feed readers in an interoperable way. we came up with the concept of feed feeds, and more recently there has been a proposal about managing feeds using feeds (which is mostly about using AtomPub for feed management) as part of the evolving atom ladscape of specifications.

in both cases, the idea is that a collection of feeds (in that case the collection of feeds publishing data on stimulus spending) can be viewed as a feed itself, allowing people, for example, to subscribe to a feed where they can see whenever a new stimulus feed is published by some agency. this metafeed or feed feed would be the ideal starting point for any exploration of the evolving set of stimulus feeds, and while i am not so sure whether it will be actually published as a feed (i hope it will, anybody interested in some community effort around this?), i certainly hope that somebody somewhere somehow will keep track of all those feeds.

currently, the guidelines make no provision whatsoever how such a feed has to be advertised or registered, so in theory it is possible that agencies publish feeds that nobody knows about, because nobody know the feeds' URIs.

once we have a stimulus feed feed, we can start using that for all kinds of activities around managing these feeds, for example tagging them, maybe validating them, and more generally subscribing to them and using their entries to actually track the stimulus spending. but wait, they do not actually contain the data, they just make public that new data has been made available somewhere...

btw, the guidelines say a feed can point to a file, but what exactly does that mean? using atom:link? or html:a? and once i know how the linking works, i am still curious about the format of those templates non-government people can not get to...

Stimulus Feeds Done Right

one of the interesting developments in the highly anticipated more tech-savvy obama administration is that the Initial Implementing Guidance for the American Recovery and Reinvestment Act explicitly mention feeds (page 54: preferred: Atom 1.0, acceptable: RSS) as a way of how information on spending has to be made available. it is certainly nice to see a government recognizing feeds as an important mechanism for information dissemination.

however, it seems that the feed is mainly intended as a trigger and not as the actual container of useful data. it seems that the actual data is expected to be made available in some format made available as a template at https://max.omb.gov/community/x/doC2Dw, but since this is a protected site, it is not even possible to find out what data format this template is using. in an ideal world, it would be XML, but something tells me that we are not yet living in an ideal world.

starting on page 55, the document becomes a bit more confusing. while there is a rather long list of interesting data elements that certainly would be interesting to aggregate across agencies, there is no actual syntax defined, and the field types look a lot like SQL datatypes. so while it would be nice to get all this information in feeds, my guess is that we will get feeds that will point to templates that will contain the data, and these templates probably will use some aggregation-unfriendly formats such as PDF or Excel. so in theory we get feeds, but in practice we probably only get feeds as a trigger and then have to deal with non-feed data.

what i am thinking about is creating a platform that aggregates all agencies' feeds, consumes the templates as new data becomes available, parses them, and republishes all information as well-designed feeds where all the data elements on pages 55-57 are encoded as Atom extension elements. this could even go a step further and for example geocode address information in the data elements and add GeoRSS to the republished feeds, allowing map-based applications to plot grant allocations on a map.

[[ february 26, 2009: to make this commetary a bit more constructive, here are some details of how to get the stimulus feed architecture right. ]]

APIs Considered Harmful

talking about APIs and web services as being almost synonymous becomes increasingly popular. it is good to see the understanding of the web being extended from a simple set of web pages to the information system that it actually is. starting from the notion that your website is your API, the idea of web services can be incrementally extended from well-designed web sites, microformats, feeds, feeds carrying extensions such as GeoRSS or application-specific extensions, all the way to full-blown RESTful services.

what i am asking myself is whether the notion of an Application Programming Interface by itself is not too much overloaded with RPC and similar functional styles, actually hurting the ability of people to build well-designed web applications. for example, looking at the variety of twitter clients for the iPhone, it amazes me that some of them simply stop working at all when twitter is unreachable. they take the idea of an API much too far, considering themselves as mere extensions of the twitter program running somewhere else, and they are perfectly content to completely fail when twitter or the network fails (tweetie is one of these ill-behaving clients). a well-designed twitter client, on the other hand, caches twitter data and displays whatever it has locally, and in the background requests updates from twitter, and then updates the display if updates were available (TwitterFon is such a well-behaving client).

interestingly, when somebody i am following recently updated their profile picture, TwitterFon promptly updated it as well, whereas tweetie used the old picture for a long period of time. this was another indication that TwitterFon properly handles twitter data as resources and treats them according to HTTP, whereas tweetie is mostly built as an RPC-style client with no clean HTTP handling. (of course i don't really know how well both implement and respect HTTP, but the difference in behavior is obvious.)

generally speaking, the widely used API metaphor for web services may lead too many programmers to believe that they just write the client part of a client/server system, and that this client part is so tightly coupled that any server or network failure will take down the client anyway. well-designed web services should be the very opposite, so that service providers and consumers are loosely coupled, and consumers have the responsibility (and the opportunity!) to deal with failures. it has been reiterated again and again that middleware-style programming models are not doing programmers a favor in their attempt to hide distribution for large-scale systems such as the web; distribution is an inherent property of such a scenario, and behavior implemented by providers and consumers must take failure into account.

while the SOAP vs. REST debate will not stop anytime soon, it might make sense for REST advocates to stop using the A-word, and start talking about services instead. the notion of open services may be more RESTful and less loaded with RPC baggage, and maybe the A-word should better be reserved for scenarios where people are trying to implement one giant distributed program, instead of a loosely coupled set of cooperating services.

Mobile Platforms and Services

with the New York Times reporting about additional traction for Microsoft on LG devices, the more interesting question is how relevant smartphone OSs will be in the long run. the iPhone has definitely stirred up the marketplace, ending the unfortunate situation for end users where installing apps on a phone was something no mortal could figure out. plus it was the first really good browser on a phone, unfortunately hampered by the fact that the touchscreen interface renders a substantial number of Ajax applications unusable.

the current smartphone OS ranking is Symbian, BlackBerry, Windows Mobile, iPhone, and the rest (mostly Android, i guess; what happened to Openmoko?). with Palm ditching their own Palm OS in favor of Windows Mobile, it seems like the first victim of some market consolidation is already established. while it is important for the phone makers and the OS makers which OS will eventually win and which ones will disappear, for service providers this may not be as important as it seems, and more importantly, it should not be as important as many think it is.

application runtimes for mobile applications can be the native OSs listed above, but they can also be runtimes layered on top of these OSs, the most important ones being Java ME (a.k.a. J2ME), JavaFX, Flash Lite, and Ajax. this mirrors the development of the web in general, where Java applets, Flash, Ajax and recently Silverlight and AIR are used as application development platforms. Google's Chrome is an attempt to also get into that space by extending the Ajax platform with some missing but essential features for standalone apps.

for developers of mobile and location-based apps, this landscape can be confusing. for example, the iPhone is very popular, but the platform has no support for map-based apps, whereas Android has pretty good built-in support, but is not very popular. you could also implement such an app using Ajax, but then it will not be usable in the iPhone's browser because of its accessibility problems. so how to proceed?

the most important thing to look at is the fact that apps are just interfaces to services, so focus on the service and worry about the front-end later. figure out what you need from the platform to deliver your service. what would make your life easier on the platform? XML support? HTTP support with solid caching? a universal feed parser? XSLT? XSLT 2.0? mapping? access to the device location? designing and implementing location-based services is not an easy task in today's landscape, and it is made harder by the fact that it is hard to predict the landscape in a year, and almost impossible to predict it in two years. which means that instead of focusing on eye candy and platform-specific goodies, focus on the service you are designing, and make sure you can move with the landscape as it is shifting.

i am still hoping to find some comprehensive comparison of all the implementation platforms described above, taking into account the many different requirements that typical mobile apps have today. it would be hard work, but it also would be highly valuable for anybody trying to make a move in that landscape. should i built the fancy iPhone app which costs me a lot in development time? why not just build an Ajax app and produce an iPhone-optimized variant of it? such a matrix would also be pretty interesting for the vendors, i guess, to better figure out where their platforms are lagging behind, and where there is uncharted territory where they could be the first movers.

for service providers, on the other hand, instead of building just web apps, it becomes increasingly important to built web services (not of the SOAP variety, i am more talking about REST here), and to make sure this service can be easily used by a variety of front-ends. twitter probably is a great example for how powerful this model can be; their front-end (the web site) is very basic and minimalistic, but their simple and well-designed service has encouraged a flood of twitter implementations on all imaginable platforms. only about 30% of all users access twitter through the twitter-provided front-end (the web site), the other 70% use it with other front-ends built by third-party developers. i am pretty confident this would not have happened if twitter had provided a WSDL-based API accessible through SOAP.

all of this boils down to the following: for mobile apps these days, focus on the service, provide a well-designed service, built a front-end on a platform that fits your requirements, and make sure you can switch front-end platforms should the mobile landscape make it necessary.

[[ update 2/18/2009: tim bray just published a post covering Palm's webOS as well, which i did not mention at all. ]]

Sitemap Names

in our recent investigation of web site metadata, one finding was that only 6.3% of all sites were publishing a sitemap (looking at the web's most popular 100'000 domains). strictly speaking, this is correct, but since sitemaps may exist without being advertised in a robots.txt, this is the lower boundary of the actual sitemaps available on the web.

one simple way to test this hypothesis would be to just test for sitemaps, if there are popular names for sitemap documents. as a first step, i decided to run brief statistics on our dataset. we found that out of our 100'000 domains, 44'832 were serving robots.txt files, and of those 44'832 files, 6'268 pointed to sitemap files. since one robots.txt can point to more than one sitemap file, we ended up with finding the names of 10'029 sitemaps. (read our technical report for more details about this study.)

stripping the prefixes and only looking at the path of the sitemap URIs, this dataset produced the following list of the 10 most popular sitemap URI paths:

1. 25.07%: /sitemap.xml
2. 8.38%: /feeds/posts/default?orderby=updated
3. 6.75%: /sitemap.xml.gz
4. 3.37%: /sitemap_index.xml
5. 1.91%: /s2/sitemaps/profiles-sitemap.xml
6. 1.48%: /sitemap.php
7. 1.24%: /sitemap_index.xml.gz
8. 1.05%: /vb/sitemap_index.xml.gz
9. 0.66%: /sitemapindex.xml
10. 0.45%: /sitemap.gz

so it does look as if /sitemaps.xml would be a good candidate to look for as a general strategy. it will actually be interesting to look at how many sites that do not advertise a sitemap in their robots.txt file still have one. that's something i have to try out next, going through all 100'000 domains again and ping each site for the most popular sitemap files names.

Location Privacy

it's only a coincidence, but it was interesting to read about a new service by Google that tracks user locations and about MySpace handing over a lot of private information in response to a subpoena. using Google's new Google Latitude service, people can update their location via a variety of ways (currently supported are Android, iPhone, BlackBerry, Windows Mobile, Symbian S60, and J2ME), and then this information is transmitted to Google, where it can be accessed by others. of course, the service has privacy controls (it's basically the same service as Yahoo! Fire Eagle, but more targeted at users than at applications), but this only controls what's exposed to others, not what Google does with the data.

the mobile privacy policy does not say anything about how long Google stores the location data, and surprisingly, the full privacy policy also does not say anything about that. does that mean Google will store my location data forever or as long as they please? remember, even though Google mostly uses this data internally, they might be forced to release this information. and your location information might just cause some collateral damage, when your location track is being released as a part of some investigation and you're not really guilty of anything, but you still were in a place where you were not supposed to be...

building centralized applications always is technically easy (as long as they scale reasonably), and also very attractive for the party doing the centralization. in particular if it is information that is very valuable for data mining and user profiling. but as a user, the important question is with how much non-monetary currency i am willing to pay for a service; in Google's case, the currency often is privacy.

for me as an iPhone user, AT&T has my location trail anyway, and i am sure they will also turn it over if somebody shows up with a subpoena. i always wanted to be able to just switch off the phone part of my iPhone, so that i can still use the Wi-Fi without being hooked up to AT&T all the time. with the new Google service, iPod touch users now also can be traced (not with GPS or cell tower positioning, though). seeing that most people are carrying there phones around all the time, this might be the tipping point where people actually realize how much privacy is eroding with all these great free services, but i has been hoping this a number of times already...

Web vs. Web Architecture

when we recently pitched principles of web architecture and more specifically feed-oriented technologies for projects around sensor data, we also got some negative feedback. surprisingly, it was not only feedback along the well-known web technologies are not appropriate to do the heavy lifting of serious enterprise-level IT stereotype (which always dazzles me as a generic answer when you haven't looked at the requirements yet), but there also was another recurring theme.

this other theme was that web technologies are dangerous because they will allow terrorists to monitor sensor data and do bad things. this one completely dumbfounded me; it assumes that less accessible technologies are more secure, following the well-known security through obscurity pattern. i don't think that some application-specific WSDL will necessarily stop any terrorist from accessing information or a service, and i certainly hope that critical sensor data resources are protected by better security measures than WSDL's interoperability problems.

while this specific argument is probably easy to counter in a discussion, it made me realize how many people still confound web architecture with the web. web architecture not only allows you to understand how the web works; it also allows you to study why it works, and what differences there are between the web and more traditional approaches to building large-scale information systems.

web architecture as an architectural approach simply informs the design of an IT architecture and is probably a good thing to take into consideration when designing IT architectures for information systems requiring massive scalability and loose coupling. this has not necessarily anything to do with making data or services openly available on the web, and might not even have anything to do with making them available on the web at all. it is all about designing and building loosely coupled systems which allow data and services to be used and recombined in creative, unforeseen, and flexible ways.

On Sitemaps

when working with Web Site Metadata, it is unavoidable to soon run into the sitemaps format, defined by Google. the first important thing to learn about this format is that despite its name, it is not a format for sitemaps. it is simply a format for URI lists so that web masters can steer crawlers to web pages on their sites.

another interesting observation about the format is that it is not all that well specified. it contains a number of examples and that's about it. many questions are left open, and the most important question for such a format, the one of extensibility, is left wide open.

hidden somewhere deep on Google's site, an extension called geo sitemaps is described, and on another page it is defined in a little more detail. unfortunately, the extension is defined with about the same rigor as the format itself, which means by some examples and little else. essentially, the format seems to do little else other than specifying the media type of a URI in a sitemap, but it does so in a very complicated way. here is an example for one URI:

<url>
 <loc>http://www.example.com/example1.kml</loc>
 <geo:geo>
  <geo:format>kml</geo:format>
 </geo:geo>
</url>

the interesting question is whether this is so verbose because sitemaps extensions can only use elements, but not attributes. from an XML perspective, something like this would make more sense:

<url>
 <loc geo:format="kml">http://www.example.com/example1.kml</loc>
</url>

but since the whole extension only seems to be about communicating media types anyway, why using something that would be a general mechanism for communicating media types, and use media types as they are used on the web?

<url>
 <loc mime:type="application/vnd.google-earth.kml+xml">http://www.example.com/example1.kml</loc>
</url>

this way, such a sitemaps extension would be short, based on existing standards, and be reusable for all kinds of scenarios where web sites want to publish the media types of resources they are providing.

on a more interesting level, it would be possible to think about spatial navigational sitemaps in the same way as we are currently thinking about navigational sitemaps. our goal is to design a sitemaps extension so that web sites can publish their navigational structure (what most people would probably think of as a sitemap in the first place). another extension could provide support for publishing the spatial navigational structure of a site. this could either be published as a specialized sitemap, or it could be embedded into the site's home page; i think the sitemap approach would be cleaner, easier to manage, and more extensible.