dretblog

Data, Models, Metamodels, Cosmologies

when i recently tweeted about the fact that the semantic web or linked data (choose your preferred brand name) had this troubling attitude of one metamodel to rule them all, i got some feedback saying that this was wrong. it's not, and i guess the reason for the confusion in that space is that data, models, metamodels, and cosmologies are somehow hard to grasp, so here is my brief attempt at displaying all of them in one handy overview (and the selection of items in that overview is definitely not complete and simply provided for illustrative reasons).

Cosmology	Key/Value Pairs	Unordered Tables		Ordered Trees		Generalized Graphs
Metamodel	?	Entity-Relationship (ER)	Relational (SQL)	SGML	XML Infoset XDM	RDF
Model	?	ER Graph	SQL DDL	DTD	DTD XSD RELAX-NG	RDFS OWL
Data	CSV Bigtable SimpleDB	?	RDBMS	SGML	XML EXI	RDF/XML N3 Turtle

what does this mean, in particular the weird cosmology part? it just starts with the observation that people usually start with an application domain, and certain recurrent features/patterns in that domain they want to see supported in their models and the tools they work with. which is the reason why the hierarchical databases of the early days were replaced by the more appropriate tabular structures provided by ER and SQL. for the same reason, document processing always had its own world, because working with documents cannot be done effectively in unordered tabular structures, so this is where the trees come in. they exist in various flavors, but the main idea is that you have a tree and that the tree is ordered, which maps well to document structures.

as long as you stay within the same cosmology, mappings usually work reasonably well, and working with the data can be done in at least comparable ways. there may be differences in features and tools and languages, but the big picture of how the data is structured is the same. sometimes there are special features of metamodels, for example whether a model is even required: SQL data always needs a database to live in, whereas XML and RDF can happily live without a specific model (i.e., schema) and can still be used.

things become much more complicated if you want to take data from one cosmology, and map it to another cosmology. this is always possible, but for non-trivial data it often results in a severe mismatch between the application model and the underlying assumptions of the metamodel. it also often means that the tools and languages in the new environment are not appropriate anymore. imagine representing a complex XML document in RDF (i am sure somewhere there is some RDFS for representing XML). it can be done, sure, but compared to the tools provided with XSD and XQuery and XSLT, working with that data will become much more complicated and very ineffective (and also probably very inefficient in terms of processing times when considering large document collections), because the inherent ordered tree structure of the data is not supported by the underlying metamodel anymore.

some people may think that because it is always possible to define mappings between any metamodels, that implies that there is one overarching metamodel that spans all of the above cosmologies (as a layer on top of all cosmologies), and since RDF is the metamodel with the fewest built-in constraints, the claim is often made by the semantic web community that RDF can be that one metamodel. however, this ignores the fact that there is a reason why there are different cosmologies and metamodels and model languages and tools: these have evolved over time to deal with certain classes of data, and they usually do a good job in their domains. RDF has had success because it, too, deals with a certain class of data, it is typically being used for metadata. metadata, even though it's only loosely defined, typically is not data with complex structures, and thus using a simple model such as RDF works well.

the motivation for this post is simply to show that RDF is not the metamodel to end all metamodels. it is one metamodel, and simply a new addition to the existing multitude of cosmologies, metamodels, and models that have been around for a long time, and it has found an application area for which it is well-suited. claiming that its simplicity (i.e., the ability to map other cosmologies to the RDF cosmology) means that all data out there can be appropriately represented and handled as RDF ignores the fact that in the end, it's not important that it's possible to do something, but only how effective you can be when you're doing it: if your job is processing large collections of documents, what models, languages and tools make you most productive at getting your job done.

Retrofitting RETRO

ICWE 2009 was an interesting conference, and one of the good parts of it was the fact that even though it still has a substantial model-driven engineering bias, there are signs that other communities start recognizing the conference as well. i hope this trend will continue, and it definitely will be interesting to compare the programs of the 2009 and the 2010 conferences (btw, ICWE 2010 will be held in vienna, hosted by TU Vienna).

one of the people at ICWE was alexandros marinos, one of the creators of RETRO. RETRO recently caused some discussions in the REST communities because it tries to provide a RESTful approach to transactions, which often are more perceived as being in the realm of RPC/middleware oriented approaches to web services. i guess, if you want to use REST and transactions, there are three basic approaches:

• leave the transaction semantics in the application, so that clients just need to know what is going on. this is not loosely coupled, and makes it impossible to provide tools and support for transactions.
• introduce new methods. this is what WebDAV did by extending HTTP with new methods. this is a very clean approach, but also one which makes adoption rather hard.
• introduce some overlay of links and/or resources that represent the transactions. the magic .../lock resource is one such thing, but as long as there is no way how it can be discovered, this also falls into the first category. otherwise, there have to be ways of how links and/or resources can expose their transaction semantics.

the RETRO approach falls squarely into the last category, and personally, that is the category that i prefer. there are three issues i have with the (admittedly very early) draft:

• it seems to me that not all of the resources introduced in the model are strictly necessary, and least not as publicly available resources. it would be interesting to look at the minimal core, and define all other resources as optional.
• RETRO introduces media types and suggests to include a certain XML fragment in lockable resources. it would probably be more consistent and more extensible to just require certain vocabularies be present in the resource representations, so that each service could (if it wanted) use its own schema, importing the RETRO schema. RETRO could still provide default schema to use for those services that do not want to define their own schemas, but link discovery should be based on namespaces and imported schemas, not on media types.
• changing URIs from transaction-contained resources to the eventual URI is a pretty major operation. at least, there should (or maybe even MUST) be an xml:base in the resource that always sets its base URI to the final URI. and generally, i am asking myself whether this identity switch is not a potential source of confusion and/or errors. would there be other ways of doing this? and is it even necessary to have these temporary resources, or are they also optional and not part of the core model?

but even though i think the model could be improved, i really like the direction, and i am particularly interested to investigate potential synergies with LDX, which currently mainly is an idea and a first sketch, but will get more attention soon. i still think that REST's idea of placing hyperlinks in the core of the interaction model demands something like LDX: a way of how to make statements about links and their semantics (in the sense of link roles). combining RETRO sans media types and using LDX as the way to specify RETRO's interconnected transaction resources could be a nicely flexible way of allowing services to use their own media types (as long as they're XML), without sacrificing the discoverability of the RESTful transaction links.

Link Discovery for XML

one of the main principles of REST is what unfortunately most often is called Hypermedia as the Engine of Application State (HATEOAS), which must be the worst acronym ever invented. regardless of that, this is one of the central REST constraints, and also one that is often ignored, with services using no discoverable links at all, or using URI templates and assuming that this already is RESTful. it can be, but it often isn't, because even if there was a stable standard for URI templates (and there isn't), it still would be impossible to know how to substitute the template variables to get URIs of available resources. which is why RESTful services should include links in resources, so that applications can simply follow those links.

in web architecture, REST's links are found by knowing a resource's media type, and knowing how to find the links in that format. the most popular linked formats on the web probably are (X)HTML and Atom, both of them define links and link semantics in their data format. however, since XML is such an popular syntax for many different content types, i have been toying with the idea of defining a generic XML mechanism for discovering links in XML (the only generic descriptions for XML available today are the built-in xml:lang, and the separate xml:id specification). notice that this is different from datatype mechanisms such as XSD, which are able to type content as a URI, but it is still not clear whether this URI is a link, and if so, what its semantics are (which i will call role from now on). as a starting point, here is a brief list of requirements:

• link discovery should work inline (information embedded in XML instances) and out-of-line (as an external description of how to find links in some XML).
• link discovery can be a perspective of a media type, i.e. different applications might have different ideas of how to find links in a given media type, or a subset of documents of a given media type.
• links can have roles, and are allowed to have more than one role. role names can be strings, qualified names, or URIs.
• the mechanism should rely on the smallest possible set of technologies, so that it can be easily implemented using widely deployed tools and technologies.
• there should be a simple syntax for fully describing the link discovery information, so that it can be serialized and exchanged.

starting from this simple and very likely incomplete set of requirements, here is my first stab at a mechanism for Link Discovery in XML (LDX). the main idea is that links are identified by XPath-based selectors (XSLT's pattern subset might be a good match here), that selected links are associated with roles, and that this mechanism can be applied to any XML-based media type. what would be the advantage of such a mechanism? a simple way to describe the links and link semantics of XML media types, the ability to build generic tools that can extract links from any XML-based media type, and the ability to better separate general REST functionality (what links can i follow in this resource?) from content-specific functionality (what does the content mean, and what does it mean to follow the links?). generally, such a mechanism could become part of a REST toolbox, and there are advantages for both service providers and consumers.

now on to the specifics. it's a bit harder to come up with a good inline format (HLink still might be remembered by some), so my first pre-alpha idea focuses on the out-of-line format. it's a simple set of selectors associated with 0-n roles. each selector selects an attribute or element content (interpreted as the string value), and this content is then recognized as a link with the associated roles. selectors probably should be able to be grouped and contextualized, so that for example some minimal LDX for XHTML might look like this:

<links xmlns:html="http://www.w3.org/1999/xhtml">
 <context match="html:html/html:head">
  <link match="html:link[@rel='stylesheet']/@href" role="stylesheet"/>
 </context>
 <context match="html:html/html:body">
  <link match=".//html:img/@src" role="image"/>
 </context>
</links>

this would allow to automatically extract all links to stylesheets and images from XHTML documents. the role names in this case are just strings, and there definitely has to be some better handling of non-string role names, in particular differentiating qualified names and URIs (which could, at least in theory, use CURIE syntax) might become a bit of a challenge.

namespaces (in the described media types) are a headache as usual, the eternal question being what to do with unprefixed names in selectors. use the LDX document's default namespace (à la XSD), or have a special attribute for setting the default namespace (à la XSLT)? i prefer the latter, but that's more syntax juggling than anything else.

the fun part is that because of the selectors, applications can even try to be smart and create LDX that works for the specific content they're concerned with. let's imagine a RESTful application using XHTML which contains links, and always contains a link to a product in the first column of each row of a specific table. using a selector such as table[@id='products']/tr/td[1]/a/@href, it would be possible to associate these links with a specific role, and with LDX support in a REST toolbox, finding all those links would be trivial. and the LDX description of the assumptions made about the content would be concise and declarative, so that it would be easy to understand the linking assumptions of the application.

most of LDX currently exists in my head only, and right now i am mostly wondering what others think of that idea. does this make sense from a RESTful point of view? would it be a worthwhile addition to a REST toolbox? maybe even to the REST landscape in general? and, assuming it does make some sense and i am tempted to start writing down a bit more of my ideas, should i start with a W3C or with an IETF template?

REST Programming Toolbox Requirements

REST is all the rage, and everybody claims they're doing REST. REST is a style of designing web services, and the question is how this style can be best supported by a programming environment. there are two possible perspectives of this problem. one is the more high-level view of how to best structure code that is built around the assumption of a uniform interface. a popular example for this is JAX-RS, released as JSR-311 last year, and implemented by the Restlet framework. this is not what i want to discuss here.

a different perspective is that of basic tools that might be useful for implementing RESTful services (in that case interpreting REST in its narrower sense of web architecture and its use of HTTP and URIs). in a recent blog post, marc hadley reports on Integrating Jersey and Abdera. the interesting thing is how his approach combines a high-level framework (JAX-RS), with more low-level tools for handling HTTP and content (in that case, feeds).

so looking at the low-level support one would like to see when programming RESTful services, what is required or useful? i came up with this list, which is probably incomplete, so any feedback about what else should go in here would be highly welcome!

• HTTP support: HTTP must be fully accessible, which means that all methods, headers, and status codes must be accessible to the application. if required, certain behavior might be enabled on request (such as following redirects), but this should be fully under the control of the application.
• HTTP encoding: since content in HTTP can be transmitted using chunked encoding or some transfer-encoding, it must be possible to either explicitly deal with these encodings, or to have this handled by the HTTP API.
• HTTP authentication: in many cases, HTTP authentication is required for gaining authenticated access to access-controlled resources. the HTTP API should support the predefined HTTP authentication methods (basic/digest), and should also have a plug-in mechanism so that other authentication methods can be used as well.
• caching: caching is one of the core features of how REST can be optimized. caching can be based on modification dates and/or on ETags. client-side caches can be shared or private caches. there must be built-in support for caching, API controls for handling shared and private caches, and there should be integration with a client-side shared cache for the client as a whole (imagine how useful it would be if all browsers on your computer were smart enough to share the same cache).
• URI parser: URI parsing is not trivial, and URIs are used everywhere. there should be URI prcessing support for parsing and resolving URIs. and depending on where URI templates are going (they are still under construction), support for them  might be very useful as well.
• language tag processing: since language tags are part of HTTP and can be used in HTTP mechanisms such as content negotiation, handling language tags is a integral part of HTTP. since language tags have a surprisingly complex logic behind them, support for parsing and matching languages is very useful.
• MIME type processing: similarly to language tags, MIME types (officially called media types) have a structure, are used in content negotiation, and often should be compared or matched. support for these operations should be part of a toolbox as well.
• XML tools: XML is the most widely used syntax for machine-readable data, so there should be a parser, validation support for maybe various schema dialects (DTD, XSD RELAX NG), selection via XPath, hopefully XPath 2.0 as the more powerful language, and finally more advanced XML processing might be supported by XML-centric languages such as XSLT or XQuery.
• JSON tools: many RESTful services support JSON, and sometimes only JSON. support for parsing JSON therefore also should be part of the toolbox.
• Atom tools: one of the most widely used formats for machine-readable data is Atom, and thus Atom support should be part of the toolbox, also supporting additional features which are part of the Atom landscape such as feed paging.
• AtomPub: AtomPub is one way of implementing a RESTful service providing write access to Atom-oriented services, and thus support for AtomPub (at least for the client side) might be useful as well.
• fault-tolerant parsing: many RESTful services access resources from a variety of sources. support for fault-tolerant parsing of HTML and feeds (in any of the RSS variants and in Atom) makes it possible to implement more robust applications.

while i am sure that this list is incomplete, i am really interested to better figure out what it really takes to provide a useful REST programming toolbox. currently, i am mostly interested to figure out what would be of general interest to most/many RESTful service developers.

REST and RDF Granularity

bob ducharme's recent post about Semantic web technology and humanities research and our recent WWW2009 tutorial talking about the route from SOA to REST (to be repeated in june at ICWE 2009) made me think about the fundamental tension between RDF-based semantic web approaches, and RESTful architectures.

it seems to me that the biggest problem of RESTful RDF is the lack of a well-defined notion of a document. one of the strengths of RDF is that you harvest triples from wherever you find them, take the graph, compute the closure of this graph under whatever schemas or ontologies you might have, and then process the resulting graph. on the semantic level this means you take whatever you can get as data, whatever you can get as reasoning mechanisms, and extend the data using inference. it is this general model of raw data and derived information that makes RDF so powerful.

the price you pay, however, is the loss of provenance and document boundaries. of course this can be overlaid using various mechanisms, but it is not a fundamental part of the RDF model, and it's easy to forget it or get it wrong.

this leads to problems when it comes to the rather document-centric model of REST, which talks about representations that need to be exchanged for enabling interactions. how do you use RDF, when the data you're working with for your application purpose is a mix of retrieved RDF and inferred information?

XML, on the other hand, has a rather static idea of a document. well, that's actually not entirely true, because XSD introduced the concept of the Post-Schema Validation Infoset (PSVI), which in a way is the same mix of source data (the original XML document) and inferred data (validation information, type annotations, and default values). type annotations can be regarded as a bonus that can make data processing much more robust and reliable. ironically, default values, however, are widely frowned upon in XSD best practices, because they tightly bind document processing to document validation, and that is largely perceived to be a Bad Idea.

and this is where it gets interesting: XML's very idea is that of enabling applications to GET XML and blindly process it by simply processing well-formed markup. RDF, on the other hand, is more rooted in the idea that the RDF you find blends into an existing set of other triples and schemas or ontologies. this latter model makes working with RDF more seamless and more convenient in an RDF-only world, because essentially, applications can ignore the fact that data had different origins, and they can work on what sometimes has been referred to as the giant global graph. it seems to me that this replicates the pattern of trying to hide distribution that underlies many of the middleware-inspired approaches to web services in a rather interesting way.

XML, on the other hand, makes it almost impossible to get this seamless view of the world, and requires applications to handle one XML document at a time, often steered by overlaid mechanisms such as XInclude at a very low level, or by following hyperlinks when it comes to RESTful data formats. what this means is that XML and this RESTful picture force you to work on the document level, and this also forces developers to explicitly deal with distribution, which means dealing with failures at the network or representation level.

so it seems as if RDF's virtues, the ability to transcend document boundaries and to seamlessly include data from all kinds of sources, become problematic when moving towards RESTful services, where coarse granularity and well-defined document concepts are an essential part of the architectural style.

this does not imply that RDF cannot be used for RESTful services and document-oriented scenarios; it simply means that XML makes it more natural (or you might say: less avoidable) for developers to deal with strict document boundaries, whereas in RDF, this requires special attention. i am really curious to find out whether somebody already came up with a set of best practices and design guidelines for building RESTful RDF applications, and this becomes particularly interesting when looking at issues such as SPARQL (and its basic model of accessing an RDF triple store through just one URI) and updates of RDF data.

it seems to me that the road towards a RESTful semantic web (if we take the RDF-centric view of that concept) is still not fully clear, and i am really curious to explore in more depth the fundamental tension between REST's things are diverse and distributed and you have to deal with it, and RDF's in the end it's all just triples world views.

Atom's Future as a General-Purpose Format

subbu allamaraju recently raised the question of Atom as a General-Purpose Format, and showed one example (Google Code Search Data API) that looked like a not-so-great way of using Atom. in my comment on his blog, i agreed to his observation that this did not look all that great as far as Atom goes, but i also argued that it might still make sense to use Atom for such a scenario.

comparing Atom with HTML (not as formats, but as methods for information dissemination) can teach us a lot, in particular when looking at possible future developments for Atom. the Atom Landscape already shows some similarities to HTML, thanks to Atom's extension semantics. there are microformats (licenses and paging, for example), there is a lot of custom code only picked up by custom clients (such as subbu's Code Search Data API example), and some of the Atom on the Web is so badly crippled that it is utterly useless for plain Atom clients. that's exactly like the good old best viewed in Internet Explorer web pages.

however, while this might not be what we want to see, it may very well be what we need to see to make Atom as exciting as i think it will be. the web is what it is exactly because HTML is so bad. HTML is bad on many different levels and not good at anything in particular. but it's sufficient as a container for an astonishing number of things, and it is simple enough to be learned and used by many. people still use PDF when they care about display fidelity and Flash when they care about multimedia features, but all of these formats are nothing compared to HTML.

REST still is on its way up and i am concerned about the fact that it has been officially downgraded into a buzzword and will suffer the hype cycle backlash at some point in time. but regardless of that, REST is what makes the web tick, and for many people, REST is still hard to grasp (our WWW2009 From SOA to REST tutorial last week was great, but also showed us that there is quite some way to go to spread the message). having best practices and design patterns and frameworks and tools will help a lot, and i expect all of these to appear around Atom.

and we will see all the things that are inevitable when something gets picked up by many. we'll see under-specified extensions (such as GeoRSS), overlapping formats (such as Apple's Podcasts and Yahoo's Media RSS), toolkits (for example supporting feed paging and archiving), and all the other things (some of them invasive and/or malicious) we have seen in HTML. and as inconvenient as that may be from time to time, it also and most importantly is the sure sign of a healthy and active ecosystem.

so what does all of that mean for Atom-oriented developers? similar things as for HTML-oriented web designers, plus some extra care that's required because your goal is to be understood by machines, not by humans. always validate what you're producing, test with other than your preferred clients (such as a plain Atom feed reader), try to reuse established standards in the Atom Landscape or promising candidates for new formats, and most importantly: build your system in a way so that you can change and adapt. if new standards or formats emerge, you should be able to support them, because you have implemented a clear and clean separation between your model (whatever it may be) and the representation of it (Atom plus some extensions).

Feeds as Query Result Serializations

Many Web-based data sources and services are available as feeds, a model that provides consumers with a loosely coupled way of interacting with providers. The current feed model is limited in its capabilities, however. Though it is simple to implement and scales well, it cannot be transferred to a wider range of application scenarios. This paper conceptualizes feeds as a way to serialize query results, describes the current hardcoded query semantics of such a perspective, and surveys the ways in which extensions of this hardcoded model have been proposed or implemented. Our generalized view of feeds as query result serializations has implications for the applicability of feeds as a generic Web service for any collection that is providing access to individual information items. As one interesting and compelling class of applications, we describe a simple way in which a query-based approach to feeds can be used to support location-based services.

Feeds as Query Result Serializations is a technical report published today. it describes what we think will be the next step in the evolution of lightweight web services. in the same way as containerization has revolutionized the global trade of physical goods, feeds are already essential as a lightweight and universal way of sharing machine-readable information, and this trend will continue.

one of the major missing pieces in the atom landscape today is the lack of well-defined query features, and we argue that such a capability would make atom feeds much more flexible in a wide variety of scenarios. this can be as simple as querying feeds by category (something already possible with GData), or as sophisticated as spatial queries when considering geo-referenced collections. more specifically, the step from the Internet of Things to the Web of Things, i.e. making the step from enabling connectivity to a web architecture for networked things, probably will be made possible by smart ways of extending and combining feed technologies. after all, many of the data streams produced by sensors or other pervasive technologies can be readily mapped to feeds, and exposing stream processing capabilities as queryable feeds would bridge the gap between web-based information streams, and information streams originating in the physical world.

RESTful XML is Graphs

the debate about XML vs. RDF (or about a web-based semantic web vs. a semantic web built on a new suite of technologies) will not calm down anytime soon. there is a lot at stake on both sides, arguments are being thrown back and forth, and since it is such an abstract debate to have, nobody can be just right or wrong anyway. it's all a question requirements and constraints.

one argument that is voiced over and over again is that XML is just trees and that RDF is graph-based and that trees may be fine for documents, but that for anything a bit more sophisticated, trees just don't cut it and you need to use graphs. and i am sure everybody who has ever worked with XML has experienced tree trauma when having to deal with the various flavors of XML trees, and tree tremors when having to deal with non-tree data (such as relational data). so there is little doubt that trees may be great for individual documents, but very often are not all that great beyond that.

what is often lost in this argument is that the argument is actually not XML vs. RDF. it is an argument of plain web technologies vs. semantic web technologies for linked open data. which means that the plain web side actually is XML+REST. and this is what often gets lost: REST's constraint of hypermedia as the engine of application state essentially turns RESTful XML into graphs; RESTful XML is traversed in graph-like manner, and each resource encountered can be anything (but often will be XML). so RESTful XML actually combines the best of both worlds: trees for documents (try to represent mixed content in RDF...), and graphs for the larger overlay of connected resources.

so apart from all the additional requirements and constraints that have to be taken into account when discussing the larger question, the claim that XML is just trees is simply not true when you look at it in the appropriate context. if it were, you could equally claim that RDF is just edges because each triple only represents one edge in the RDF graph.

Stimulus Feed Guidelines

published february 18, the Initial Implementing Guidance for the American Recovery and Reinvestment Act guidelines created an impressive echo on the web, because they were an exciting sign that government openness and transparency make one significant further step, setting up a feed-based infrastructure for information dissemination. these guidelines are only an initial version (the document itself claims that more detailed guidelines will be published in 30-60 days), and our first reaction as well as some more detailed feedback was still not sufficient to explain what we thought was required to make this really work in a reliable and robust way.

together with my colleagues eric kansa and raymond yee, we have now finished a technical report about next steps for the recovery act guidelines, making specific recommendations for technical guidelines and best practices around them. this is the abstract:

The Initial Implementing Guidance for the American Recovery and Reinvestment Act of 2009 provides guidance for a feed-based information dissemination architecture. In this report, we suggest some improvements and refinements of the initial guidelines, in the hope of paving the path for a more transparent and useful feed-based architecture. This report is meant as a preliminary guide to how the current guidelines could be made more specific and provide better guidance for providers and consumers of Recovery Act spending information. It is by no means intended as a complete or final set of recommendations.

the most exciting part about writing this report was not so much writing the report itself, but creating a sample dataset for demonstration purposes. the sample dataset homepage has links to the underlying XML, schemas for the three feed types (for communications, formula block grant allocations, and weekly reports), fake agency web pages, fake agency feeds, and even visualizations of our fake data in Google Maps, Timemap, even featuring a KML version to load into Google Earth. all of this is currently generated from the initial XML with a simple XSLT transformation (the goal would of course be for all of this data to be made available in a distributed fashion).

for something as significant as the recovery act and as heavy as $787 billion, it is of course impossible to create complete and final guidelines in just two weeks. thus, our mail goal was to demonstrate that the guidelines don't need to require anything complicated to make exciting things possible; all that is required are guidelines specific enough so that data can be found reliably and in a machine-readable way. recovery.gov and its lists of agency sites and agency reports are a promising starting point, and with our report we hope to make some contributions to how to improve the transparency and openness of stimulus feeds.

Finding Stimulus Agencies

starting from Finding Stimulus Feeds and figuring out that this requires Finding Stimulus Sites, we are now back to square one, so to speak, trying to just figure out how to start the search for sites, so that we can find the feeds. recovery.gov has a list of available recovery sites, but it is not machine-readable and suspiciously short.

we are currently trying to compile our own list of eligible agencies, which should, according to the guidelines, set up recovery sites at agency.gov/recovery. it would be great to get some expert opinions on which agencies are actually eligible and thus would have to set up web site and feeds, if they decide to spend stimulus money.

ironically, there also is no machine-readable list of government agencies we could find. usa.gov has an A-Z Index of U.S. Government Departments and Agencies, but there only is a choice of paged HTML or PDF. we assumed that this would be a good list to start with, so we scraped that list and turned it into XML (and here is an HTML version of it). this list contains 587 agencies and their home pages as listed on usa.gov; most of them are .gov sites, but there are also a number of others, such as .mil, .us, and .net.

the next step will be to ping all of them for recovery pages (at agency.gov/recovery) and then see whether they return something. but before we start doing that, it would be great to get some expert feedback on whether these 587 agencies are what we really should focus on, or whether the recovery act guidelines apply to a different set of agencies that includes some we haven't in our dataset.