dretblog

Web Services for Recovery.gov

together with my colleagues eric kansa and raymond yee, we have just published a new paper about Web Services for Recovery.gov as UCB ISchool Report 2009-035. the main goal of this report is to highlight a side of the open and transparent data debate that is often overlooked. some people look a lot at web site design in terms of HTML/CSS design and usability/accessibility, and other people look a lot at how to make data available, focusing on what essentially boils down to building download portals. our main contribution is to highlight the fact that download-oriented data access is not sufficient to provide timely access to evolving data sources, for two main reasons:

• data is never complete. data is always connected to more data, and download formats often disrupt this connection, making the data decontextualized and making it harder to understand how it is embedded into the world. once data is decontextualized, it can be very hard or almost impossible to contextualize it again, re-establishing all the connections to related data that existed where the data originated.
• data is never static (unless it's purely historical data). downloads not only are decontextualized in terms of connectivity with other data, they also are decontextualized with regard to time. any data source that is frequently updated must have simple ways for synchronizing data, so that more efficient ways of data access than periodical downloads can be established.

starting from our earlier report on how ARRA reporting should use lightweight and web-based technologies, we are again focusing on technologies which are powerful enough to implement dynamic and machine-friendly services, but on the other hand also simple enough so that citizen developers can easily understand and use them. refining our earlier report, we are now focusing on widely established and supported technologies such as XML and feeds. we make concrete recommendations how to build web services for recovery.gov that are both more sophisticated than download approaches, while still being easy to implement both on the provider as well as on the consumer side. here's our report's abstract:

One of the main goals of the Recovery.gov Web site is to provide information about how funds for the American Recovery and Reinvestment Act (ARRA) of 2009 are allocated and spent. In this report, we propose a reporting architecture that would focus on the reporting services rather than the Web site and page design, and that uses these Web services to build the user-facing part of ARRA reporting. Our proposed architecture is based on simple and well-established Web technologies, and the main goal of this architecture is to provide citizens and watchdog groups simple and easy access to machine-readable data. Our architecture uses a more sophisticated framework than simple downloads of data files. Our proposed architecture is based on the principles of Representational State Transfer (REST) and uses established and widely supported Web technologies such as feeds and XML. We argue that such an architecture is easy to design and implement, easy to understand for users, and easy to work with for those who want to access ARRA reporting data in a machine-readable way.

you can find the Web Services for Recovery.gov report in the iSchool's eScholarship repository. any feedback about the report is very welcome, and please make sure to also visit our new Architecting Transparency web site, which has been specifically built to highlight and demonstrate some of the issues described in the report.

New Stimulus Report Scrape

we just updated our scraped data for stimulus reports, this comparison page shows the reports we scraped from recovery.gov as well as the reports we got from agency sites directly. we now have 416 reports we found on recovery.gov, and 92 reports we found at agency sites. but please keep in mind that finding agency feeds is not all that easy, so we might have missed some agency feeds that recently became available. it is also important to keep in mind that even though the february as well as the april OMB reporting guidelines specify that reporting has to be done via feeds, nobody seems to enforce this, and OMB itself is only accepting report from agencies via email, which kind of defeats the purpose of having feeds for the agencies.

it also is interesting to point out that there is a new template out there, version 1.4. we usually find out about this because our code crashes; some new template is encountered, and then we have to reverse-engineer the new structure and add this to the scraping/parsing/transformation process. the interesting thing about the new template is that it is the first time that there's some location data in the reporting, in this case in the form of a state code (here are examples in HTML and in XML, as well as the original Excel). this is probably not the granularity where things get exciting, but it is definitely a step in the right direction: get your map mashups ready!

with new guidance expected next week (and not this week as i originally thought), i am sure all of this will change again. and playing whack-an-excel with the changing data gets less exciting, now that we have a toolset in place. but this also shows how hard it still is to follow the data flow about stimulus spending, if you really want to have all the data (and ideally from the source, the agency feeds). publishing HTML/Excel on the web is only marginally open and not transparent at all, and even the recently added feeds do not help all that much, because they are neither paged nor archived. (well, our feeds are not paged as well, but at least they're complete, and we don't run on a $84 million budget.)

so we are definitely looking forward to next week's updated stimulus reporting guidance, and to a certain extent, we can probably judge the quality of the new guidance by how much easier it will make it for us to do what we have been doing so far. after all, all we are trying to do is to keep track of all of the reporting that is published somewhere, and so far, this job has required much more reverse-engineering and continued updating of code than it should.

New ARRA Reporting Guidelines

the OMB will release an updated set of stimulus reporting guidelines this week, at least this has been suggested by some unofficial sources. it will be very interesting to carefully look at the guidelines. the initial guidelines, published in february, got us very excited, because they looked as if somebody actually had a pretty good grasp of openness, transparency, and web architecture: they specified information flows to be based on feeds, and looked really promising. it was the first time feeds were recognized as a relevant way of moving government information around.

we had some quibbles about the way how the feed architecture was underspecified, had a closer look and started suggesting what exactly needed to be fixed, and eventually published a report how it would have to be fixed. that report did not specify everything down to the last detail, but would have been a good starting point to get down to business with the feed-based architecture. in the meantime, only few agencies really published feeds, even fewer of those feeds were findable and usable, and the real reporting was done by email, invisible to anybody other than the OMB back office.

to a large extend, this cannot be blamed on the agencies, but has to be blamed on the guidance, which left open many crucial questions, and also provided no help for agencies to publish feeds, or to test and validate their published feeds. this led to odd developments such as NASA's monofeeds, where each report is published in a new feed.

when the guidelines were updated in april, we were disappointed that the section that would have been essential to make openness and transparency work, the section about feeds, remained basically unchanged. this would have been the section that would have required a serious overhaul, and instead of getting it to the point where it could serve as implementation guidance for agencies, reporting still is being done by email in the most intransparent way imaginable. it will be interesting to see whether the next update of the guidelines will eventually get rid of email reporting and make feed-based reporting mandatory, based on well-defined and well-supported guidelines.

given that the budget for recovery.gov is $84 million, spending a small fraction of this on specifying the actual reporting architecture would be critical. if it is feed-based as mandated in the two initial guidances, then a lot of the back-end black magic can be turned into open and transparent feed-based data management. however, it seems as if the web site overhaul and the reporting architecture will be mixed up, and that is not a good idea to begin with. transparency means that the reporting data is available from its sources, the agencies and the states. recovery.gov then can use this data in the same way as anybody else who wishes to work with the reporting data.

the most transparent way to go would be to clarify the feed guidelines, provide a Federal Feed Cloud as supporting infrastructure to the agencies who do not wish to host their own feeds, and have recovery.gov working on the data in that cloud. instead, what might happen is that the recovery.gov overhaul is outsourced as one project, and whoever gets it will implement a traditional silo with intransparent back-end processes, and a nice and polished front-end to make up for it.

Recovery.gov Overhaul

Recovery.gov gears up for a makeover, and while the official announcement has not yet been released, this initial announcement looks a bit troubling. In our stimulus feed guidelines, we have made the point that the recovery architecture (here our first reaction, more detailed feedback, and our comments about the updated guidance) has to look at the bigger picture, and not just at the web site.

The main issue with the overhaul is that it seems to focus just on the site, probably in an effort to provide open access to stimulus data. However, if transparency is also a goal of this initiative, then the whole recovery reporting architecture must be designed and implemented in sync. In particular, there must be open and accessible data flows from the reporting entities into the recovery site. The feed approach described in the initial and updated guidance (which as of today is still not enforced, not supported by any tools or tests, and not used by the recovery site itself) would be the best way to ensure open access and transparency. The announcement reads as if recovery.gov will become another silo that is fed by opaque processes, so that it essentially is the very opposite of transparent.

For a truly transparent and forward-thinking architecture, a federal feed cloud would be the best approach. It would provide a hosted infrastructure for agencies to manage and publish their reporting (if they are not hosting their own feeds), provide tools and tests to make sure data is being published according to the guidelines, and would be a completely transparent way of how everybody could access stimulus reporting data. Recovery.gov would just be one site using this data, and others could build their own sites, if they wanted to.

In theory it is possible that the overhaul could be used to create such a transparent reporting architecture. In practice, however, the challenging deadline and the absence of specifically enforcing a feed-centered architecture make it likely that the new site will have the same intransparent data flows as the current site (which is still being fed by reports being emailed to the OMB, not by the few feeds that are actually available).

Our hope is that the overhaul process will clearly and cleanly separate the reporting architecture, which should be feed-based and fully transparent, and the publication of that data on recovery.gov. The new Recovery.gov should not be based on an intransparent submission process, but instead should finally implement the forward-looking feed architecture mandated in the initial and the updated guidance documents.

It is still not too late to design and implement such an architecture, and it could set a very important precedent for how formerly internal and opaque data flows inside of the government become accessible and transparent. Devaney's office [ the Recovery Act Transparency and Accountability Board ] received $84 million in stimulus funds, the majority of which will be used for the site maintenance and redesign. It would be an excellent use of stimulus money to design and implement an architecture that not only realized the immediate goal of providing transparent stimulus reporting, but also served as a demonstration of how openness and transparency can be implemented based on simple and accessible standards, leading the way for other e-government initiatives. Spending the money on implementing a silo with a polished interface would not only miss and important opportunity, it would also ignore the requirements set forth in the forward-looking guidance documents. It will be very interesting to read the full set of requirements for the recovery.gov overhaul when they are published.

New Recovery Act Guidance

on april 3, the OMB has published Updated Implementation Guidance for the American Recovery and Reinvestment Act of 2009, replacing the 62 pages of the initial guidance with 175 pages of more detailed guidance.

from the information architecture point of view, this is a major step backwards. when the initial guidance was published, the official OMB position was that the email-based reporting would only be an intermediary step, and that eventually, feeds would be the only way how information would flow. this not only would have been an amazing step towards building a loosely coupled federal information architecture, it also would have assured that the same information made available to the OMB could have also be consumed by anybody else interested in this data.

we were very excited to see the government making such a bold step, and not only prepared a pretty detailed report suggesting more detailed guidelines for the final feed architecture, but also started following the available feeds as well as the reports made available on recovery.gov. our tracking of the reports available through feeds vs. the reports available through recovery.gov did not look to promising, though, and the new guidelines make clear why this is the case.

appendix 1 of the new guidelines contains the same underspecified feed guidelines that were part of the initial guidance, and the document as a whole makes it very clear that feeds are optional, and even if they are provided, they can be RSS or Atom, there is no specification of how the data should be included in the feed, there is no requirement for paged or archived feeds, and there is no requirement for feed discovery. given all of this, feeds may not be prohibited by the new guidelines, but they certainly do not help at all to make feeds a reliable or robust source of information.

instead, now it's back to the good old silo approach, which ironically was also championed by the NGOs participating in the process. it seems that the superficial convenience of a centralized silo is preferred over true transparency, which by its very definition would mean to be able to look inside and beyond a silo. the NGOs clearly missed this unique opportunity to get true transparency from the government, by focusing on the convenience of a single provider from which they can get all their information, instead of praising the federated approach and helping with making it work. i am still very confused by how the term transparency is often used in this context: a centralized architecture is the very opposite of transparency; it may be open (and let's hope it will be open), but it is completely opaque.

the new guidance definitely is a big step backwards, and it is understandable why this happened. implementing such a federated feed-based architecture would have been very challenging, and would have required fairly sophisticated information architecture. so from the OMB point of view, it is quite clear why the centralized approach is now being favored. From the NGO's point of view, however, it really was not the smartest move to ask for a convenient short-term one-stop-shop, instead of recognizing that this could be a landmark shift in how information architecture is designed and exposed in the federal government.

with many things renamed and redesigned, we will now have to revise our Stimulus Feed Tracking site. but we can now probably stop tracking the few feeds actually available through agency sites, because they will remain a small fraction of all reports available. the new guidelines say nothing about the availability of information on recovery.gov, so we assume that we will continue to see only weekly reports, and no major communications and no funding reports. it will be interesting to see when this data is made available, because it is already being reported to the OMB. but since reporting is now implemented through publicly invisible channels (email and maybe a web-based submission interface in the future), it is impossible to get to that information (and our stimulus feed tracker does not show a lot of activity in the publicly available major communications and formula block grant allocation feeds).

finally, when recovery.gov and data.gov go online, it will be interesting to see how the report data is made available. will it be the original speadsheets? XML? RDF? XHTML+RDFa? will we get feeds for all data? it still strikes me as weird that recovery.gov is a completely feed-free zone, even though drupal (which is what recovery.gov is using) has feed modules and all that would be required would be to simply throw a switch and make some feeds available; since we apparently cannot get reports via feeds, can we at least get the news as a feed?

[[ update april 8, 2009: i have received some comments saying that page 68 of the guidance states that agencies are required to provide a feed. but i would argue that the guidance not even makes discoverability of the feeds mandatory, let alone specifies a way in which these feeds, if they were discoverable, could serve as a reliable source of information. our Proposed Guideline Clarifications for American Recovery and Reinvestment Act of 2009, published mid-march, were explicitly licensed to be reusable and could have served as a simple copy&paste source for making stimulus feeds much more usable. the fact that the feed section was not updated at all to me seems to make clear that this part of the guidance is not central to OMB's plans and interests. ]]

Recovery Reporting Report

now that we have started Collecting Stimulus Feeds, we quickly realized that there are not so many available (a total of 25 feeds currently), which of course also greatly diminishes the ability to collect useful amounts of data via these feeds. recovery.gov has a much bigger selection of weekly reports on their weekly reports page, because they depend on the reports being sent by email to them.

we are primarily interested in getting to the point where the feeds are a dependable and robust way of getting Recovery Act spending information, but this is currently not the case. so we started scraping recovery.gov's weekly report page in order to get all currently available report.

to better understand how reporting via feeds compares to reporting done by email (and then being published on recovery.gov), we have dropped both datasets (the feed crawl and the reports scraped from recovery.gov) into SIMILE Timeline, a user-friendly package for visualizing time-based data series. the result not only looks pretty nice, it also shows that feeds may be a bit faster, NSF's newest report is available via the feed, but not yet available via recovery.gov. it also shows that the data is sometimes changed, a misdated ED report (2008-02-23) has been republished with a fixed date (2009-02-23) by recovery.gov.

we now have all weekly reports in Excel and XML (which is extracted from the template-based Excel-files), but we have not yet made the XML publicly available. we will do this in the next days, and then republish all the weekly reports we have as feeds in a way similar to the method we used for the demo site we produced for our proposed guidelines: as HTML embedded in the feed, linking to an XML version of the data.

Collecting Stimulus Feeds

Finding Stimulus Feeds turns out to be quite a challenge. starting from our current set of 29 Agency Recovery Sites (i would not expect to find any feeds on the 50 State Recovery Sites), i started looking for stimulus feeds. not entirely unexpected, it is quite a challenge to simply locate those feeds that exist (most sites don't had feeds i could find). some sites support feed autodiscovery, but most don't, and sometime the links to feeds are hidden deep inside of some text or even on sub-pages of the recovery page.

just browsing these pages that have been set up according to the Initial Implementing Guidance for the American Recovery and Reinvestment Act of 2009 guidelines makes it clear that more guidance would have been better. most sites don't have feeds and simply post links to PDFs and/or Excel files, which makes it hard to get to the data. my favorite example for a site that tried to use feeds but really did not succeed too well is NASA's recovery site, which on its weekly reports page has a different feed for each published report. i decided to call this pattern the monofeed (unifeed was another idea), and finding this was simply proof of the fact that more guidance (and maybe simply a well-designed sample site) is required.

after looking at all sites, we ended up with a total of 25 feeds, but because every site would be required to carry three feeds (major communications, weekly reports, and formula block grant allocations), this result is not all that great. our list of Agency Recovery Sites now contains links to all these feeds. we have also set up a page for new items from all feeds, grouped by feed type, and on this page you can see that so far a grand total of three formula block grant allocations have been published via stimulus feeds from all agencies combined.

so the dataflow through the feeds currently is more a trickle than it is a large amount of information that we could use for starting to work with that data, but once we get to the weekly reports, there should be a bit more to work with. however, the information we will be able to get through the feeds will be substantially less than what we could get by scraping recovery sites, and that probably also is substantially less than what has been emailed to the OMB so far (our 29 Agency Recovery Sites still have to be matched up against the 587 agencies we scraped from usa.gov).

Tracking Stimulus Sites and Feeds

writing about Tracking Stimulus Sites and the the problems of Finding Stimulus Sites and Finding Stimulus Feeds is one thing; doing it is another. as part of our initiative for better Stimulus Feed Guidelines and our general goal to make the information flow for the Initial Implementing Guidance for the American Recovery and Reinvestment Act of 2009 more open and transparent, we have now set up a Web site for tracking stimulus sites and feeds.

http://isd.ischool.berkeley.edu/stimulus/feeds/ is our new page for tracking stimulus sites and feeds. We have yet to fully set up the feed listing (and we have not even started crawling all feeds systematically, but we're close!), but at least we have a list of agency recovery sites and a list of state recovery sites; both are currently identical in content to the agency list and the state list on recovery.gov.

the list of agency recovery sites also lists an initial set of stimulus feeds, grouped by the three current feed types Major Communications, Weekly Reports, and Formula Block Grant Allocations. our main goal is to collect more of these feeds, to start crawling them, and generally to explore more of the growing ecosystem around the Initial Implementing Guidance for the American Recovery and Reinvestment Act of 2009 (and it will be very interesting to see what the detailed OMB guidelines say once they are published).

most importantly, we would like to get some feedback what others would like to see in such a registry of stimulus sites and feeds, and what services (such as export data formats, we already have feeds for new agency sites and new state sites) would be useful. our next project is to publish a feed of stimulus feeds, so that it becomes easier to track what new feeds are made available, and to publish all feeds as OPML, so that they can be easily used in any feed reader. suggestions are very welcome at this point, let us know what you think.

also, additions and corrections for our agency sites, state sites, and stimulus feeds would be highly welcome! please send email with any suggestions, we have not yet set up a web-based form for feedback.

Federal Feed Cloud

one of the great things about web architecture is that it talks about loose coupling and how to set up an environment that focuses on cooperation, rather than integration. this is an essential difference, because integration always implies to give up specifics for the benefit of bringing things together, whereas cooperation focuses on keeping things separate, and to only harmonize things that need to be harmonized (RESTfully speaking, the representations of the resources that are required as foundation for cooperation).

our work on stimulus feeds and in particular our recently published report on how to improve the initial stimulus feeds guidance was an interesting opportunity to think about the specifics of implementing such a feed-oriented architecture in a scenario as big and heterogeneous as the U.S. agencies.

it is unlikely that all agencies are willing or able to set up feeds that not only carry reports in a useful machine-readable representation (most likely XML), but also adhere to additional requirements such as supporting feed paging and archiving, using HTTPS to provide some authenticity, and maybe should even have (if this really will be the official channel to do recovery reporting) digital signatures in the authoritative XML version of the data, so that authenticity and origin can be securely verified.

if the OMB is serious with the underlying idea of the Initial Implementing Guidance for the American Recovery and Reinvestment Act of 2009, then the thing to look at would be to provide support for those agencies who are not willing or able to set up feed hosting on their sites. this does not mean, though, that everything has to be centralized in the classical tightly coupled way. it simply means that OMB provides feed hosting for agencies who want or need support to conform to the (yet to be published) detailed feed publishing requirements for stimulus feeds.

for recovery.gov (and maybe even beyond that; but recovery.gov would be an ideal test case and platform for deploying and demonstrating innovation in open government approaches), this could mean to provide a Federal Feed Cloud. here are the services i envision to be provided by this feed cloud:

• secure and reliable hosting: following the cloud meme, such a feed cloud should be secure and reliable, so that data and services provided by that cloud can be used safely and reliably. whether this cloud should be based on commercial cloud services or implemented on IT hardware of the federal government is a tricky and far-reaching question.
• forms for human data entry: using simple web forms, all data required to be published in feeds can be entered in forms and then can be published in a feed by simply submitting the form.
• AtomPub for machine-oriented data entry: for publishing entries in feed through a web service, AtomPub should be used. it is RESTful and custom-designed to provide write access to collections which are published through feeds. Atom-Pub submission should only accept XML and validate all submissions before accepting them.
• transformations for human-readable feeds: even though the primary data format is XML, every feed has transformations (probably using XSLT) attached to it, so that data published in that feed is transformed to HTML, and thus the feed carries human-readable content.
• cloud queries: all the information hosted by the cloud can be consumed using feeds, but it should also be possible to query the cloud, either by querying individual feeds, or by querying sets of feeds. in this scenario, the cloud plays the role of a data warehouse, and feed queries are the language to access it.

i am sure such a Federal Feed Cloud will not become reality tomorrow, but i think it would be the right architectural direction. if agencies don't need it, they don't have to use it at all. but agencies with limited resources for IT development could use such a cloud to easily and rapidly publish their data in a loosely coupled way.

Stimulus Feed Guidelines

published february 18, the Initial Implementing Guidance for the American Recovery and Reinvestment Act guidelines created an impressive echo on the web, because they were an exciting sign that government openness and transparency make one significant further step, setting up a feed-based infrastructure for information dissemination. these guidelines are only an initial version (the document itself claims that more detailed guidelines will be published in 30-60 days), and our first reaction as well as some more detailed feedback was still not sufficient to explain what we thought was required to make this really work in a reliable and robust way.

together with my colleagues eric kansa and raymond yee, we have now finished a technical report about next steps for the recovery act guidelines, making specific recommendations for technical guidelines and best practices around them. this is the abstract:

The Initial Implementing Guidance for the American Recovery and Reinvestment Act of 2009 provides guidance for a feed-based information dissemination architecture. In this report, we suggest some improvements and refinements of the initial guidelines, in the hope of paving the path for a more transparent and useful feed-based architecture. This report is meant as a preliminary guide to how the current guidelines could be made more specific and provide better guidance for providers and consumers of Recovery Act spending information. It is by no means intended as a complete or final set of recommendations.

the most exciting part about writing this report was not so much writing the report itself, but creating a sample dataset for demonstration purposes. the sample dataset homepage has links to the underlying XML, schemas for the three feed types (for communications, formula block grant allocations, and weekly reports), fake agency web pages, fake agency feeds, and even visualizations of our fake data in Google Maps, Timemap, even featuring a KML version to load into Google Earth. all of this is currently generated from the initial XML with a simple XSLT transformation (the goal would of course be for all of this data to be made available in a distributed fashion).

for something as significant as the recovery act and as heavy as $787 billion, it is of course impossible to create complete and final guidelines in just two weeks. thus, our mail goal was to demonstrate that the guidelines don't need to require anything complicated to make exciting things possible; all that is required are guidelines specific enough so that data can be found reliably and in a machine-readable way. recovery.gov and its lists of agency sites and agency reports are a promising starting point, and with our report we hope to make some contributions to how to improve the transparency and openness of stimulus feeds.