dretblog

Recovery Reporting Report

now that we have started Collecting Stimulus Feeds, we quickly realized that there are not so many available (a total of 25 feeds currently), which of course also greatly diminishes the ability to collect useful amounts of data via these feeds. recovery.gov has a much bigger selection of weekly reports on their weekly reports page, because they depend on the reports being sent by email to them.

we are primarily interested in getting to the point where the feeds are a dependable and robust way of getting Recovery Act spending information, but this is currently not the case. so we started scraping recovery.gov's weekly report page in order to get all currently available report.

to better understand how reporting via feeds compares to reporting done by email (and then being published on recovery.gov), we have dropped both datasets (the feed crawl and the reports scraped from recovery.gov) into SIMILE Timeline, a user-friendly package for visualizing time-based data series. the result not only looks pretty nice, it also shows that feeds may be a bit faster, NSF's newest report is available via the feed, but not yet available via recovery.gov. it also shows that the data is sometimes changed, a misdated ED report (2008-02-23) has been republished with a fixed date (2009-02-23) by recovery.gov.

we now have all weekly reports in Excel and XML (which is extracted from the template-based Excel-files), but we have not yet made the XML publicly available. we will do this in the next days, and then republish all the weekly reports we have as feeds in a way similar to the method we used for the demo site we produced for our proposed guidelines: as HTML embedded in the feed, linking to an XML version of the data.

Collecting Stimulus Feeds

Finding Stimulus Feeds turns out to be quite a challenge. starting from our current set of 29 Agency Recovery Sites (i would not expect to find any feeds on the 50 State Recovery Sites), i started looking for stimulus feeds. not entirely unexpected, it is quite a challenge to simply locate those feeds that exist (most sites don't had feeds i could find). some sites support feed autodiscovery, but most don't, and sometime the links to feeds are hidden deep inside of some text or even on sub-pages of the recovery page.

just browsing these pages that have been set up according to the Initial Implementing Guidance for the American Recovery and Reinvestment Act of 2009 guidelines makes it clear that more guidance would have been better. most sites don't have feeds and simply post links to PDFs and/or Excel files, which makes it hard to get to the data. my favorite example for a site that tried to use feeds but really did not succeed too well is NASA's recovery site, which on its weekly reports page has a different feed for each published report. i decided to call this pattern the monofeed (unifeed was another idea), and finding this was simply proof of the fact that more guidance (and maybe simply a well-designed sample site) is required.

after looking at all sites, we ended up with a total of 25 feeds, but because every site would be required to carry three feeds (major communications, weekly reports, and formula block grant allocations), this result is not all that great. our list of Agency Recovery Sites now contains links to all these feeds. we have also set up a page for new items from all feeds, grouped by feed type, and on this page you can see that so far a grand total of three formula block grant allocations have been published via stimulus feeds from all agencies combined.

so the dataflow through the feeds currently is more a trickle than it is a large amount of information that we could use for starting to work with that data, but once we get to the weekly reports, there should be a bit more to work with. however, the information we will be able to get through the feeds will be substantially less than what we could get by scraping recovery sites, and that probably also is substantially less than what has been emailed to the OMB so far (our 29 Agency Recovery Sites still have to be matched up against the 587 agencies we scraped from usa.gov).

Tracking Stimulus Sites and Feeds

writing about Tracking Stimulus Sites and the the problems of Finding Stimulus Sites and Finding Stimulus Feeds is one thing; doing it is another. as part of our initiative for better Stimulus Feed Guidelines and our general goal to make the information flow for the Initial Implementing Guidance for the American Recovery and Reinvestment Act of 2009 more open and transparent, we have now set up a Web site for tracking stimulus sites and feeds.

http://isd.ischool.berkeley.edu/stimulus/feeds/ is our new page for tracking stimulus sites and feeds. We have yet to fully set up the feed listing (and we have not even started crawling all feeds systematically, but we're close!), but at least we have a list of agency recovery sites and a list of state recovery sites; both are currently identical in content to the agency list and the state list on recovery.gov.

the list of agency recovery sites also lists an initial set of stimulus feeds, grouped by the three current feed types Major Communications, Weekly Reports, and Formula Block Grant Allocations. our main goal is to collect more of these feeds, to start crawling them, and generally to explore more of the growing ecosystem around the Initial Implementing Guidance for the American Recovery and Reinvestment Act of 2009 (and it will be very interesting to see what the detailed OMB guidelines say once they are published).

most importantly, we would like to get some feedback what others would like to see in such a registry of stimulus sites and feeds, and what services (such as export data formats, we already have feeds for new agency sites and new state sites) would be useful. our next project is to publish a feed of stimulus feeds, so that it becomes easier to track what new feeds are made available, and to publish all feeds as OPML, so that they can be easily used in any feed reader. suggestions are very welcome at this point, let us know what you think.

also, additions and corrections for our agency sites, state sites, and stimulus feeds would be highly welcome! please send email with any suggestions, we have not yet set up a web-based form for feedback.

Federal Feed Cloud

one of the great things about web architecture is that it talks about loose coupling and how to set up an environment that focuses on cooperation, rather than integration. this is an essential difference, because integration always implies to give up specifics for the benefit of bringing things together, whereas cooperation focuses on keeping things separate, and to only harmonize things that need to be harmonized (RESTfully speaking, the representations of the resources that are required as foundation for cooperation).

our work on stimulus feeds and in particular our recently published report on how to improve the initial stimulus feeds guidance was an interesting opportunity to think about the specifics of implementing such a feed-oriented architecture in a scenario as big and heterogeneous as the U.S. agencies.

it is unlikely that all agencies are willing or able to set up feeds that not only carry reports in a useful machine-readable representation (most likely XML), but also adhere to additional requirements such as supporting feed paging and archiving, using HTTPS to provide some authenticity, and maybe should even have (if this really will be the official channel to do recovery reporting) digital signatures in the authoritative XML version of the data, so that authenticity and origin can be securely verified.

if the OMB is serious with the underlying idea of the Initial Implementing Guidance for the American Recovery and Reinvestment Act of 2009, then the thing to look at would be to provide support for those agencies who are not willing or able to set up feed hosting on their sites. this does not mean, though, that everything has to be centralized in the classical tightly coupled way. it simply means that OMB provides feed hosting for agencies who want or need support to conform to the (yet to be published) detailed feed publishing requirements for stimulus feeds.

for recovery.gov (and maybe even beyond that; but recovery.gov would be an ideal test case and platform for deploying and demonstrating innovation in open government approaches), this could mean to provide a Federal Feed Cloud. here are the services i envision to be provided by this feed cloud:

• secure and reliable hosting: following the cloud meme, such a feed cloud should be secure and reliable, so that data and services provided by that cloud can be used safely and reliably. whether this cloud should be based on commercial cloud services or implemented on IT hardware of the federal government is a tricky and far-reaching question.
• forms for human data entry: using simple web forms, all data required to be published in feeds can be entered in forms and then can be published in a feed by simply submitting the form.
• AtomPub for machine-oriented data entry: for publishing entries in feed through a web service, AtomPub should be used. it is RESTful and custom-designed to provide write access to collections which are published through feeds. Atom-Pub submission should only accept XML and validate all submissions before accepting them.
• transformations for human-readable feeds: even though the primary data format is XML, every feed has transformations (probably using XSLT) attached to it, so that data published in that feed is transformed to HTML, and thus the feed carries human-readable content.
• cloud queries: all the information hosted by the cloud can be consumed using feeds, but it should also be possible to query the cloud, either by querying individual feeds, or by querying sets of feeds. in this scenario, the cloud plays the role of a data warehouse, and feed queries are the language to access it.

i am sure such a Federal Feed Cloud will not become reality tomorrow, but i think it would be the right architectural direction. if agencies don't need it, they don't have to use it at all. but agencies with limited resources for IT development could use such a cloud to easily and rapidly publish their data in a loosely coupled way.

Flashy iPhone

while the next iPhone model may still be a couple of months away (version 3.0 of the OS was recently announced, and most people expect an updated phone version as well when the OS will be released this summer), there is one thing i really hope it will have. an LED. at some point in time, apple realized that the screen-only approach is inconvenient when you actually missed a call or a message, because, well, you missed it, and thus you are not necessarily switching on the phone to check for messages on the display. as an iPhone owner, you might get used to doing this all the time, but if this is what happens, it is a sure sign that the designers forgot something important. so apple added a feature (i don't know with which OS release) which lets the phone repeatedly ring for messages. but there is no such thing for missed calls, and the repeated ringing is a nuisance, particularly at night. maybe this was the best the designers could come up with as a software patch, but there would be a much better way of doing this.

my first mobile phone (back in 1998) was an ericsson SH888 (it was great because it had a built-in modem and IrDA; those were the pre-GPRS days). it was extremely expensive, had a bauhaus-kind of forms follows function beauty (which i actually liked), and was built like a rock. what it also had was a green LED that emitted very short pulses every few seconds as a power indicator. and when there was a notification on the screen (for a missed call or a received message), the LED switched to faster blinking. very simple, but highly efficient. and over time, i also liked the constant power-on blinking of the phone, because it allowed to find the phone in the dark easily, and since it was just a very short pulse, the blinking never was annoying.

my second mobile phone (2001) was an ericsson T39m, the first WAP phone available and absolutely pathetic in this department (not ericsson's fault, just the general stupidity of building a parallel web for mobile phones). it even had POP-based email, which was insanely expensive due to the ridiculous data plans available back then (this one had GPRS). also, reading email on a three-line display was not terribly convenient. one funny thing about the T39m was the messed up firmware (with regard to the little LED which it inherited from the SH888). while the LED still switched to fast blinking as long as the screen backlight was switched on for displaying a notification (missed call or received message), it switched back to slow blinking as soon as the screen backlight was switched off. this minor bug annoyed me to no end, i was so used to simply look at the LED to check for missed calls or messages that i realized how convenient that had been when it was no longer possible. i even brought it to a dealer to get a firmware update just to get the LED fixed, but as it turned out, that part of the firmware was not fixed in the new version i got.

what i want for the next iPhone is such an LED. i am sure apple designers would choose a white LED and teach it all kinds of tricks (such as fading) to make it look better, but the basic idea of having some kind of notification without having to switch on the phone would be extremely useful. plus the added benefit of being able to find the phone at night. and i am sure battery life would not be affected significantly by this, and it also requires almost zero additional space. it could be used to visualize timers and countdowns without having to look at the screen all the time. it would allow more subtle phone ringing than the not-so-subtle vibrate. and if the LED would also get an API, additional uses would be possible by allowing apps to access it. maybe it could even by used by push data, so that external requests could trigger it. i have no high hopes for this to actually happen, but it definitely would be nice to see apple's attention to detail coming to the conclusion that an additional notification mechanism would be useful in a variety of ways.

The Web According To Google

It is interesting to ask the question how the Web would look if it would be reinvented today. And of course this question would be answered very differently by different communities, depending on their focus and interest. Since Google is such a big player on the Web these day, and increasingly shapes the way in which the Web is perceived and develops, it is particularly interesting to think about what The Web According To Google would look like, and what that would mean.

While The Web means very different things to different communities these days (the three biggest communities with rather distinct world-views are the RDF-flavored Semantic Web, Ajaxy Rich Internet Applications, and SOAP-style Web Services), some of the main principles underlying these different perspectives remain the same. The Web at its very core is URIs, HTTP, and HTML; in a slightly extended perspective, CSS and JavaScript (in particular its XmlHttpRequest-based flavor of Ajax) also have become essential for many Web-based scenarios. Looking at these fundamental technologies, what would The Web According To Google look like?

• URIs are brittle and dangerous: URIs have proven to be the source of many confusions and problems. People have a hard time remembering URIs and often forget them, domain names have become a market of their own with many problems lurking in the areas of domain-name-based security concepts, and a Unicode-enabled DNS, and with resources generally becoming inaccessible due to removed resources or server reorganizations. So at the very least, the address bar in a browser has to disappear, so that this source of errors and insecurities is safely eliminated. A search box is sufficient to find any resource on the Web, and URIs have introduced more problems than they have provided value. Instead of URIs, the Web should simply be organized as one huge distributed hash table, for which efficient implementations are known and tested, and which generally avoids the problems and inefficiencies of crawling and deep crawling. Servers publishing pages have to insert them into the table, and when resources are deleted, they are removed from the table.
• Search-based Navigation: As mentioned above, URIs and URI-related security and usability issues have long caused problems on the Web. Instead of trusting a Web page to link to the correct site (it could always link to a phishing site), it is much safer to search instead of navigate, and to relate on the wisdom of the crowds when it comes to identifying and avoiding phishing and other security risks. Navigation should thus be restricted to within-domain navigation, and any navigation outside of a site's domain should be done by search, or should at least be supplemented by search, so that the search results could be used to alert uses of unintended consequences when following links to malicious sites.
• Information Brokers: The ideal Web has people interacting with information brokers constantly, either explicitly by searching or following ad links, or implicitly by tracking them through cookies across the vast number of pages somehow affiliated with those brokers. The more brokers know what users are doing, the better they can serve them information that's valuable for them. So instead of having to go through the security and deployment pains of having to deal with cookies and having to install search bars on millions of computers, the Web should have a built-in mechanism of how people can be tracked. Maybe cookies should be moved into HTTP/HTML itself and browsers should stop being so picky about all this privacy stuff. After all, standardization's most noble job is to codify common practices, and if there is one really widespread common practice on the Web, it's user tracking.
• Finding Content: Instead of trying to make HTML a better hypermedia language (so that people can easier navigate the Web, for example by using more specific links to document fragments), it makes much more sense to focus on making HTML a better document format. Linking and navigation is inherently dangerous and prone to link rot and organizational difficulties of large information repositories. So the biggest improvement of HTML would be to simplify its structure so that it can be more reliably indexed, so that users can enjoy better search services. Eventually, it should be considered to get rid of the misleading HT prefix and merge HTML and PDF into PDML, so that content can be easily published, indexed, and printed. why would anybody want to have structured documents when everything is indexed and can be searched and found?
• Applications should be Web-based: Desktop-based applications are a thing of the past, and their slow turnaround time for patches introduces another large security problem. Web-based applications are more agile, are available on demand, and can be used to transform the Web not only into an information cloud, but also into an application cloud. Browsers need a bit more functionality to make this a reality. One example for this is the ability to use local storage for offline work and data access optimization. Google Gears (built into Chrome) is one first step for this and allows applications to persist data. The second step required is for browsers to persist applications, so that offline mode can also be used when the application has to be loaded. It's not too unlikely that one of the future Chrome versions will provide exactly that feature. After this, all applications can be moved to the Web, all their data can be moved to the Web as well, and this will allow information brokers to provide even better recommendation services.

While this may be a bit of a caricature, it is not that far away from what's happening. Many people seem to have given up on the idea of the Web as something that should be decentralized and open to exploration. The Web never was a sophisticated hypermedia system, but its rudimentary and basic hypermedia features proved to be extremely powerful. It turns out that the simplicity and fault-tolerant design of the Web were the killer features, and not the much more advanced hypermedia features researchers had worked on for decades.

The main point of these considerations is not to say that Google is doing a bad job at contributing to the standards process. The main point is to say that Google's contributions are influenced by its perspective of the Web, and by its business model. After all, Google is a spectacularly profitable for-profit company and very likely tries to maintain that status. Google's interest in the Web is a business interest, and its contributions to the development of the Web are shaped by this fundamental perspective. Google's Web is a centralized Web with Google in the middle and everything else is revolving around that central hub. And even though I try to minimize my Google time, the number of times per day I interact with Google (Search, AdSense, DoubleClick, iPhone Map App, Google Reader, ...) must be very impressive.

As a final note on how these things already start showing up in reality: The W3C's Geolocation group is currently specifying a geolocation API, and even though it has not been heavily publicized, this is a very significant activity because it it the first time the W3C ventures into the area of Location-Based Services (LBS), an area with a huge potential for growth. One of the contentious and very important questions is that of privacy: how much control do users have over their location information when they (or their device on their behalf) interact with Web-based services. One of the reasons why the current API has very weak privacy considerations are the claims that (a) these could not be enforced anyway, and (b) by setting too restrictive privacy policies, users might break service functionality. The interesting question in the context of these seemingly technical terms is: what is service functionality when you're Google? If your service is to serve ads based on behavioral and location profiles, then anything preventing you from gathering behavioral and location information means breaking service functionality. It is this view of the Web that already starts shaping the way in which the Web will work in the next couple of years.

The Rise of The Micropayments

now that apple has revealed its plans for the iPhone 3.0 software, one of the exciting things about it is what apple calls In App Purchase: Allow users to purchase content or services from your application using the Store Kit framework. This new framework handles the financial aspects of the transaction, processes payment with the iTunes Store, and provides your application with information about the purchase.

this may actually turn out the be the return of the micropayment. apple's App Store was the first successful way to get an ecosystem of mobile apps off the ground; and it was quickly copied by almost all major mobile OS vendors. it was what made me make my first purchases in bed, browsing featured or popular apps and buying some relaxing game. or interactive fish. whatever, it definitely introduced a model where purchases are done on the level of an impulse buy, which is great for small-scale business.

with this model being extended to within apps, apps can start charging the same small amounts for content, or access to advanced features. it'll be very interesting to see how this feature develops when 3.0 becomes available this summer, but i would not be surprised to see a huge wave of successful micropayment-based apps, disproving the micropayments will never work mantra repeated for such a long time.

another interesting aspect of this is that apple gets to keep their usual 30% share of every purchase. that's what i would call a substantial margin. if this new feature really takes off, apple will make tons of money with it, and at some point in time providers will ask why apple charges about ten times as much as credit card companies. and the answer to that is: because you can do what you like when you have a monopoly.

iPhone 3.0

while i haven't touched my iPhone To Do List for quite a while, the rumors around the unveiling of the iPhone 3.0 software roadmap tomorrow made me look at the list again. even though i never updated the list for the 2.0 versions of the software, many of the issues are still open. some sites say that cut&paste may eventually be included, which would finally spare us the humiliation of having to write down things so that we can type them in again.

but more importantly, it would be nice if apple released its death grip around what it considers its monopolies on the phone. for example, i would pay handsomely to buy an iPhone app for SMS messaging, simply to get rid of the annoying chat metaphor, and more importantly get a character count so that i'd know whether AT&T will charge me for one or two international messages (and these are still obscenely expensive, so i'd really like to know). actually, i'd like to get an SMS messaging app that would stop using AT&T for sending messages altogether and simply use a web-based service for sending them...

more generally, i'd be paying even more handsomely for an application pulling together SMS, email, voicemail, and twitter, to give me the unified messaging i really want, also extending contacts with twitter ids and using this information as the switchboard for all apps. and while i am at it, why not add fax capabilities as well, there are numerous pretty decent web-based fax services that could be used.

of course, this is not going to happen, because the iPhone is not a computer but a phone, and thus historically is treated as a walled garden, and not as a general-purpose device. this traditional view of how telecoms and phone manufacturers look at what their customers are allowed to do aligns nicely with apple's world view, so there is little hope to see an open iPhone that would allow me to install and use the apps that i want to use.

and i guess i'll continue to use the iPhone because it's well-designed and useful and still miles ahead of the competition that is only now slowly waking up. but i just don't buy the claim that i am better off with the resticted iPhone than i would be if i had the freedom to install a crappy phone or messaging app that would not work well.

Stimulus Feed Guidelines

published february 18, the Initial Implementing Guidance for the American Recovery and Reinvestment Act guidelines created an impressive echo on the web, because they were an exciting sign that government openness and transparency make one significant further step, setting up a feed-based infrastructure for information dissemination. these guidelines are only an initial version (the document itself claims that more detailed guidelines will be published in 30-60 days), and our first reaction as well as some more detailed feedback was still not sufficient to explain what we thought was required to make this really work in a reliable and robust way.

together with my colleagues eric kansa and raymond yee, we have now finished a technical report about next steps for the recovery act guidelines, making specific recommendations for technical guidelines and best practices around them. this is the abstract:

The Initial Implementing Guidance for the American Recovery and Reinvestment Act of 2009 provides guidance for a feed-based information dissemination architecture. In this report, we suggest some improvements and refinements of the initial guidelines, in the hope of paving the path for a more transparent and useful feed-based architecture. This report is meant as a preliminary guide to how the current guidelines could be made more specific and provide better guidance for providers and consumers of Recovery Act spending information. It is by no means intended as a complete or final set of recommendations.

the most exciting part about writing this report was not so much writing the report itself, but creating a sample dataset for demonstration purposes. the sample dataset homepage has links to the underlying XML, schemas for the three feed types (for communications, formula block grant allocations, and weekly reports), fake agency web pages, fake agency feeds, and even visualizations of our fake data in Google Maps, Timemap, even featuring a KML version to load into Google Earth. all of this is currently generated from the initial XML with a simple XSLT transformation (the goal would of course be for all of this data to be made available in a distributed fashion).

for something as significant as the recovery act and as heavy as $787 billion, it is of course impossible to create complete and final guidelines in just two weeks. thus, our mail goal was to demonstrate that the guidelines don't need to require anything complicated to make exciting things possible; all that is required are guidelines specific enough so that data can be found reliably and in a machine-readable way. recovery.gov and its lists of agency sites and agency reports are a promising starting point, and with our report we hope to make some contributions to how to improve the transparency and openness of stimulus feeds.

Tracking Recovery Sites

Finding Stimulus Agencies is one problem, and when you have found them, Finding Stimulus Sites is the next problem. while working on the 587 agency names we extracted from usa.gov, and trying to match them up with the currently sparsely populated list of recovery sites on recovery.gov, we already found the first agency on recovery.gov which is not in the list on usa.gov: the Corporation for National and Community Service (CNCS) has a recovery site, but is not listed on usa.gov.

this is unfortunate, because it means that we cannot use the list of 587 agencies as a reliable starting point for our search for recovery sites, which is what we initially hoped to do.

starting from the XML list of agencies scraped from usa.gov, we created a little XSLT that generates a list of agency sites and likely recovery sites for those agencies, but since not all agency site follow the nice http://agency.gov/ pattern (not all agencies have their own domains, or home pages with no path segments), it is not all that easy to reliably generate likely recovery site names.

reading the Initial Implementing Guidance for the American Recovery and Reinvestment Act guidelines, they contain the following statement (on page 3):

1.4 To which agencies does this Guidance apply?
The provisions of this Guidance apply to all Executive Branch departments and agencies involved in or impacted by the Recovery Act or which otherwise perform services for agencies that receive such appropriations.

this language probably intentionally keeps the possible recipients of stimulus funding very loosely defined, so our approach of trying to find a closed list of potential recipients might have been too limited. this would make even more important, however, to have the list on recovery.gov made available in machine-readable formats, so that it is possible to retrieve that information reliably.