The robot.txt revisions at the whitehouse.gov

Yesterday, Atrios posted a story about how the folks at The White House Website have changed their robots.txt file to prevent search engines from cataloging pages about Iraq. While no one is quite sure, yet, why they're doing this, one theory that has been floated is that it is to prevent those pages from being cached by Google, thus making it harder to prove that anything has been changed - such as earlier this year when they edited all the pages that has referred to Bush declaring that "combat operations" in Iraq had ended to show, instead, that Bush had said that "major combat operations" had ended.

For the record, in his original speech, Bush did say that it was "major combat operations" that had ended, but in the articles posted to the White House website and in the press releases sent out about the speech, the phrase was shortened to just "combat operations". Then, in August, a reporter asked President Bush about his having said that "combat operations" had ended, and Bush corrected him, pointing out he'd said "major combat operations". Shortly after that incident, the headlines were changed on the whitehouse.gov site. I had posted about the change at the time it occurred, and had included a few screen shots of pages that hadn't been updated yet showing the original text.

Part of what bothered me about the change was that these were copies of "press backgrounders" and such that the White House had sent out under the original headline using the shortened "combat operations", which is how the press then began referring to the situation. By going back and changing the website without noting that the original press releases had been sent out with a different phrasing in the headline, the White House is trying to erase the evidence that the use of "combat operations" instead of "major combat operations" was essentially endorsed by the White House Press Office itself, making it easier to imply that it was the press who got it wrong in the first place and that President Bush is, once again, simply being misrepresented by the media.

At any rate, this is one theory about why the White House might have decided to keep search engines out of most of the public files on Iraq, though there probably are other ways to achieve the same goal of preventing caching without blocking the files completely from search engines. Whatever the reasons, the change appears unlikely to be something that was done by accident.

Keith at bway.net appears to have made the initial discovery, and has posted two pages explaining exactly what changes were made. For those who have a fairly good technical understanding of things like a robot.txt file, he offers a technical explaination, and for those who are less familiar with Unix arcana, he has a non-technical explaination.

Whatever the reason for the change, I find it bothersome. The White House website should be easily accessible for all people. The information it contains is, obviously, publicly accessible and, one would presume, put there so that people can find out what our government has to say for itself. To systematically try and prevent external search engines from being able to find information on certain topics reduces its functionality considerably. I know for myself, I rarely go directly to the White House website to look for information unless I already know that what I'm looking for is likely to be there. More often than not, when I've found something at the White House website, its been because a Google search has sent me there to find what I'm looking for.

This tactic, though, fits well with the secretiveness and evasion that we've seen throughout the government during Bush's term. This administration doesn't seem to understand that, in America, the ability to govern is by the consent of the people, and that in order for us to give our consent, we not only need to know what the government is up to, but we have a right to know - except in cases where making certain information public would pose a threat to national security. I seriously doubt, however, that allowing publicly available articles about our activities in Iraq to be accessed through major search engines qualifies as a threat to national security.