October 28, 2003

Update on the robots.txt story

The website 2600 has done a bit of additional research on the White House's modifications to the robot.txt file at the whitehouse.gov website:

According to White House spokesman Jimmy Orr, the blocking of search engines is not an attempt to ensure future revisions will remain undetected. Rather, he explained, they "have an Iraq section [of the website] with a different template than the main site." Thus, for example, a press release on a meeting between President Bush and "Special Envoy" Bremer is available in the Iraq template (blocked from being indexed by search engines) or the normal White House template (available for indexing by search engines). The attempt, Mr. Orr said, was that when people search, they should not get multiple copies of the same information. Most of the "suspicious" entries in the robots.txt file do, indeed, appear to have only this effect.

According to the robots.txt of October 24, though, the In Focus: Iraq section of the site was blocked from search engines. Some of the information there does not appear to be available anywhere else on the White House site. However, it seems that, in response to inquiries from 2600 and other sources, the White House web team has recently changed their robots.txt so that these files are no longer blocked. (The current Last-Modified date on the robots.txt is 23:22 GMT, October 27th, after work on this article had already begun.)

While it's unlikely that the White House would have admitted it if it was trying to ensure that revisions would remain hidden, on it's face, the explaination offered seems at least somewhat reasonable. I'm not much of a techie, however, so I can't really evaluate it in any significant depth. I recommend reading the full article, though, for any additional information or updates that they might add.

Posted by thorswitch at October 28, 2003 07:19 PM | TrackBack


Comments

Wow, you really got the Bush administration now. Changing headlines for articles to match the words he actually used!

Posted by: jack nunyo at October 31, 2003 12:28 AM

Well, Jack, the point of the whole thing is that the White House initially sent out press releases that DIDN'T use the words he'd actually said - it left the "major" out. Those press releases are very carefully vetted before they're sent out, and their being listed on the website should be a historical record of what the news agencies were sent. By going through and modifying it months after the fact, they are trying to make it appear that it always did have the word "major" in it, and that if anyone reported otherwise, that was their own fault, rather than acknowldging that they were following the White House lead.

Press releases are designed to be "usable" as-is - they reflect what the person or organization WANTS to be perceived as having said or done - it is the suggestion from the person or organization of how they want their story presented. To change it after the fact is an attempt to disclaim that they were the ones who wanted the story to read a certain way, so that if there are questions about it later, they can claim they never said otherwise. It's inapropriate and dishonest, and entirely within the character of this administration, sad to say.

Posted by: kriselda jarnsaxa at November 1, 2003 04:35 AM

Jack : Some people, such as Fearless Leader, only read the headlines. http://www.foxnews.com/story/0,2933,98006,00.html

Posted by: AC at November 6, 2003 06:12 PM