Cites & Insights: Crawford at Large
ISSN 1534-0937
Libraries · Policy · Technology · Media


Selection from Cites & Insights 4, Number 12: October 2004


Perspective

Wikipedia and Worth

Walt Crawford

Late summer saw a whole bunch of foofaraw about wikis and specifically Wikipedia. After one columnist suggested Wikipedia as a resource for computer history, other writers assaulted Wikipedia as worthless trash; at least one librarian made noises about the difference between online junk and authoritative sources; some wiki advocates pontificated about the awesome error correcting capabilities of community-based collaborative media. Alex Halavais of the School of Informatics at Buffalo University made 13 changes in the English language Wikipedia, “anticipating that most would remain intact and he’d have to remove them in two weeks.” Presumably, if that had happened, there would have been evidence that the ease of modifying Wikipedia makes it suspect as a resource.

Long-time readers may be aware that I haven’t found wikis suitable for my own needs—and that I once, more than two years, concurred with a published “pan” of Wikipedia. My offhand comment at the time called it “one of those grotesque ‘let’s all make an encyclopedia’ efforts...that help some of us appreciate professional efforts.” That was a cheap shot, although based on what I remember of Wikipedia from early 2002, I’m not going to apologize for it.

So what happened this time? Am I going to rant about the uselessness of Wikipedia and why only Authoritative Sources (those blessed by/part of traditional media) should be used? Read on.

The Halavais Test

Some people assaulted Halavais for his deliberate vandalism of Wikipedia. I won’t get into that discussion, although the fact that Halavais fully intended to remove the changes counts for quite a bit.

More significantly, all the changes were identified and removed within a couple of hours. Halavais reported this and found himself “impressed.” Vandalism may be less of a problem than some might have thought—if it’s readily detectable vandalism, e.g., simple graffitiesque changes or changing facts that can be readily verified by a Wikipedia contributor or editor. (Another tester made a series of more subtle changes and says none of them were corrected over the test period.)

One Wikipedia technical team member noted “some of the hurdles a vandal has to deal with”: a “Recent Changes Patrol,” personal watchlists that inform contributors of changes made to articles they’ve registered interests in, the ease of tracking all edits from a given IP address when one edit has been identified as vandalism, “the people” and the enormous rate of Wikipedia edits, and tools for dealing with persistent vandals. It’s an interesting list (frassle.rura.org, August 30, 2004). I could take issue with part of one paragraph, following the note that there were almost a million edits in June 2004:

The articles are being improved at a tremendous rate and even obscure changes are likely to be noticed within weeks or months, with the time depending on just how obscure the article is. Obscure is potentially harmful to fewer people and perhaps more likely to be seen by those who have knowledge of the topic sufficient to spot clear mistakes.

“Improved” is a value judgment not automatically implicit in a fast rate of change. Maybe all those edits are improvements; maybe not. My real problem is with the idea that errors (deliberate or otherwise) in obscure topics are less important. I think it’s the other way around. Obscure topics can’t be verified as readily against other sources. If Wikipedia had 1869 as the end of the Civil War, it would be an obvious and readily-verifiable error. If Wikipedia asserted that HTTP GET should never be used for URLs in excess of 256 characters (as opposed to the reality, that a fairly old RFC notes that some old servers may not handle very long HTTP GETs properly), a user might not have an easy way to double-check.

The Halavais test is an interesting datum. The Wikipedia community can spot changes in existing articles fairly rapidly and has good tools to deal with troublemakers. Does that make Wikipedia a trusted source? Of course not—but, for me at least, it reduces one cause of angst about taking Wikipedia data at face value.

Other Commentaries

I haven’t collected the full range of comments on Wikipedia, the offending newspaper article(s), and the Halavais test. I did note two interesting commentaries (one of them not about Wikipedia as such), one by Ed Felten at Freedom to Tinker, one by David Mattison (a wiki user) posted on Web4Lib.

Ed Felten notes the two sides—“Critics say that Wikipedia can’t be trusted because any fool can edit it, and because nobody is being paid to do quality control. Advocates say that Wikipedia allows domain experts to write entries, and that quality control is good because anybody who spots an error can correct it”—and goes on to note that much of the debate ignores the best evidence: The actual content of Wikipedia.

Felten took a look at its entries on “things I know very well: Princeton University, Princeton Township, myself, virtual memory, public-key cryptography, and the Microsoft antitrust case.” His findings? The first two entries were excellent. The entry on Edward Felten was “accurate, but might be criticized for its choice of what to emphasize.” (They also had his birth date as uncertain, which he corrected.) The technical entries “were certainly accurate, which is a real achievement” and were both backed with the kind of detailed information that wouldn’t be feasible in a traditional encyclopedia—but neither did a great job making the concepts accessible to non-experts. As he notes, that’s a quibble.

Unfortunately, the article on the Microsoft case was “riddled with errors”—factual errors, mischaracterization, terminology errors. His conclusion?

Until I read the Microsoft-case page, I was ready to declare Wikipedia a clear success. Now I’m not so sure. Yes, that page will improve over time; but new pages will be added. If the present state of Wikipedia is any indication, most of them will be very good; but a few will lead high-school report writers astray.

Comments asked about possible heuristic indicators for estimating likely accuracy and offered plausible reasons (“gravitational pull of content”) that some articles may be oddly focused. (Entries are written to “scratch an itch,” and it’s likely that one entry will generate related entries.)

David Mattison addressed the overall issue of whether a wiki is appropriate for scholarly communication. His answer: “Banks are probably not appropriate for keeping money and valuables because they get robbed.” Thus, many banks and wikis have gatekeeping and security protocols to keep the valuable cash and data from being tampered with—but wikis can operate with totally open-door policies. “It’s the very nature…of this ideal type of wiki that makes some of us nervous and thrills others for various reasons, not all of them socially acceptable.”

Mattison goes on to say that a wiki can be highly appropriate for scholarly communication if all the scholars trust one another, are collaborating on something, and use appropriate security and rollback mechanisms. These concluding paragraphs firmly separate Mattison from extreme “the community is always right” advocates:

Wikis are just another tool in what I, borrowing from others, call the Collaborative Web: technology and applications that let individuals work together or independently directly through the Web browser without a gatekeeper (e.g., a Webmaster) standing in the way.

The question of whether what emerges from that collaboration is authoritative or scholarly depends on other factors often above and beyond the collaborative process itself.

That doesn’t address Wikipedia directly, to be sure; neither is it intended to. I find nothing in Mattison’s post to disagree with. Wikis do add another interesting tool that can be used for good or bad. The fact that I haven’t found wikis useful (as a participant) to date is just that: A fact with no broader significance, similar to my lack of a personal weblog.

Later—after I wrote the first draft of this perspective—Ed Felten added two more comments, and I did my own testing. Felten’s first comment (September 7) was a “Wikipedia vs. Britannica smackdown” in which he looked at the same seven topics in the Britannica. Wikipedia did a little better on Princeton University and Township and a lot better on Edward Felten and virtual memory (neither of which has articles in Britannica); public-key cryptography is a tossup—but, while Britannica’s coverage of the Microsoft antitrust case is minimal, at least it’s not wrong. His overall verdict:

Wikipedia’s advantage is in having more, longer, and more current entries. If it weren’t for the Microsoft-case entry, Wikipedia would have been the winner hands down. Britannica’s advantage is in having lower variance in the quality of its entries.

The comments on that post are almost as interesting as the post itself. Scott Preece notes that the crown jewels of the Britannica are its long essays, not its short entries. Another poster asserted that Wikipedia is a “revolution” that will “outstrip all other encyclopaedic forms soon” simply because it grows so fast—to which another, suggesting a scientific survey, noted that quantity does not automatically lead to quality. And Eric Burns offered a good comment to the effect that most of the test cases are biased toward Wikipedia by their nature; “It is not unlike comparing the Catholic Encyclopedia with Britannica, and choosing a statistical universe that is 80% theological as its basis.” Burns does believe Wikipedia “will one day eclipse Britannica.”

Felten’s September 8 post questions the claim that constant and rapid change leads to continuous improvement in entries, so that in the long run Wikipedia becomes better than conventional encyclopedias. He suggests that, as entries mature, the quality will level off, with additional changes “executing a quality-neutral random walk.” He suggests a similar story for Wikipedia as a whole and asks whether enough effort will be spent to reach a quality plateau—and what that plateau should be.

Perspective

Wikipedia is certainly not worthless. Wikipedia is also not automatically better than a traditional encyclopedia because of the community of writers. I would tend to use Wikipedia entries as starting points, to be used on a “Trust but verify” basis. But isn’t “trust but verify” the base heuristic for almost all resources, traditional or new?

My assumption is that lots of specialists have contributed good work to Wikipedia, particularly in areas related to the web and digital resources. My assumption is also that some Wikipedia content is faulty, biased or wildly incomplete. In the latter case, I’d make the same assumption about a traditional encyclopedia, up to and including Britannica. Personally, I doubt Wikipedia will “eclipse” traditional encyclopedias (note that Britannica is once more available in print form), just as I doubt that weblogs will replace newspapers or that econtent will sweep away print media. Another comparison may be apt: While Encarta may have doomed Funk & Wagnall’s (and incorporated it at one point), it hasn’t doomed Britannica—or vice-versa. Different forms, different media serve different people with different needs.

I did my own tests—only of Wikipedia, and if I was going to compare it with a traditional encyclopedia I’d probably use Encarta or its ilk. My results were mixed. Some entries were very good and reflected considerable expertise (I was particularly impressed by the network of entries on lossy compression technologies), while some were dry as dust and gave no flavor of the subject they dealt with. I didn’t catch obvious errors, but I wasn’t really looking.

What is an authoritative resource? How about articles in refereed STM journals? Are they automatically trustworthy because they’ve been through that rigorous peer review process? Not really, as candid observers within the field will tell you. Very few observers argue with the cynic’s view of peer review as it applies to authors with an axe to grind: Peer review doesn’t determine whether an article will be published, only where it will be published.

There was a dustup on the SPARC Open Access Forum about quality and inclusion in indexes. Implicit in the dustup—which I won’t review since it’s pretty specialized—is the notion that a great many peer-reviewed journals are full of crap. Sometimes that’s implicit (to a knowledgeable observer) from the name of the journal. Sometimes you need to be aware of the standards of the field to know which journals are, in the words of one observer, essentially vanity presses.

This is a tangent, but I sometimes wonder about the dissociation of article and journal that appears in online aggregation and OAI. Specialists will have their own mental lists of first rank, second rank but trustworthy, and FoC [see previous paragraph] journals; they’ll use those mental filters to judge the worth of new articles. But what about non-specialists? Are those filters readily available? Who will tell a person looking for health guidance which peer-reviewed journals to avoid at all costs?

I think there is a connection to Wikipedia. Alternative publishing does not imply lack of worth. Traditional publishing isn’t an automatic indication of worth or veracity. If the key is “trust but verify,” we need better ways to verify likely worth and probity. I assume today’s librarians are finding such ways, and finding ways to communicate those methods to the rest of us. At least I hope so.

Cites & Insights: Crawford at Large, Volume 4, Number 12, Whole Issue 54, ISSN 1534-0937, is written and produced at least monthly by Walt Crawford, a senior analyst at RLG. Opinions herein do not reflect those of RLG. Comments should be sent to wcc@notes.rlg.org. Cites & Insights: Crawford at Large is copyright © 2004 by Walt Crawford: Some rights reserved.

All original material in this work is licensed under the Creative Commons Attribution-Non­Commercial License. To view a copy of this license, visit http://creativecommons.org/licenses/by-nc/1.0 or send a letter to Creative Commons, 559 Nathan Abbott Way, Stanford, California 94305, USA.

URL: citesandinsights.info/civ4i12.pdf