How to Peer Review on the Internet

Recently, the string theorist Lubos Motl complained about the "proliferation of crackpots on hep-th" ... an idea which actually coincided with some concerns of mine in recent days.

Not being a practicing physicist myself, I am not subscribed widely to science journals, so I rely on the ArXiv.org website to get direct access to scientific papers. The point of ArXiv is that scientists have a place to post electron pre-prints (sometimes called e-prints) of papers, presumably while they wait out the peer review process in a formal journal, allowing for broader access to the papers.

Well, in the Motl post he laments that the ArXiv is becoming a bit of a victim of its own popularity, and a dumping ground for "crackpots" (especially in the hep-th portion, which stands for "high energy physics - theoretical" - in other words string theory, loop quantum gravity, and other significant and highly controversial theoretical physics papers).

In a response to Motl's blog, I suggested a pseudo-peer review system:

I wonder if the ArXiv could implement some sort of feedback system, perhaps locked only to those who have somehow been verified to hold PhD's (although it's unclear exactly how this verification would happen), so that anything could be posted, but there would be an internal system of "pseudo-peer review" that these experts could rate the articles based on their own review of the documents.

Presumably there would still be some danger of a crackpot article gaining a following, but I think that the majority of people who go to ArXiv do so out of a genuine desire to learn more about science ... and have no desire to push a crackpot notion. A crackpot article could get a few dozen votes in favor of it, but a solid scientific article would have dozens or perhaps hundreds of votes in support.

This could be supported if the identity of the voters was verified in some way, and it would be possible to obtain a list. Scientists would be hesitant to lend their support to a crackpot idea if they know that it's going to be on record ... unless, of course, they genuinely believe strongly in the crackpot idea, in which case that's their prerogative.

Well, it turns out that (according to the Wikipedia article on ArXiv) there is in fact a system in place in which a group of moderators review the papers, and a newer system of "endorsements" in place ... but both of these have apparently been criticized as being too limiting (if not outright censorship) and, when you view the archive, these ratings are not at all apparent.

What I'm proposing would essentially open every ArXiv article up to be peer reviewed by volunteer physicists anywhere in the world ... a type of "sharing economy" as discussed by Stanford law professor Lawrence Lessig in his recent book Remix: Making Art and Commerce Thrive in a Hybrid Economy.

The idea is somewhat spontaneous, and there are probably logistical considerations that I haven't yet taken into account ... but what do you think about this proposal? Would it be an effective way to separate the crackpots from the justifiable papers, or would it just mean that more crackpots would gravitate toward these open access sites, hoping to boost the ratings of their particular brand of crackpot-ism? Would the public nature of the ranking actually keep scientists from supporting the crackpot papers? Or, for that matter, is Motl's complaint invalid in the first place, and crackpot articles are not in fact a problem? Let us know what you think!