Sign onto a social network or online forum and, in theory, you should be able to find a network of people who might give you useful advice. That’s probably true for getting a tattoo or baking a ham. But it’s tough to get advice online about technical issues from practitioners who are really experts. Particularly in science and engineering, knowledgeable people generally have better things to do than hang out on forums dispensing pearls of wisdom.
On this score, it is interesting to see what happened when research journals tried to use online forums to improve the peer-review process. The thought was that commentary from the “community” was preferable to that coming from a small enclave of peer reviewers. The goal was to get around the possibility of having potentially important research papers blocked from dissemination by a few experts who might have axes to grind or hidden agendas.
There was another reason for getting away from limited peer reviews: The evidence is that peer reviewers tend to miss most errors in research papers. Better, it was thought, to mimic what happens with open-source software when a community of online experts ferrets out problems.
But peer-review journals that tried this model have rapidly abandoned it. The drawbacks quickly become evident in an episode related by David H. Feedman, a science journalist who, in his book aptly titled Wrong, examines why expert opinion is often incorrect. In 2006, the scientific periodical Nature tried collecting online comments about papers considered for publication. Few authors were interested. Of 1,369 eligible papers, authors of 71 agreed to have their work posted. Thirty-three of those papers received no comments at all. Comments totaled 92, and more than half of them were on just eight papers.
Journal editors who examined the comments didn’t think much of them. They also found it difficult to get established researchers to make observations. The remarks that did come in were so general as to be useless (a typical post: “Nice work”). A former Nature editor told Freedman, “Scientists are sufficiently busy that they aren’t motivated to troll through the Web to write commentaries on whatever catches their fancy.”
Substitute the word “engineers” for “scientists” and the point is still true.
It looks as though the same could be said for most online forums that focus on technical content. To cite one example, a magazine called The Scientist started a forum in 2008 with idea of providing a place where a community could “identify breakthrough research and commercialization opportunities before they receive widespread attention.” It hasn’t worked out that way. As Freedman relates, only about 1% of visitors post a comment, and those who do tend to focus on career issues rather than technical ideas.
Of course, both researchers and engineers have good reasons to play things close to the vest when it comes to putting information online. Engineers can probably relate to one researcher’s explanation of why she didn’t post anything of substance: She’s busy. And besides, when she has a good idea she wants to save it for publication, not just throw it onto a forum.
— Leland Teschler, Editor