Friday, May 29, 2009

On Standards

One reason I am perhaps a bit quieter on this blog lately than I used to be is because frankly I've reassessed my standards.

I like writing about interesting and (what I consider to be) important things, but I have also become very aware that even the most well-meaning and innocent attempts to share one's interests and opinions publicly can end up adding to the "noise" component of the signal-to-noise ratio on any topic.

Obviously I am not going to demand that either I or anyone else first achieves "perfect" knowledge before writing anything about anything.

(The only people I've seen actually demanding that are people who tend to be enthusiastic about various forms of pseudoscience and superstition. Actual scientists (at least if they're doing actual science), and reasonably scientifically literate laypersons, on the other hand, generally seem to understand that opening up one's ideas to criticism is part of the picture when putting said ideas out in public.)

But: nevertheless, the fact that asking for perfect knowledge of anyone who wishes to communicate about anything is ridiculous does not mean that standards don't matter or that it isn't irresponsible to not check one's own standards frequently. The bottom line is that these days, when I want to write about something weighty or (per my own standards) important, and I have the slightest inkling that there might be actual, factual data pertaining to that thing, I at least make an effort to find and understand the primary sources.

Now, I freely acknowledge that I am not always good about listing citations. And I do not think there is no place for "informal" blogging. I am not suggesting that we ought to all be totally stodgy, never silly, and never say anything without reference to a research paper -- that might help improve accuracy in certain topics, but it would also possibly effectively crowd out people who just might have something worthwhile to say despite not having standard or expected credentials, style, etc.

But when it comes to certain topics I guess I've just come to see it as important to avoid letting either enthusiasm or concern wreak havoc on standards.

For example: If someone tells you something you're writing about is "fringe science", don't take that as an insult -- lots of things start out on the fringes, and certainly it doesn't help your case to insist that science lower its standards to admit your pet idea rather than you going off and trying to actually get your idea tested.

(This is a litmus test I always invoke now when coming across a new claim about longevity medicine -- I am a lot more likely to take seriously efforts that are looking to submit their ideas for testing in stringent conditions than "efforts" whose primary energy expenditure seems devoted to going on about how they "have a secret your doctor doesn't want you to know!")

And if someone tells you that your well-intentioned efforts might actually be having the opposite effect to what your intentions claim, you might want to try and see if you can see what they are talking about.

(This is especially important for people (and groups) who have achieved some level of influence, and it can be very weird sometimes for people accustomed to having little to no influence to suddenly find themselves in the position of having it.

I know I was totally baffled and slightly disturbed when several people e-mailed me after seeing my two minute BBC interview on the Visions of the Future special asking me what I thought about the future of human life -- that definitely gave me a sampling of how even having been on television for a tiny speck of time and portrayed in an even remotely positive light makes people more likely to think you have some kind of "expertise" even when you might be making as wild speculations as anyone else could.)

In short, all humans need to be careful of letting confirmation/deconfirmation bias undermine their goals in trying to transmit accurate information, particularly when the subjects they claim to care about (and the public discussion thereof) has the potential to impact so many who may presently have very little power.

So, when I write these days, it is taking me longer (that is, when I am not making obvious personal speculations that make no claims of being super-sciencey or posting photos or robot-themed artwork or metablogging like this). This does not mean my interest has dropped off in anything -- it means I've realized I really do not want to be part of the noise problem, whether the subject be biogerontology or autism research or anything similarly weighty. I know I won't be perfect at this but I am willing to try harder at least.

12 comments:

FrF said...

One thing I noticed immediately after I started to read EIW two or three years ago is Anne's carefully crafted, unusually self-reflective, graceful prose!

Confirmation/Deconfirmation Bias: Do I detect a slight "Overcoming Bias" influence here?

(I'm more certain about Dale Carrico's influence on EIW's general outlook. My hunch is that Anne's current distance to the "Transhumanism" meme owes itself not in small part to Amor Mundi.)

I noticed a similar subtle change in tone -- thanks to OB and Bayes, I assume -- at Accelerating Future where Michael Anissimov is talking a lot more about "probabilities" than in the past.

AnneC said...

FrF: Huh? The notion of "confirmation bias" and its inverse have been around way longer than "Overcoming Bias: The Website" has. I know about OB and have read it a bit but no, I would not say that using the word "bias" here with or without modifiers means I have been "influenced".

As for the "Amor Mundi" influence: that there probably is some of but only in the sense that once I got what Dale (and Richard Jones and other folks who have written critically on superlativity) were going on about, I realized how much superlative notions tended to "shoot themselves in the foot", so to speak, and lead to a really obnoxious "you are either a transhumanist or a Luddite" dichotomy that really tends to impoverish discussion and encourage the lauding of "armchair philosophy" far more than it deserves.

AnneC said...

Actually if anything influenced this post it was probably this item on MindHacks discussing (in part) the lack of fact-checking that seems to be going on in journalism these days. In particular this bit:

"But despite getting lots of opinions from everyone from attention researcher David Meyer to lifehacker Merlin Mann only one single 'study' on the distracting effect of technology is mentioned in the New York Magazine article: "people who frequently check their email has tested less intelligent than people who are actually high on marijuana".

This is quite amazing because not only was the 'study' in question not an actual scientific study, it was PR stunt for Hewlett Packard, this isn't even an accurate description of it (users were interrupted with email during an IQ test and scored worse, big surprise)."

Now mind you the MindHacks article was written in response to another article on "digital distractions". But I am not trying to spark a discussion on that subject right here, it is actually kind of irrelevant.

Rather, the relevant point for me is just how easy it can be to write something that is poorly fact-checked and rife with lots of speculation and opinions and over-simplifying interpretations -- and get it prominently published.

And then there's this pattern I have noticed as well where issues get over-reduced to two "sides" where it is just assumed that if you are critiquing the standards to which something is apparently being held (or not held), you must be in fact questioning the author's "good intentions" or "hard work".

joel said...

i have goals that i do not understand. so i refer to my feelings sometimes for my searching, even guessing. i had to give myself a few items of faith for relief. one, is that i have a soul and that i am my soul. from that followed that i have a home where i belong, somehow, and that i will get there, somehow.

i was in a coma once a few years ago. i had terrible nightmares. it was a 'semi-coma';i still had high brain activity. now i have similar nightmares. so i have mysterious challenges. i have incomplete scientific knowledge of my situation to say the least.

i believe it is my right as a being to have my mind be my own. but i am not free from intrusive thoughts, memories, dreams and emotions. i have scraps of truth to go along with. i'd never get anywhere without some guessing, intuition, even irrationality.

i read existence is wonderful looking for more to use in my work, which is to create myself better daily. i'm looking to others for anything good, proven or mysterious.

i'm not as sharp and accomplished as some of my peers but i strongly feel we are in the same boat around these sites; overcoming bias, less wrong etc... i know if i stay involved in these things, and my music, paintings, therapy, education, family - more good is possible.

i'm very much here to sing harmony in the noise, though my thoughts have so many signals and noises.

so, on standards, perhaps intent is key. deep consideration and good intent could be something important to check in our comments.

outlawpoet said...

It's an interesting question, if you're looking to increase the quality of your output, do you write more, or less?

WillowBl00 over on livejournal had a discussion on that very topic where she encouraged people to be quieter on the internet to improve their signal/noise ratio. She was particularly polemical about twitter.

I rather think quite the opposite. If I had the time I'd be writing every day. I don't think that holding back improves ideas and works, but practice, thought, and feedback do. I'd much rather have a portfolio with twenty really good works, and many middling ones, than five good works and few middling ones.

Perhaps that complicates another issue you touch on, which is what people think of you and your work. WillowBl00 might not like me, given my many middling works, my chatty twitter account, my participation in lj memes. I think people need to filter what you're doing anyway, even if you only wrote one paper a year, unless you were of great scientific stature, nobody would read every one.

People's interests overlap in surprising ways. I've spoken with EY in the past about AI, of course, but we've also talked about roleplaying games. We almost ended up doing a webcomic together, back when I had more time. (I muffed that one). I think that someone with more output has more value, just because if it's too much for me to read, I'll filter, but there's more of a chance they'll touch on something I'm interested in.

This has an interesting intersection with work requirements. If you were, for example, to dedicate most of your time to writing papers for publication in peer reviewed journals, your production of writings would be much slower, and more painful. The time required for each one goes up a great deal because of standards and practices. This is undoubtably valuable for the journal you're submitting to, and their readers. But it is valuable to you, and the generic reader, me, who would read your work anyway? I'm not sure.

I'm not convinced that such academic standards do more than weed out obviously incautious thoughts, and force people to defend themselves before publishing. It's another kind of filtering, which, quite honestly, I think I can do better. (This is quite separate from journal or scientific requirements for experimental data and design, which can be very valuable indeed)

So I suppose my response is, standards for what? For what purpose? Who benefits? I think in many cases, inspection of such standards would rather lead you to write more. Oh there are other things, like cultivating a private place to put most of your writings, which lies behind a public place which only gets the best, for example, or working to get only very good things published wider and comment fiercely on your blog, but these are details. Presentational, I think.

FrF said...

Anne, of course I know that OB hasn't "invented" cognitive biases. I just think -- and this is based only on anecdotal evidence -- that these concepts are more influential since OB rose to prominence in those quarters of the blogosphere that care about such things.

I hope you were not bothered by my speculating!

AnneC said...

FrF: I find your speculating perplexing and vaguely annoying (it feels a little like being psychoanalyzed) but it's nothing I'd lose sleep over.

Lindsay said...

I worry about this stuff, too, since I also do a fair amount of sciencey blogging. Am I understanding the research I'm writing about correctly? Have I described it in a way that is simultaneously accurate, detailed and penetrable by most people? Are the original ideas I add --- whether in the form of extra background information, tie-ins, broader contexts, potential implications or criticisms of the researchers' methods or interpretations of their findings --- good ones?

This is why my sciencey posts take me several days to write.

(Yes, I just finished one. I think the next thing I post will be a pretty picture!)

AnneC said...

Lindsay: YES, to pretty much everything you listed. That is exactly the kind of thing I think about. I know that with a blog I have the freedom to write whatever comes to mind, but I don't want to just yammer on speculatively about any old thing...nothing wrong with fun, frivolity, or pretty pictures, but I definitely think there's something to be said for being clear about the KIND of post one is doing.

Re. your recent post -- I notice it has the "Research Blogging" tag. I have been seeing that around and wondering how one tells if they are doing "research blogging". Does it mean you are, in your post, commenting on and referring to some (hopefully reputable) primary source? And if so, does using the tag require being able to access the full text paper? Of course the full text paper is best but I have seen some cases of abstracts being decently extensive.

AnneC said...

(And re. my last comment, I want to make it clear that I am NOT suggesting anyone go out and read a bunch of paper abstracts and assume on that basis they know all about what the paper was really about...just wondering if it is EVER appropriate from a research-blogging-integrity to comment on just an abstract if that's all you can access at a given point in time. If it's not, that's fine, I just am curious about what the standard is here!)

Lindsay said...

Yes --- there's a website, .ResearchBlogging.org, where you can sign up, generate citations for your posts, and upload your posts to their site.

They have a set of rules posters are supposed to follow, which include having read and understood the entire article, including a formal citation and a link to the original source (and/or a DOI or other identifier), and reporting accurately and thoughtfully on whatever research you're writing about.

As to the reputability of the source, all it says is that they (the original papers, not your posts!) must be peer-reviewed.

AnneC said...

Lindsay: Oh cool, thanks for the link. I am definitely intrigued by this Research Blogging concept!