Our weblog & articles get comfortable
and have a read

Conducting research on the Internet:
Part 1: Addressing skepticism

by Dr. Alison Darcy 4 July 2008

When I first started using the internet as a tool for psychological research in the field of eating disorders in 2002, there was wide-spread skepticism around the viability of this approach to scientific investigation.  By the time my two-year research project came to be evaluated in 2004, we still had no-where to turn in terms of an Irish academic appropriately qualified in both fields - internet research and eating disorders.  Concerned about the skepticism that many academics and psychologists had about internet-delivered services, we went instead to Stanford University.  Stanford were (and are) leaders in the field of developing internet-delivered prevention interventions for eating disorders, and our work was awarded with distinction.  Believe me when I say, if it’s good enough for Stanford, home of some of the world’s most highly respected labs, it’s good enough.

I am not the only one that has experienced skepticism first hand.  An acquaintance of mine, an experienced clinical psychologist and forerunner in the area of internet-delivered therapy, told me that he had once submitted a white paper to a scientific journal discussing the usefulness of the then, newly developing internet, as a tool for discussion and therapy.  As a sex therapist, he spoke about how the internet might become an ideal forum for people to discuss issues of a sensitive nature, he foresaw group forums and even message boards.  His paper was dismissed outright, with one reviewer commenting that whenever they started publishing science fiction, they’d let him know.  

In many ways, many of the most common concerns held by many in the scientific community were, in themselves, wildly unscientific, not having any basis in evidence.  Skepticism in a scientist is, of course, a healthy trait.  But the kind of skepticism that prohibits one from objectively considering the viability of something new, is not.   

Fortunately, we now have good evidence that supports the use of the internet as a research tool as viable as traditional methods.  And yet, I am still surprised at how many people think of Internet-mediated research as of somehow lesser value – somewhat dodgy even.  In fact, the opposite is often true.  Sometimes it is a good thing to eliminate humans from scientific evaluation.  There are, however, a number of special considerations in terms of sampling procedures, and ethical questions that I will discuss in later blogs.

This is the first part of a series of blogs that we will be publishing on internet research.  In this first blog I will address some of the most common concerns around conducting online research.  If I have not addressed a question that you would like answered, please submit a comment and I will address it.

What are the concerns?

1.    Results from internet studies will be adversely affected by non-serious or malicious participants.

This is probably the most commonly cited concern, not just for internet research, but for all internet-delivered services, with the anonymous nature of the internet being at the route of concern.  In research terms, this will largely depend on the sampling procedure employed, the agency conducting the research, and in some cases, the topic.  A call from a generic website to the general public to complete a survey measuring attitudes toward the Bush Administration may attract some malicious responses.  However, a request to complete a set of questionnaires from a reputable non-profit organisation is not as likely to attract malicious users.  There is little value in corrupting this kind of research on a nonprofit website.

Theoretically, it is as easy to fudge a paper and pen questionnaire as it is an internet-delivered one.  It has long been established that traitional survey methods have their own difficulties, especially in Univeristy settings, where much psycholological and sociological research takes place using students as the sample of convenience (for a full discussion, see Gosling, Vazire, Srivastava, John, 2004).

Having said this, it is possible to test whether web-delivered questionnaire data have been adversely affected by non-serious respondents.  Johnson (2001) suggests screening each set of responses for suspect markers such as long strings of identical responses (eg. all 1s).  Another method, also employed in paper and pen questionnaires, is to check the reliability of the scale (setting Cronbach’s alpha coefficient of at least .7 as the acceptable threshold) as well as discriminent validity (John & Benet-Martinez, 2000).  Gosling et al. (2004) investigated the reliability and validity of a web questionnaire versus a paper and pen version of the same questionnaire, and found that both of these attributes tended to be equivalent regardless of the presentation medium providing evidence to suggest that web questionnaire data can be at least as valid and reliable as that yielded by conventional paper and pen questionnaires.

2.    Internet samples are not representative or generalisable.

This may have been a valid concern in the early days of the internet, however, internet use is rising equally among all groups of users, and the ubiquity of computers in schools and libraries, not to mention internet cafes, also challenge this assertion.  Broadly speaking, generalisability of findings to the general public may still be a problem (Couper, 2001a; Robinson, Neustadtl, & Kestenbaum, 2002; Smith, 2002).  But having findings generalisable to a specific group is a different matter, and we would argue that in some cases, internet research can make it easier to recruit participants from specialist groups.  Eating disorders are a good example.  

Eating disorders have low base rates in the population relative to other mental health difficulties.  In addition, clinical samples tend to be biased in some respects, favouring those in upper socio-economic brackets who have better access to clinical services and are better at availing of them.  In this case, recruiting from community samples when investigating things like risk factors and causation is more sound from a methodological point of view, since the data paint a more representative picture.  How is one going to recruit from the community?  In schools and colleges etc.?  Yes, but this still means that findings are, at best, only generalisable to school or college-aged people.  Conducting a large-scale, two-stage screening study, the most favoured method for conducting incidence/prevalence studies that ensures good generalisability, is a very expensive and resource-intensive endeavour, and not an option for most research teams.  Internet studies make participant recruitment much more efficient, and samples are not restricted by geography, age, profession, etc.  Conducting research from a website for people with eating disorders (eg. Bodywhys, the National Eating Disorders Association of Ireland’s website, www.bodywhys.ie) means that one can recruit from the appropriate population pool, efficiently, and cost-effectively.  In an evaluation study of an online support group, we found that 60% of our sample were not in treatment (Darcy & Dooley, 2007).  This is in line with other studies reflecting how many people with a particular disorder are not in treatment and thus would not be included in research that uses clinical samples.   Thus, we argue that in fact, findings from internet studies can be more representative than those from traditional methods.  

3.    Results from web-based studies are not representative of those found in studies employing traditional methods.

There is now a large body of evidence to suggest that across a wide range of topics and constructs, effects obtained from internet studies are consistent with findings obtained from traditional methods (eg. Graham & Papendonatos, 2008; Vallejo et al., 2007; Kongsued, Basnour, Holm-Christensen & Hjollund, 2007; Ritter, Lorig, Laurent & Matthews, 2004.)

4.    The same ethical safeguards that are employed in traditional studies are not possible or compromised in web studies.

A recent report of the Board of Scientific Affairs Advisory Group on the conduct of research on the Internet concluded that the internet has “democratised data collection” (Kraut et al., 2004).  This means that the internet can facilitate more people than would normally be able to access populations to conduct research.  The flip-side of this means that less-qualified people can conduct research, people who may not have formal training in research methods, and so the quality of research can potentially be compromised.

There are now very good ethical guidelines for conducting research studies on the internet.  There are of course, special considerations when conducting internet research.  I will cover these in more details in a subsequent blog.  However, once these considerations are addressed, we find no evidence to support the notion that internet studies are any less ethically sound than conventional studies.

5.    Findings from web-based studies will be difficult to get published in reputable journals.

I have a lot of experience searching for peer-reviewed research articles on major article databases.  As a brief exercise, I went to PubMed, one of the largest of these databases for health-related published research, and typed in “internet” as a keyword. This query returned 34, 645 articles.  There are many journals that are specific to internet research as well as those that publish research on internet-specific constructs such as Journal of Medical Internet Research (www.jmir.org );  CyberPsychology and Behavior (http://www.liebertpub.com/publication.aspx?pub_id=10 ); and Telemedicine and E-Health (http://www.liebertpub.com/publication.aspx?pub_id=54 ).  Internet-research appears in more traditional peer-reviewed journals.  There are many examples of internet randomised controlled trials that are published in very high impact factor journals such as the American Journal of Psychiatry (132) and the British Journal of Psychiatry (108).  These journals typically employ the most stringent review procedures.

In broad terms, as long as research has a sound methodology, is conducted ethically, and makes a contribution to knowledge, then it should not matter whether findings are yielded from an internet study or from more traditional methods.