Monday, May 20, 2013

The state of statistical analysis in LIS research

I've been asked at MPOW to conduct some kind of training in the use of statistical analysis for our librarians who are just venturing into the research realm.  I'm actually a bit excited about this - it has been a while since I've done any training, and I would also like to contribute to the improvement of the quality of research being conducted by LIS professionals.  In my application essay for the MPH program, I mentioned that I was dissatisfied with the level of quality in LIS research and was looking forward to participating in a field with a greater rigor.  Of course, everything is relative.  Those in medicine consider much of research in public health to be of low quality.

While I have had an intermediate level of graduate training in statistics, I did struggle with the more advanced aspects associated with my job of analyzing the data in the clinical trials, as biostatistician.  I was asked to run multilevel hierarchical analyses on the repeated measures to look for effects of trauma on levels of cortisol in teenagers.  Yeah, "What?" was my reaction, too.  I read and re-read and read again the literature on this method and ran the procedures in SAS exactly as it was written.  I still didn't understand it.  I gave my best interpretation of the results to my PI (primary investigator), with lots of caveats.  She said that made sense to her, too, but she was an MD, not a statistician.

So I conceded the limits to my abilities as a biostatistician and did not pursue that career any further.  But I do feel I have a good understanding of, well, an intermediate level of statistical analysis.  So I feel confident that I can conduct the most common kinds of research in librarianship, as a practicing librarian.  And I also feel confident that I can introduce my peers who haven't had this training (or at least, not for a long time) to the basic concepts.

But I really would like to know what the current state of statistical analyses is in LIS research.  Analysis does go hand-in-hand with research methods (you can't adequately analyze data statistically if the methods used to collect the data were inappropriate, incomplete, or inadequate).  So I've been scouring the literature looking for studies that addressed my question...but I haven't found a whole lot, and what I did find was not inspiring.  In 1984, Charles R. McClure wrote a letter to the editors of CR&L admonishing them for publishing a research article of low quality, stating that "the 'research' was so poorly done, that the results have little meaning."  His diatribe continues (emphasis is added):
The sample is never shown to be a valid representation of academic librarians or a subgroup of academic librarians and thus, is not generalizable (especially with a 52% response rate), a copy of the questionnaire is not available as an appendix for the reader, key definitions (such as what exactly constitutes an "article") are not provided, huge assumptions are made as to participants' interpretation of questions, no explanations of the limitations and weaknesses of the study or its findings are offered to the reader, statistical techniques are poorly utilized, and there are no indications of the reliability or validity of the data reported.1
Mr. McClure had been publishing for quite some time prior to this, and perhaps the topic of the article (librarian tenure) was particularly close to his heart given that his dissertation was on job titles.  But his tirade did not go without notice.  Just a few years later, an article was published providing an overview of basic statistics and some resources for future reference. The authors, Donald Frank, Leslie Madden & Nancy Simons, referred to McClure's letter as the impetus for their article.
This paper focuses on a rationale for the correct use of statistical procedures and techniques.  Basic assumptions and elements of descriptive and inferential statistics are noted.  The importance of the thesis or problem statement as well as the relevance of hypothesis testing is emphasized. Additionally, ways to become more familiar and comfortable with statistical usage are reviewed. The paper is written for the librarian who is not aware of these basic techniques, and is interested in research processes.2
But that was published in 2001 - has there been any more recent examinations or evaluations or discussions since?