The impact of Google on Library databases

A recent thread on the Summon discussion list has spurred further discussions elsewhere.  The thread started with a complaint about the lack of predictability among the results of boolean searches in Summon.  Essentially, it was hard to determine why particular results appeared in the list due to other "relevance factors" that "polluted" the ranking of the results.  The librarian mentioned having a preference for "the pure drop" - predictability.  From this initial comment, other discrepancies of results of the same searches in different systems, notably relating to differences in processing boolean operators, stemming, and phrase searching.  This led to additional posts calling for "advanced" search functions, such as proximity operators, wildcard (or "hard stemming"), and search set manipulation, all of which could be described as "librariany".  The thread was pretty much ended by the comments from the other end of spectrum - those who were concerned that such changes would revert the Discovery system back to a typical library database.  These posters pointed out that such systems were purposefully designed not to be like your typical library database - built for the easy use by students and non-experts, not for the librarians.

This thread was picked up by Aaron Tay, Senior Librarian at National University of Singapore.  He started thinking about the effect Google and search engines in general have had on library databases, including catalogs and abstracts & indexes.  He considered how such systems have slowly adopted selected features and functions that users have come to expect in any search engine of any kind, notably ranking by relevancy, "implied AND", automatic stemming, and full-text searching (when possible).

As he discussed the implementations of these features, I noticed that predictability of search results is a common thread of concern among librarians.  We want to explain why each and every result is in that set.  We want to be sure we get "the pure drop".  This is an essential difference between librarians and the clients they serve.  Students do not appear bothered if there are a few irrelevant results - as long as they get more relevant ones than not.  They effectively "bleep" over them and move on.  Furthermore, for most students and public library users, the level of satisficing is lower than for librarians and some expert searchers.  These attributes may explain why our users (and many front-line reference librarians) love these Discovery services so much more than expert reference and research librarians.

In my view, Summon has done well to bring more resources to the attention of our users - full-text usage is up, as well as audiovisual downloads.  The expert searching features and functions should only be introduced if they do not interfere with the system's core feature - a simple keyword search interface.  Certainly, I agree that they should not "treat it like the 'normal'" library database.  There are different tools for different tasks - choose wisely.

Digging into Data Challenge - what's next...

While reviewing the ALA's summary of the ARL meeting in Chicago, my eyes locked into the Digging into Data Challenge.  Sponsored by various humanities and social sciences organizations, this challenge offers prizes and prestige to those who develop innovative and interesting ways of gaining insight from untapped data resources.  The first round was in 2009, with recipients proposing ways of analyzing literature, music notations, letters, speeches, images and railways to understand the human condition.  The recipients for the second round were announced just this past January.  Key projects that caught my attention especially included:

• An Epidemiology of Information: Data Mining the 1918 Influenza Pandemic
• Electronic Locator of Vertical Interval Successions (ELVIS): The First Large Data-Driven Research Project on Musical Style
• Imagery Lenses for Visualizing Text Corpora
• Digging into Connected Repositories (DiggiCORE)
• Integrating Data Mining and Data Management Technologies for Scholarly Inquiry

One that was particularly interesting to me was (emphases added):

Digging into Metadata: Enhancing Social Science and Humanities Research
Principal Investigators: Mick Khoo, Drexel University, IMLS; Diana Massam, University of Manchester, AHRC/ESRC/JISC. Additional participating institutions: University of Glamorgan. 

Description: The project will automatically generate new forms of metadata tags from existing metadata records and associated resources that will support discovery across multiple repositories.  The project will utilize four repositories that vary in size, domain, metadata creation method and workflow, and quality.  PERTAINS, a tool developed by one of the partner schools, will be used to analyze the metadata records in each repository and then to generate Dewey Decimal Classification-based tags.  Clustering algorithms will be used to generate an index of similarity and match between resources in different repositories.  After conducting a search, the user will retrieve a list of resources from the different collections that have been tagged in similar ways. Visualization techniques will be used to display the results in ways that enhance the research process.

I look forward to seeing these completed projects in about 2 years.   

Data, data, data - on journal prices!

I was forwarded info about the latest release of the Journal Cost-Effectiveness project, put together by  Ted Bergstrom and Preston McAfee.  They are researchers who have become quite heavily involved in the fight against the practices of overcharging by commercial publishers.  Because of their emphasis on the continued rising prices of subscriptions on top of very large profits, their data is divided by for- and non-profit entities.  Their methodology is well-described and transparent, and they provide the complete set of data for all years (2009-2011) for downloading.

The data set includes price (2010 prices) per article, price per citation (citations in 2009 papers to articles published 2004-2008), composite price index ("geometric mean of the Price Per Article and the Price Per Citation"), a relative price index (relative to mean of non-profit subscription price within same subject), and a ranked "value" based on the relative price index (<1.25 is "good", 1.25-2.00 is "medium", >=2.0 is "bad").

I used SPSS to see the distribution of the measures by profit status (for- or non-profit):

Doing a t-test of the differences in the mean showed that the differences were, of course, all statistically significant.  But that doesn't do justice to the extreme differences between these two groups.  Also important is the amount of variation within each group.  This time I looked only at mean price per article for each of the 3 "value" categories.  Here's an error chart:

This demonstrates not only the extreme differences between the price per article for non- and for-profit publishers, but also the differences in variation.

Now, same data, only by value and clustered by profit status:

Notice that the variation in the "good" value category is the least (very tiny bars), is somewhat larger in the "medium" category, but much larger in the "bad" category.

This is a look at just one aspect of one measure of this very nice data set.  I'm going to have fun looking at the other aspects of this and the other measures.  But what I'd really like to do is merge this with our own list of journals and see the value we're getting.  Of course, the prices are based on the published subscription prices for libraries (for tiered prices, they used that for a large academic library), and may not match ours.  I am disappointed that they did not include all of the data, including prices, number of articles, and number of citations.  Maybe they'll provide it if I ask...

The Fifth Law

I find Ranganathan's Fifth Law of Library Science the most intriguing: The Library is a Growing Organism.  While the First Law can be considered a "Trivial Truism", which he compares to Newton's First Law of Motion, the Fifth Law is an abstraction of natural laws.  Essentially, "an organism which ceases to grow will petrify and perish."  Without this understanding, librarians and library administrators deny their charges the necessary nutrition (funding) and nurturing (planning), and these libraries will stagnate (loose their impact) and die (closed due to lack of interest or support).

He describes both growth in size (of collections, of readers, of staff), as well as in form (evolution), although he devotes most of this chapter on the former.  Ranganathan had already spent much of the first half of the book detailing the evolution of the laws of library science from "Books are for Preservation" to "Books are for use", and from "Books are for the Chosen Few" to "Every Reader His Book".  And while he "cannot anticipate fully...what further stages of evolution are in store for this Growing Organism -- the library...", his Fifth Law ensures the understanding that libraries will continue to grow and evolve, or they will perish.

Regarding the growth in size, Ranganathan laments the "modesty" that library administrators underestimate the rate of growth of their charges. Even worse, he charges, is acting on this assumption of little change, and setting in motion "a faulty organisation obstructing the free development of a library...to its full stature."

He then associates library collection sizes with publication rates in the world.  He lists the book production for 1927.  I added 2011 numbers for comparison:

Country	1927	2011	Rate
Russia	36,680	123,336	336%
Germany	31,026	93,124	300%
Japan	19,967	78,555	393%
India	17,120	82,537	482%
Great Britain	13,810	206,000	1492%
France	11,922	67,278	564%
United States	10,153	288,355	2840%

Ranganathan then lists the annual rates of accession in some of the largest libraries, listed here with the latest rates I could find:

Library	1927	2011	Rate	Source
Library of Congress	202,111	454,212	225%	2010 Annual Report
Cambridge University Library	90,916	139,948	154%	2010 Annual Report
Birmingham Public Library	28,566	166,847	584%	Local Library Standards Audit 2010-2011
Imperial Library, Calcutta	7,832	62,773	801%	Calculated by comparing data from the Encyclopedia of Library Science1 and the National Library's Web site

1. Gaur, R. C., Jeyaraj, V., & Kumar, K. (2009). India: Libraries, archives and museums. Encyclopedia of library and information sciences, third edition (pp. 2291-2329) Taylor & Francis. doi:doi:10.1081/E-ELIS3-120044942


I wanted to include others listed, but getting the acquisitions data on libraries is not easy.


It is interesting, though, that it appears that Ranganathan's attempt to associate publication rates with library collection size has broken down over time. It is interesting that the LoC acquisitions was only two & a quarter times the number of over 80 years before, while the number of items published is 28 times. This indicates that libraries have ceased being the storehouses (even the LoC) of all information. That merely changes the relationship of the publication rates with the collection growth rates.


But Ranganathan moves on to describe how libraries should be able to accommodate the natural growth (which R. assumed would continue unabated). After addressing how to make the physical building and furniture flexible for this growth in size, he then addresses the need for the classification system to accommodate growth. Not unsurprisingly, he laments the Library of Congress Classification System for using a "primitive method of leaving gaps int he ordinary serial use of numbers," while promoting Dewey's system as "a demonstration of the immense potentiality of the decimal fraction." But most of all, he admonishes libraries to use standard classifications rather than tweaking systems or making their own.. He does mention the Colon Classification system he was then developing (at the time of the second edition), in that its mixed alphanumeric format and use of decimal formats are able to apply the three aspects of analytico-synthetic scheme: phase, facet and zone.


Next, Ranganathan turns his attention to the growth in number of readers that will inevitably occur as both the population increases and the Second Law brings about changes to that population (greater availability of reading material, increased literacy, "open access" shelves). I worry a bit, though, that with the decreasing emphasis on reading print books, libraries have been moving their books into long-term storage. While there have been improvements that decrease the amount of time it takes to retrieve a book, this policy effectively closes the stacks again, reversing the efforts made over a century ago. As electronic books can eventually replace print books, the print-to-electronic ratio may take a while to reach 1:1. Will the shift of collections reduce the availability of materials to such an extent that it has a detrimental effect on access to information?


Finally, he addresses not only the growth in size of the library, but also in form. Here, however, his foresight is limited and he focuses the progress that has already occurred. He discusses the gradual shift from keeping books chained like prisoners to "stock-taking" to limited "stock-use" to the "highly differentiated and complicated character of the organisation of the library to-day". He returns to the "growing organism" metaphor by celebrating the variety of libraries as "species" with their own "problems and peculiarities" but also with "common features".  


He ends this chapter on his final law detailing "the vital principle of the library" -

"it [the library] is an instrument of universal education, and assembles together and freely distributes all the tools of education and disseminates knowledge with their aid."

There would be no better way for S.R. Ranganathan to end his treatise on library science and librarianship.  

Proposition: Libraries are Obsolete (update)

Here is the video of the entire debate...I'm listening to it now, so I'm interested in seeing how it came out...


Harvard will host "an Oxford-style debate" on this proposition: Libraries are obsolete.  This proposition appears to be a continuation of the dialogue that was started by the Harvard Library Reorganization Plan, which caused an Occupy Harvard movement.  It is also interesting who will be debating in favor of the proposition: R. David Lankes, the author of The Atlas of New Librarianship, the controversial new textbook.   Lankes will be joined by the Headmaster of the Cushing Academy, a private prep-school focusing on leadership.  Speaking in opposition are James G. Palfrey, whose focus is on intellectual property and access to knowledge, and Susan Hildreth, whose blog posting, Libraries Succeed by Constantly Evolving, was re-posted on the Huffington Post.

It is hoped that the event will be recorded and video of it made available in the near future.  I hope so...it will be interesting to hear what issues will be raised by both sides.

Fighting the high cost of scholarly communication

There are two items that I have read recently that provide specific and targeted responses to the rising costs of developing library collections, specifically related to scholarly communication.  SPARC just put out a "Now what?" follow-up to the Elsevier Boycott initiated by scholars themselves.  The document, "You've signed the Boycott, now what?", provides some "Next Steps" for researchers to take that address the overall problem beyond just Elsevier.  Foremost among these steps is, of course, considering publication in Open Access journals (no surprise, given the source).  Listed second, interestingly, is a link to the libraries that serve the intended audience. Too often, these discussions become segregated with researchers working separately from librarians.  This is, after all, how the Boycott originated - not from the library community, but from the scholarly community.  Thus, advising those who have taken the initiative to sign the boycott and who have shown their concern for the problem to see for themselves the scope of that problem, can help unite these communities into a single voice.  While other steps do involve the domain of influence that the researchers have (as creators of knowledge, editors of journals, and referees of articles), the final step recommends getting involved in the "larger policy debate".  It is necessary for this community to see that the problem extends well beyond Elsevier.

The other item that has come to my attention is the Faculty Advisory Council Memorandum on Journal Pricing from Harvard University.  It is very forthright in its first statement, describing the situation as "untenable".  This coming from Harvard, one of the most well-funded higher educational institutions.  Giving such facts as the 145% increase in costs from two providers over six years and 35% profit margins as background, the Council clearly states that the "costs are prohibitive" to continue the same contracts with at least two major (and unnamed) providers.  Also poignantly stated is how these increases "would seriously erode collection efforts in many other areas, already compromised," (what's missing is how much the costs may have already eroded these efforts).

The Council memo then provides a list of specific "options" that both faculty and librarians could take in response to potentially changing the subscriptions, including many of the same recommendations mentioned in the SPARC document mentioned above.  First in the list for faculty is making their own papers accessible in the university's institutional repository.  Also in the list for researchers is to encourage the professional organizations in their own fields to "take control of scholarly literature in their field or shift the management of their e-journals to library-friendly organizations" (emphasis added).  Interestingly, of the 9 options listed, only three are addressed to librarians, none of which are possible without united efforts of both the scholarly and library communities.  Few publishers offer "pay-per-view" to libraries at a reasonable cost, or contracts that can be made public, or un-bundled packages.

Refusing these "untenable" conditions will bring pain to the users and the librarians, who must deal with declining collections and frustrated clients.  But like those who fought the untenable labor conditions of manufacturing, farming, and transportation, we are beginning to see that uniting our efforts against the unsustainable practices of those who control the the capital of scholarly communication will stabilize the delicate balance of powers and enable the progress of knowledge.

The Fourth Law

Save the Time of the Reader.  That is S.R. Ranganathan's Fourth Law of Library Science.  Unlike the first three laws, this one "makes its approach from the side of the readers."  While he initially states that the Fourth Law is demonstrated "from the moment he [the reader] enters the library," he later discusses the importance of saving the time of the reader to get to the library by appropriate placement of library buildings, branches, and mobile libraries.  A more modern approach would include online manifestations of library resources and services.

The author starts with a review of the "Closed System" of library stacks, which required readers to scour catalog books or cards, fill out slips with the required information precisely transcribed, submit those slips to the appropriate person, and wait...and wait...and wait...from an average of 30 minutes to upwards of hours.  Just to receive the books, or to be told that they weren't available or that the information was not correct.

Of course, Ranganathan was very supportive of the Open Access system.  Indeed, I think he would also be very supportive the 21st century version of Open Access - having books and articles freely available, at least online.  In addition to supporting the Fourth Law, he advocated Open Access shelving "on the grounds of national economy", given how much time is wasted by the researcher or assistant waiting.  He calculated 36,000 man-hours were wasted at his Madras University library, based on number of visits and average time waiting.

Another development that Ranganathan cheered as in support of the Fourth Law was the arrangement of non-fiction books on the shelf by subject.  Again, he supported his claims by providing calculations of time needed to locate books on a topic if the shelves were arranged by author.  He also advocated having the most heavily used subjects located closer to the front of the library for easier access (think the opposite of grocery stores, which put their most wanted items in the back forcing us to walk by things we wouldn't otherwise think about). My concern with this idea is that it is not predictable and thus could be consider not user-friendly.

Interestingly, when discussing charging systems and staff efficiencies, Ranganathan heavily advocates using the then-modern card-based systems rather the more common log-based systems.  This was new at that time, using cards to hold a set of information about a book or a reader and then filing the cards for easy retrieval.  The examples he provided clearly demonstrated the beginnings of databases, with each card a record which included a number of fields (boxes and lines).  It is no wonder that Ranganathan was an early advocate of computer automation of library-related tasks.

Like the other 3 Laws, the Fourth Law can be applied to all aspects of librarianship and library service in a variety of ways, including:

• Collection Development - Ranganathan focuses largely on improving the efficiency of procedures and tasks in order to get the books to the readers more quickly.  His suggestions were largely technical, with examples of cards for improving the tracking of book selection and acquisition.  I would add that proper book selection itself saves the time of the reader by having the right books available in the first place.  It also saves the time of the reader to sort through a more limited set of resources that have already been selected for their quality and usefulness.
• Cataloging & Technical Services - As mentioned in previous posts, Ranganathan strongly advocated for proper subject analysis in the catalog record.  This was particularly important for books that had interdisciplinary content or critiqued works from other authors.  He provided some examples that demonstrated how time-consuming it would be for a reader to find the right books by scanning them individually, rather than find the cross-reference cards for the desired topic or author.  
• Reference - The author notes that reference staff should acquire greater familiarity with the bibliography tools and the library's collection more rapidly than most users, given their regular use of this material.  However, there is a reluctance to hire a sufficient number of staff to meet the needs of the community.  This most apparent in universities, where "academic libraries seem to be lacking in their power to perceive the value of time and are hence halting in appreciating the need for such staff."  He mentions the two basic types of reference services provided in libraries - Ready Reference, which requires quick thinking and good memory, and Long-Range Reference, which requires specialization in a subject area, as well as ability to approach and interact with a range of reader experiences and knowledge.  It would be interesting re-examine this issue today - in this day and age of greater self-service and online Ready Reference sources, are library reference services properly staffed?
• Programming & Marketing - While not explicitly mentioned by Ranganathan, the programming and promotional efforts of libraries also demonstrate the value of the Fourth Law. He does advocate for proper placement of libraries, strongly supporting centralized locations for urban areas, and strategic placement of branches for larger areas.  He also mentions the growing value of mobile libraries, using motorized and animal-driven vehicles, when necessary.   
• User Training & Education - While not specifically mentioned by Ranganathan, I believe that user training and education can have the greatest effect on saving the reader's time.  By teaching the users to fish, these librarians save them their time of having to wait in line for assistance and explain their need (often multiple times before their need is correctly expressed and communicated).  This is especially true for students and researchers who would likely need to find similar resources many times.