A report titled "Citation Statistics" from the Joint Committee on Quantatitive Assessement of Research of the International Mathematical Union (IMU) was released on 6/11/08. Anyone using citation statistics needs to read this report and look for comments about it and consider other aspects of how citations are used/misued. Example:

Chronicle of Higher Education
Steven Harnad's Blog on Open Access
"Cite Check" Article from Inside Higher Education July 8, 2008
"Love at First Cite: A Serious Reader's Guide to References" Guest Editorial by Thomas Graves in
Compendium, April 2008, 194-200.

Measuring Productivity and Impact Using Citation-Based Metrics: Suggestions and Caveats

Cited Reference Searching in Web of Science is the oldest and best-known measure of productivity and impact of an individual or a group of researchers (2,3). Other measures, the h-index for example, new search engines, open access publishers such as BioMed Central which flag articles as Highly Accessed¹⁰ ...and the increased emphasis on quality control...have led to the emergence of an evaluation industry.⁸

Suggestions:

• If you need to measure productivity and impact for promotion/tenure or immigration purposes, please arrange a consultation with your department or college librarian. Citation practices differ between kinds of faculty (researcher v.s. clinician) and between fields of science. The librarian can assist with finding articles and books on evaluation in your discipline so that you choose the evaluation measure(s) appropriate to it. He/she can ensure that the search results are complete and accurate should you decide to use Cited Reference searching in Web of science.
• Faculty should check with their college to see if there is a policy covering the use of Citation-based metrics for promotion and tenure. If there is one, use it to interpret the results. If there is no policy, consider carefully the context of the citations of each paper^8,9 and the caveats below.

Caveats:

• See New report on citation statistics from the International Mathematical Union 6/12/08
• Not all journals are covered in the creation of the source records. Consult the journal lists for the Arts & Humanities Citation Index, the Science Citation Index (expanded), or the Social Sciences Citation Index at http://scientific.thomson.com/mjl/.
• A list of publications, with full authorship, should be used to search for cited references. First authors (last name plus initials) should be used for maximum retrieval of cited references in the ISI Web of Knowledge. Secondary authors are indexed only if the cited reference has been used as a source document in the citation database.
• Consider the following when searching for an author:
• Names can be misspelled.
• Names with hyphens and apostrophes may not be handled consistently.
• Inconsistent use of initials by authors/editors/journals
• Common names and initials may retrieve papers written by someone with the same name but in a different discipline.
• If a person is not the first author on a paper that is not covered by the citation index, then one must search for that paper by the first author.
• Errors may occur in the year, volume and/or page numbers of a citation.
• Self-citation may account for some of the cited references.
• Not all cited references are positive. The initial paper on "cold fusion" has a large number of citations from papers covering the fact that the results could not be reproduced.
• Book reviews, letters, meeting abstracts, etc. are also source records.
• Review articles may be cited more than those that offer new concepts or ideas.
• Cited papers may be cited for reasons having more to do with the sociology of science rather than the importance of the paper - e.g. a former graduate student may cite his/her advisor; the more widely known are cited more than the less widely known for political reasons or due to grantsmanship.⁹
• See reference 8 below for additional caveats.

For more information, consult:

1. Courtois, Martin. "Tips for searching the ISI citation indexes for personnel decisions." Database, 16(3): 60-67. 1993.
2. Garfield, Eugene. "How to use citation analysis for faculty evaluations, and when is it relevant? Part 1." Essays of an Information Scientist. http://www.garfield.library.upenn.edu/essays/v6p354y1983.pdf
3. Garfield, Eugene. "How to use citation analysis for faculty evaluations, and when is it relevant? Part 2." Essays of an Information Scientist. http://www.garfield.library.upenn.edu/essays/v6p363y1983.pdf
4. Lawrence, Steve. "Free online availability substantially increases a paper's impact," Nature 411, 521 (2001).
5. Meho, Lokman I. "The Rise and Rise of Citation Analysis." Physics World 20, no. 1 (January 2007): 32-36.
6. Saha S, Saint S, and Christakis DA. "Impact factor: a valid measure of journal quality?" Journal of the Medical Library Association 91, no.1 (January 2003): 42-46.
7. Hirsch Jorge E. "An index to quantify an individual's scientific research output," Proceedings of the National Academy of Sciences of the United States of AmericaVolume: 102 Issue: 46 Pages: 16569-16572 Published: NOV 15 2005
8. Leydesdorff "Caveats for the use of citation indicators in research and journal evaluations" Journal of the American Society for Information Science and Technology [1532-2882] yr:2008 vol:59 iss:2 pg:278 -287
9. Wendl MC "H-index: however ranked, citations need context," Nature Vol 449 27 Sept. 2007 page 403.
10. Most viewed articles on BioMed Central Web page accessed 2/14/08 http://www.biomedcentral.com/info/about/mostviewed.
11. Vucovich, LA; Baker, JB; Smith, JT. "Analyzing the impact of an author's publications," Journal of the Medical Library Association 96 (1): 63-66. Jan. 2008.

Gayle Baker
University of Tennessee, Knoxville Libraries
December 2000

Updated June 2008 by Ann Viera, Veterinary Medicine Reference Librarian and Josh Hogan, Graduate Assistant
Pendergrass AG-VET MED Library