This page provides more information on the Usefulness and limitations of bibliometrics (e.g. citation analysis) in the humanities.
- Possibilities and limitations of such databases as Google Scholar, Scopus and Web of Science
- Examples of Google Scholar analyses for publications in various domains
- Using the h-index
Possibilities and limitations of bibliometrics in the humanities
Traditional bibliometrics has a considerable number of limitations when applied in the humanities. That is mainly owing to the data sources used. They consist of data that can be traced to a pre-defined set of scientific/scholarly journals. This set consists mainly of English-language journals with an international scope (e.g. Web of Science (WoS) or Scopus). However, much of the research conducted in the humanities is published in non-English-language journals. While these journals may well be internationally oriented and have an international reputation, they are not included in the aforementioned data sources (Van Leeuwen, 2013). In addition, these data sources do not include books and book chapters. As the profiles of the research cultures demonstrate, these forms of scientific/scholarly communication are crucial for most domains in the humanities. Traditional bibliometrics may prove useful in humanities domains in which internationally oriented (English-language) journals are an important channel of communication (such as linguistics).
Google Scholar
Google Scholar is another way to collect bibliometric data. It is a search engine that searches through the files of most of the world's largest university libraries, large-scale repositories, the complete electronic versions of journals published by major publishing houses, and books indexed and made accessible by Google Books. This set of sources is much larger than that covered by WoS or Scopus, although its actual size is unclear. Google Scholar provides much more information than WoS or Scopus, including citations in books and edited volumes referencing other books and edited volumes. Google Scholar can therefore be used for bibliometric analysis in the social sciences and humanities (Prins et al., Research Evaluation, 2016).
Google Scholar does have some obvious limitations: it does not index all journals, for example because they are only held by libraries that are not available to this search engine. It is also unable to access some forms of citation, depending on the editorial rules maintained by certain journals and, in some instances, on citation practices and cultures in the various humanities domains.
The diagram below shows how certain domains differ in terms of the percentage of journals indexed in Google Scholar, and in the degree of fit between each one’s citation culture and the methods used by this search engine. It reveals, for example, that less than half of the journals selected by the Art History panels are properly indexed in Google Scholar and that most of the journals that are indexed have a citation culture that differs from the rest of the domain. That means that Google Scholar cannot be used for bibliometric analysis in the Art History domain. That is otherwise for Islam Studies and Cultural Studies, although journals in languages other than English are not as well indexed, making Google Scholar less useful in these domains.
Why the h-index does not work for research assessment, including in the humanities
Definition: A researcher has index h if h of his/her total output of N publications have been cited at least h times in other publications and the other (N-h) publications have been cited no more than h times.
The h-index is a fairly straightforward bibliometric index for quantifying a researcher’s publication impact. It was devised by physicist Jorge E. Hirsch. In the original h-index calculation, journal publications indexed by the Web of Science and Scopus play a major role. It is possible to correct this bias to some extent by using Google Scholar. However, doing so leaves you with three different scores, which is in turn problematic. In addition, the Google Scholar quantification is not without its issues, because it is not clear how this search engine collects citations. What is more important is that the h-index does not provide any calibration specific to the subject area (but that is also true of WoS and Scopus). The h-index also suffers from a few technical problems, for example when unpicking a researcher’s oeuvre (the problem of homonyms and synonyms), but it also rewards those who publish prolifically, seems to encourage ‘salami slicing’ (dividing up interrelated research outputs across multiple publications), rewards ‘one-indicator thinking’, and