Why clinical citations deserve their own metric

This is a guest post contributed by 
Christina Lohr, Product Manager Research Metrics at Elsevier.

In 2014, an article was published on the “Immunization for Streptococcus pneumoniae infections in high-risk children”. Despite its practical relevance in treating children with immunodeficiency, to this date, it has only been cited four times by other articles. So did this output make any difference in the context of clinical practice? A closer look at the tools that clinicians rely upon to make decisions reveals an answer: Three discussions in a point-of-care reference tool cite that particular article, indicating how it is being used in clinical practice.

Health outcomes in themselves do not necessarily produce journal citations. In times where the measurement of research impact is essential to help direct the allocation of funding, this can pose a problem for researchers that work in translational medicine, a field that is difficult to evaluate with traditional measures of impact such as citation counts. Citation based indicators have been shown to undervalue more applied medical research in comparison to basic research. [1],[2]

And while citation counts are still valuable measures to reflect how other scientific authors assess a piece of research, one of the advantages of altmetrics is that they offer access to the opinions of the interested general public: undergraduates, professionals, government, journalists, and many more. This is especially interesting in the context of clinical research.

It is estimated that non-publishing – and hence non-citing or cited – users constitute about one third of the research community.[3] This third includes large numbers of undergraduate and graduate students as well as researchers operating in the corporate sector. Their interests, however, are effectively hidden from citation data. In addition, research publications are also used by professionals outside academia, and the context in which they are used can be plentiful. Many practitioners do not author publications, but the majority read journal articles as part of their daily activities. Journal articles were also reported to be the most useful publication type for residents in a hospital to help them answer questions directly related to patient care. [4],[5]

Sources of clinical impact

Unlike areas such as biochemistry or immunology, which are well represented by article citations, additional measures are needed to track the impact of medical research. When considering new sources to track the use of clinical research, a few are of particular interest:

Clinical Guidelines

e.g.: National Institute for Health and Care Excellence (NICE), PubMed Clinical Practice Guidelines

Citations in clinical guidelines indicate to what extent research informs health practice and are thus a measure of influence on policy making. Clinical guidelines intend to collect the best and most current evidence to guide decisions regarding the prevention, diagnosis, management and treatment of a clinical problem. However, it is worthwhile noting that for some guidelines these kind of citations are unlikely to reflect recent research, as it can take decades for medical research to translate into health policies. Hence, clinical guidelines are best at reflecting mid- to long-term medical impact. An interesting aspect to look at in this context is the level of policy impact (international, national, or local), and the organizations that were influenced.

Clinical Trials

e.g.: ClinicalTrials.gov, International Standard Randomised Controlled Trial Number Register (ISRCTN)

Clinical trials investigate the safety and efficacy of new treatments, such as novel drugs, devices or medical procedures. Citations in clinical trials are an interesting complement to citations in clinical guidelines, because they are substantially quicker and more likely to cite key innovations. And even though only few trials lead to new treatments, the mere existence of a trial demonstrates the potential practical value of an idea.[6] Registries such as clinicaltrials.gov provide large collections of clinical studies.

Clinical Alerts and Advisories

e.g.: NIH Clinical Alerts and Advisories

Clinical alerts are notices that are used to speed up the release of findings that could significantly impact a patient’s morbidity or mortality. These alerts do for instance announce when a particular trial has been stopped or a treatment has been discontinued. They are a valuable measure of the immediate impact of clinical work on health and health systems.

Clinical Decision Support Systems

e.g.: UpToDate, DynaMed Plus

Practitioners worldwide use tools that improve patient care by offering evidence-based clinical decision support at point-of-care. These tools aim at delivering the most current information about a topic, covering diseases, conditions, drugs, and more. The content is continuously updated based on recent findings, for instance from medical journals.

Systematic Reviews

e.g.: Cochrane

While not unique to the clinical world, in the practice of evidence-based medicine, systematic reviews of randomized controlled trials are key. They aim to provide complete and exhaustive lists of current literature relevant to a specific research question and are seen as the strongest form of medical evidence. To review existing studies is often quicker and cheaper than embarking on a new study. It should be noted though that in rapidly changing medical fields, systematic reviews might already need updating at the time of publication. Depending on the field, these studies are a good measure of short- to medium-term impact.

Over the past few years, real progress has been made to uncover the impact of clinical research with valuable reporting for the short- and medium-term impact, with the goal to find the holy grail – measuring the long-term impact of healthcare research: the allocation of healthcare resources, quality of care and service delivery, and education in healthcare.[7] Clinical citations are one step in that direction.

How clinical citations help researchers

Clinical and translational medicine plays a critical role in promoting a flow of information between basic and clinical scientists. Researchers in this area of work optimize new technologies, improve clinical application of treatments, and ultimately improve the quality of life for patients. By giving researchers the tools to showcase their talents, we can help them in telling the stories of the impact of their research with the potential benefits:

  • Researchers and institutions alike can benchmark themselves against others in terms of clinical impact.
  • With more ways to get recognition for their contributions, researchers may keep working in fields that they would otherwise opt out.
  • Hospitals have an alternative way of showcasing their talents.
  • Attention can be drawn to fields that might otherwise be overlooked.

 How clinical citations help funders

In times of financial constraints, health researchers must demonstrate societal impact and value for money to their funding agencies. More and more organizations recognize that clinical citations are a step in the right direction to demonstrate precisely that. By gaining access to the work cited in clinical guidelines, new opportunities open up for researchers and funders alike to better understand how research can move out of the laboratory to the bedside. Ensuring accurate and complete acknowledgement information in published work is key in tracking research outputs. Much can be done to enhance the quality and completeness of funding data, and we are just beginning to uncover the full potential of these data.

Andrea Michalek and Polly Allen will both be at the 4:AM Conference and would be happy to talk more with you about this topic.


[1] Seglen, P. O. Why the impact factor of journals should not be used for evaluating research. BMJ (Clinical research ed.) 314, 498–502 (1997).

[2] van Eck, N. J., Waltman, L., van Raan, A. F. J., Klautz, R. J. M. & Peul, W. C. Citation Analysis May Severely Underestimate the Impact of Clinical Research as Compared to Basic Research. PLoS ONE 8, (2013).

[3] Cronin, B. & Sugimoto, C. R. Beyond Bibliometrics: Harnessing Multidimensional Indicators of Scholarly Intent. MIT Press, Chapter 17 (2014).

[4] Schilling, L. M., Steiner, J. F., Lundahl, K. & Anderson, R. J. Residents’ patient-specific clinical questions: opportunities for evidence-based learning. Academic medicine : journal of the Association of American Medical Colleges 80, 51–6 (2005).

[5] Mohammadi, E., Thelwall, M., Haustein, S. & Larivière, V. Who reads research articles? An altmetrics analysis of Mendeley user categories. Journal of the Association for Information Science and Technology 66, 1832–1846 (2015).

[6] Thelwall, M. & Kousha, K. Are citations from clinical trials evidence of higher impact research? An analysis of ClinicalTrials.gov. Scientometrics 109, 1341–1351 (2016).

[7] Rivera, S. C., Kyte, D. G., Aiyegbusi, O. L., Keeley, T. J. & Calvert, M. J. Assessing the impact of healthcare research: A systematic review of methodological frameworks. PLoS Medicine 14(8), e1002370 (2017).