2:AM Altmetrics as indicators of economic and social impact

15:25 – 16:40: Altmetrics as indicators of economic and social impact

Chair: Euan Adie, Founder of Altmetric.com

Anup Kumar Das, Jawaharlal Nehru University

Altmetrics and the Changing Societal Needs of Research Communications at R&D Centres in an Emerging Country: A Case Study of India

Altmetrics necessitates changing research communications strategies in research departments in universities and R&D centres in emerging countries. However, there is a lack of specialization or professionalization in managing and formulating research communications strategies in these research centres. This makes these R&D centres less compatible to global outreach. Research papers emanated from these R&D centres and departments don’t get adequate international visibility or attention or recognition due to lack of planning and strategic approaches by the policymakers and institutional decision makers. Research communications should also embrace the specialization available in the research department. For example, the services of documentation officers, research officers and information scientists available at the research departments in Indian universities can be adequately utilized for strategic planning and increasing institutional outreach to global scholarly audiences. They can be trained and made custodian of the research communications channels freely available online such as academic social networks, institutional repositories, open data repositories, and electronic discussion forums. There is also need of creating a rank of specialized personnel for managing institutional research communications channels. However, in many developing countries, including in India, the R&D centres and research departments are in public sector and they need to take proactive role. Thus, every R&D centre and research department should think of engaging the information specialists or re-designating their research librarians for managing their research communications channels. This paper will highlight a model institutional research communications strategy to increase global visibility of the institutional research. This paper will also cite a case where strategic use of research communications channels in an Indian institution bears the fruits of higher altmetric scores and global visibility of institutional researchers.

Juan Pablo Alperin, Simon Fraser University

Evolving altmetrics to capture impact outside the academy

There are an infinite number of ways in which a research article can serve society without ever being mentioned on the Web. The most often cited example is that of a patient walking into a doctor’s office with information about a new medical treatment in the hopes of better managing their illness. There are, however, more subtle ways in which the access to research can have an impact on society. A similar patient may read about a treatment and simply be given hope about their prognosis. Other examples might include an NGO tweaking their programs to better serve their community as a result of reading a study, or a government agency taking up a policy recommendation from a paper. Within academia, there are also societal impacts that cannot be measured by citations. A paper used for didactic purposes in a classroom, even if never cited in a student essay, contributes to the human capital development, and as a result, to the strengthening of the higher education system as a whole. These are but a few uses that will never be captured by citations, but that could potentially be glean from altmetrics (or, more likely, from a future iteration of what we now know as altmetrics).

Looking at the number of times scholarly articles are mentioned on the Web will never be an adequate way of understanding these indirect and subtle forms of the societal impact of research. While we have not yet reached the limits of what altmetrics can offer us in their current form, we can be sure that altmetrics today will do not yield much insight into the societal impact of research. To explore the societal impact of research through altmetrics, they must evolve to better capture: 1) who are the people that generate events that lead to metrics; and 2) what are the contexts and rationales that trigger those events. It is only by exploring these questions that practitioners and researchers will be able to guide the development of altmetrics to capture the type of non-academic impacts that many hope they will.

In a small-scale study attempting to answer the first question, I found that Twitter users who shared articles from the Brazilian collection of the Scientific Electronic Library Online (SciELO) were made up of a high percentage of non-academics, including professionals, homemakers, patient groups, independent researchers, and journalists. The extent to which these users are present among those who share research from other regions, languages, and disciplines is yet to be studied, but if it is similarly widespread, then Twitter may prove to be a suitable place to search for impact of research outside of the academy. It is therefore necessary to evolve altmetrics to better identify these groups and the ways in which they use the research.

Lauren Ashby (SAGE) & Mathias Astell (Nature Publishing Group)

The Empty Chair at the Metrics Table: Discussing the absence of educational impact metrics, and a framework for their creation

The recent drive to establish more nuanced metrics to account for the different forms of a scholarly output’s impact – so called altmetrics – has largely neglected one key element of the scholarly research cycle: educational impact. Recent shifts in the assessment of UK university outputs, including a mooted Teaching Excellence Framework to accompany the next REF cycle, suggest that the work achieved by scholars in education (i.e., the impact on student learning) is becoming a more central part of performance assessment. Such a shift may go some way towards reducing the current unhelpful bifurcation of academia into research and education – however, without a set of metrics by which to complement any such assessments, education and teaching will ultimately remain on an unequal footing with research. It is telling in this respect that the recent 163 page report on the use of metrics in the scholarly world, The Metric Tide, only made 5 mentions of educational impact.

To begin to unpack how best scholarly outputs can be assessed for educational impact, the following presentation will consider the ways in which scholarly outputs – for the purposes of this presentation defined as any DOI trackable output in any academic field – play a role within the areas of teaching excellence and overall learning at Higher Education Institutes (HEIs). The presentation will then move on to explore the different educational impact pathways of scholarly outputs within these areas, along with the sources of data (such as online syllabi, online university and college course reading lists, graduate dissertations and theses, educational presentations, digital journal clubs, reference management system data, university and public library holdings and usage data, as well as digital library holdings and usage data) and analysis techniques (similar to those utilised in the assessment of other alternative metrics, e.g., frequency of appearance, usage, citations, etc.) that could be utilised to create the basis for a robust set of educational impact metrics.

In exploring the use of scholarly outputs in HEIs and the educational impact pathways these outputs follow, along with an assessment of the data available through which to track these elements, this presentation will lay down the basic tenets of a new paradigm in academic metrics: one tracking educational impact of scholarly outputs.

Prof Theng Yin Leng, Nanyang Technological University, Singapore

Altmetrics: Rethinking and Exploring New Ways of Measuring Research Outputs

Traditional metrics for scholarly and scientific publishing such as citation counts and impact factors have been criticised for being too shallow and too narrow. In order to go beyond limited comparisons of like-for-like and to become generally useful, we need better metrics, techniques and tools to measure research outputs more accurately.

Worldwide, universities, research institutes and policy makers realise that there is a need to have metrics that allow equitable comparisons, and can be used to benchmark research outputs, and efforts are needed to develop more robust metrics for general use. Today, with access to large volumes of data from diverse sources, it is possible to define alternative research impact metrics that capture domain-specific insights.

Therefore, the overall aim of this project is to investigate new approaches offered by Interactive Digital Technology and Social Media to rethink and explore ways to measure research outputs. Specifically, this project focuses on scholarly and scientific publishing in which in recent years, altmetrics have appeared as new and alternative metrics. We intend to concentrate on exploring and rethinking ways of measuring research outputs of a scholarly and scientific nature of individuals and aggregated individuals belonging to a university, institute or centre. We aim to design, develop and evaluate a system prototype called “Altmetrics for Research Impact Actuation” (ARIA), to gather and compute altmetrics to measure research impact.

The specific objectives of the project are:

a) Conduct a literature survey of traditional metrics and altmetrics to measure research outputs.
b) Develop a framework and a set of hypotheses for cross-metric validation.
c) Develop new metrics on research impact that can be harvested from social media platforms (e.g. Twitter, Weibo, etc.).
d) Develop new research impact metrics for the different disciplines, broadly divided into: (i) “hard sciences” research which include sciences, computing and engineering; (ii) “non-hard sciences” research humanities, arts, business and social sciences.
e) Develop new metrics to measure research innovation and commercialisation.
f) Design, develop and evaluate the ARIA prototype based on inputs from (a) to (e).

This project started in February 2015 and will progress over three years. We plan to evaluate our approach with researchers from the Nanyang Technological University (NTU) in Singapore, cooperating with the NTU library and the NTU research office. This research will help funding agencies, and educational and research institutions, to measure and benchmark research outputs. We believe that findings from this project will be useful in our long-term objectives of computing better measurements of research outputs, hence raising standards.

Rodrigo Costas, (CWTS-Leiden University, the Netherlands) & Stefanie Haustein (Université de Montréal, Canada)

Citation theories and their application to altmetrics

Researchers are increasingly pressured to provide evidence that their work has impact—within the scientific community, on the economy or on society at large. Altmetrics have been suggested as indicators of various forms of impact aiming to replace citations as the sole measure of academic success. The acts on which various altmetrics are based are, however, quite heterogeneous: likes on Facebook, mentions on Twitter, saves on Mendeley, and expert recommendations on F1000 are acts that differ in terms of user community, engagement, motivation and audience. While the act of citing has been an essential part of the scholarly communication process since the beginning of modern science, it is unclear whether the acts that altmetrics are based on are relevant in scholarly communication. While empirical studies have shown that most altmetrics correlate weakly with citations—suggesting fundamental differences between these various metrics—conceptual discussions about their meaning are rare, leaving it unclear what is actually measured. We argue that the heterogeneity of user communities, levels of engagement, motivations, as well as of the audiences to which the various metrics are associated affects their respective meaning.

In order to go beyond correlation analyses and better understand the meaning of altmetrics, we discuss acts on social media from a theoretical and conceptual perspective using citation theories, chosen because of the analogy that is often made between altmetrics and citations. By discussing various acts on social media from the perspective of the normative, social constructivist, and concept symbols citation theories (Haustein, Bowman, & Costas, 2015), we show that they are more or less suitable, and that the level to which they can apply depends on the particular act considered. For example, Merton’s ethos of science can help to explain the mechanisms behind reviewing and recommending on F1000 and (to some extent) blog citations and Mendeley readership. Users on Twitter and Facebook are less likely to adhere to these norms, as can be reflected by the important tweeting intensity associated with papers that have funny titles, focus on curious topics or to retracted publications. Social constructivist theories, and in particular the Matthew effect, seem applicable to most acts on social media due to their networked nature; explaining the concentration of events on a few prominent papers or actors on social media. The concept symbols theory was helpful to explain mentions on Twitter, for example with respect to hashtags.

Theoretical discussions will be supported by concrete practical examples to begin debates surrounding the development of a framework that accommodates the broad and diverse spectrum of acts recorded by different altmetric sources.

Leave a Reply