University rankings and publication figures — Is there something to compare from the library’s point of view?

What really is in the interest of the stakeholders and library directors is to find a way to compare the library’s impact on the success of the university, and if and how the investments of the library — acquisitions and services, staff and collections, equipment and space — affect the success of the university for instance in the different international higher education rankings.

The background information shows some ranking from the years 2011 and 2012. Here are the CWTS Leiden ranking 2015 showing each of our university’s size-independent ranking in biomedical and health sciences in comparison to the other universities in the same country — not in comparison to each other as we have realized that the funding systems and the organizations make direct comparisons difficult and hard to explain.

See CWTS Leiden Ranking 2015 for the explanations of the impact and collaboration factors. The first two pictures show collaboration and impact figures of Belgian universities in biomedical and health sciences, the next two the same figures of Finnish universities, and the last two the same of Norwegian universities.

Did our libraries have any impact on these? We would really like to find collaborators with statistical analysis skills to e.g. dig deep into our biomedical and health sciences electronic collections numbers of usage, to help in finding our if investing in them is worth the money in each university in relation to the figures that the different ranking lists provide.

SciVal is a set of integrated modules that enables institutions to make evidence-based strategic decisions. SciVal consists of three modules: Overview, benchmarking and collaboration. Does using them tell us anything about the libraries’ impact? Or, when we compare the figures, do we see something the library can do better?

The overall pictures of our universities look like this:







We can compare the scholarly output:

The can compare the scholarly output also in the top 10 publications category:

The figures are interesting but how much impact the library and its services has on these figures? How could we know that?

Our next step will be to explore qualitative indicators regarding the libraries’ impact on publications and research.


Comparing statistical information

In the beginning of the project we collected and shared plenty of statistical information about our libraries and universities. The plan was to compare the activities and results. Areas were

  • Library areas, facilities and equipment
  • Services for the public, including loan, ILL and user training
  • Collection management, bibliographic records
  • Institutional repository
  • Library staff, both number and staff training
  • Financial data

Statistical data will give more value when comparing with others or with oneself, over time, but which statistics can be compared? One example is loans. Number of loans is easy to compare, and the numbers can be extracted from the library systems. For 2013 we have the following statistics for loans, visits to the libraries and size of collections. But we miss important data, e.g. on downloads of articles, and use of e-books.


*) Include renewals, for NTNU is for example the number for first time loan is about 50%.

**) Apply to the whole institutions, not just the medical libraries. Medicine has focus on articles more than books, so these numbers are not valid for medicine.

How to compare?

We can observe that NTNU and UCL have quite similar number of loans, and almost twice as much as UEF, at the same time  NTNU and UCL have almost twice as many visits as UEF. And the inter library loan at NTNU is three times as high as at UCL. Can we see any correlation at all? It is also easy to compare interlibrary loans. Use of collections in medical libraries tend to have a predominance on articles, at the same time prices of electronic journals are increasing more then price of books. This mean that no library are able to have all the journals needed in their collection. So ILL can say something about the quality of the library collection – and also about the size of the media budget. An example: the NTNU Library (BMH) use about 2.5 % of the media budget on buying copies, a very small amount when more than 90% of the budget is used on electronic resources and journals.

Many element affect the statistics and the use of library services. Number of loans must be seen in relation to the size of the universities. The NTNU part of BMH serves 3000 students and 11000 staff members at NTNU and St. Olavs Hospital. UCL serves about 6000 medical students and 760 academic and research personnel, and the UEF Library serves a total of 3000 university staff members and 15 000 students (about 1400 medical and dental students). Other elements affecting statistics are size of the collection, how updated is the collection, acquisitions per year, number of users, the amount of e-books and e-journals, the number of printed books replaced by e-books and so on.

Should we be able to compare, we must use indicators; this will be discussed  in a separate blog post. Examples of other useful indicators could be

  • Number of e-journals / download of articles
  • Number of e-books / number of pages read
  • Loans from library collection / ILL
  • Loans from library collection / number of students

As an example we can calculate the relation between visits to the library and loans. NTNU have 0.42 loans per visit, while at UCL and UEF are respectively 0.17 and 0.1 loans per visit. If this is done for several years, one can get a picture of how user activity in the library develops over a period of time. And then it also gives more meaning to comparing libraries.

Follow the money

When we started collecting data, we did not know how to compare. We have lots of data, but not necessarily the most interesting or useful data. It became clear that though we all three are medical/health sciences libraries that serve both faculty and university hospital and also other users, we are neither organized nor financed in the same way. Due to these differences  it is difficult to compare economic data. Yet, it would be useful, and the library directors are keen to compare both financial and other data. While visiting the three libraries, we had discussions with the  library directors and got suggestions for further work on statistics and data. In Trondheim we were encouraged to measure the impact of the library, and to look at the connections between quantitative and qualitative indicators. In Louvain-la-Neuve we talked about library statistics and economics, and that it is important to seek out at least some comparable indicators nationally and internationally. In Kuopio we discussed statistics and other data as useful background information.

Next step for our project is to find indicators for library performance. More on this topic in another blog post.

Information skills training

Facts and figures

Information skills, or information literacy (IL), training in the three libraries can be compared to some extend. As our universities, faculties and hospitals are not the same size — the numbers of students, researchers and clinical staff are not comparable — and as the organization of the training is varied, it is not useful to compare mere numbers. The most useful numerical comparison is probably hours per library staff members. These statistics are from year 2013:

  • NTNU’s BMH library gives about 220 hours of user training per year for about 1200 participants. This 22 hours and 120 participants per staff member though all staff members do not teach.
  • UCL’s BSS library gives about 75 hours of user training per year for about 2770 participants. This is about 8,5 hours and about 86 participants per staff member though all staff members do not teach.
  • UEF’s KUH library gives about 43 hours of user training per year for about 530 participants. This is about 8,5 hours and about 106 participants per staff member though all staff members do not teach.
    • The KUH numbers do not include the working hours spent in Moodle online learning environment where most of the teaching is conducted; two staff members of KUH medical library are responsible for the online courses for about 350 undergraduate medical, dental and other health sciences students per year.

These numbers tell only a part of the story. By taking a look at the teaching methods — lectures, hand-on sessions, assignments etc. — and contents of the courses, we get some qualitative data, too. We notice that comparing numbers provides some answers but also arises new questions, and interesting paths towards a deeper understanding of each others’ activities.

The teaching methods

Lectures are held in all our organizations (in Brussels, Kuopio, Trondheim). Hands-on sessions are organised in Brussels, Kuopio and Trondheim. There are no assignments in Brussels; in Kuopio usually, and in Trondheim sometimes, but not at all the courses. Exams are rare: in Brussels there can be exams — depending on the professor who asked for the course — but in Kuopio and Trondheim there are no exams.

In Brussels online courses are not given yet; in Kuopio almost all courses in Finnish for undergraduate students are online in Moodle, but courses for international students in English and courses for doctoral students / postgraduates are blended, and some of them using the flipped-classroom model; in Trondheim NTNU Library has developed VIKO; an online  guide to finding information and academic writing and BMH is preparing an online course for Zotero, that they also are planning to use as part of a flipped classroom course.

Also 1-to-1 support is offered in all the three libraries.


Classroom in Trondheim

The contents

The list of databases covered includes but is not limited to:

  • Academic Search Premier: Brussels
  • Best Practice: Trondheim
  • Cinahl: Kuopio, [edit:] Trondheim
  • Cochrane Library: Kuopio, Trondheim
  • Embase: Trondheim
  • Google Scholar: Trondheim
  • Medline/PubMed: all three
  • PsycAtricles: Brussels
  • PsycInfo: Brussels, Kuopio, [edit:] Trondheim
  • Science Direct: Brussels
  • Scopus: all three
  • UpToDate: Kuopio (only hospital), Trondheim
  • Web of Science: Kuopio, Trondheim

In addition, Trondheim teaches also other resources from the Norwegian Health library, and Kuopio usually includes Medic, a Finnish database of health sciences.

The reference management tools that are taught are RefWorks in Kuopio and EndNote in Trondheim, planning to teach also Zotero in the future.

Curruculum or not curriculum

In Brussels some lectures are included in the curriculum, depending on the faculty, the course and the professor who asked for it. In Kuopio all the courses for university students are integrated (and credit points given by the library) in the curricula, and there are also continuing education courses for the university hospital staff where information skills training is part of the course. There are also open training sessions that are not part of any course — often offered online (streaming) in Adobe Connect or Lync/Skype. In Trondheim some of the courses are integrated in the curricula of the medical faculty. Others are offered by the library to both students and staff from the university and the hospital.

Why is it important to teach information skills?

An interesting question, naturally, is also what are the aims of our training and teaching practices. More use of databases and journals? Information literate physicians and other health care professionals? Research with more impact and influence? More “student points” in order to get (more) funding? Better impact factors or other metrics that could make it possible for the university or hospital to get more funding? Better health care?

The most important point for us is information literacy that is the basis for all the aims mentioned above. We also would like to stress better intellectual autonomy of students with the abilities of critical thinking and source criticism. But how to measure that?