Recent trends in the policy of research and its development include, among others:
The explosion of the assessments in the “ The STAR METRICS is a data platform that is voluntarily and collaboratively developed by US federal science agencies and research institutions to describe investments in science and their results (Largent & Lane, 2012). The first objective of the European Commission (2014) Expert Group, in which the author of the present paper took part, was indeed to “propose an analytical framework for identifying how the implementation of different ERA priorities and components observed at institutional level (i.e. research performing organizations) and national level (i.e. national policies and funding organizations policies) impact the research system performance (at institutional and national level).” The The The recent debate on
The
Is science really becoming increasingly data-driven? Are we moving toward a data-driven science (Kitchin, 2014), supporting “the end of theory” (Anderson, 2008), or will theory-driven scientific discoveries remain unavoidable (Frické, 2015)? There is little agreement in the literature. More balanced views emerging from a critical analysis of the current literature are also available (Debackere, 2016; Ekbia et al., 2015), leading the information systems community to further deeply analyze the critical challenges posed by the big data development (Agarwal & Dhar, 2014).
Data sources indeed “are not simply addenda or second-order artifacts; rather, they are the heart of much of the narrative literature, the protean stuff that allows for inference, interpretation, theory building, innovation, and invention” (Cronin, 2013, p. 435). Making data widely available is very important for scientific research as it relates to the responsibilities of the research community toward transparency, standardization, and data archiving. However, to make data available, researchers have to face the huge amount, complexity, and variety of the data that is being produced (Hanson, Sugden, & Alberts, 2011). Moreover, the availability of data is not homogeneous for all disciplines and the cases of “little data” and “no data” are not exceptions (Borgman, 2015).
These recent trends and the issues they underline require a new framework for the analysis. The theoretical framework (intended as a group of related ideas) that we propose in this paper is designed to be a reference for the development of models for the assessment of the research activities and their impacts. A framework is required to develop models of metrics. Models of metrics are necessary to assess the meaning, validity, and robustness of metrics.
We claim that our framework can support the development of the appropriate metrics for a given research assessment problem or for the understanding of existing metrics. This is a very difficult question because, among other things, it refers to a complex phenomenon for which there is the lack of a reference or a benchmark to compare the metrics against. The purpose of our proposed framework is exactly to offer a
Often, indicators and metrics are used as synonyms (see also Wilsdon et al. (2015)). In this paper, indicators are combinations of data that produce
It is important to develop models for different reasons, including:
More specifically, a model is an abstract representation, which from some points of view and for some ends represents an object or real phenomenon Some interesting readings on modeling can be found in Morris (1967), Pollock (1976), Willemain (1994), Myung (2000), and Zucchini (2000).
For quantitative models the analogy with the real world takes place in two steps:
The practical use of a model depends on the different roles that the model can have and from the different steps of the decisional process in which the model can be used. A model can be considered a tool for understanding the reality. The potentiality of models can be expressed for description, interpretation, forecasting, and intervention. These different roles may be correlated or not, depending on the objective of the analysis and the way the model is built. To be successful the modeling has to take into account the specificities of the processes and systems under investigation, and in particular consider that the behavior is free and finalized to given aims; history and evolution matter as the behavior of systems and processes changes over time (see e.g. Georgescu-Roegen (1971)).
Hence, the modeling activity related to the assessment of research involves several methodological challenges. What is required today is to develop models, able to characterize strongly connected or
Evaluation In this paper evaluation and assessment are used as synonyms.
The
Limits in the
There are some
Possibility that the The The intrinsic complexity of calculation of the objective of the analysis.
The ambition of our framework is to be a general basis able to frame the main dimensions (features) relevant to developing multidimensional and multilevel models for the evaluation of research and its impacts Vinkler (2010) presents a systematic view of units and levels of analysis in research assessment
We propose a framework, illustrated in Figure 1, based on three dimensions:
An illustration of our framework including its three implementation factors (tailorability, transparency, and openness) and its three enabling conditions: convergence, mixed methods, and knowledge infrastructures.Figure 1
We detail each dimension in three main building blocks and identify three operational factors for implementation purposes. The main building blocks of
Definitions of education, research, and innovation.Term Definition Education In general, education is the process of facilitating the acquisition or assignment of special knowledge or skills, values, beliefs, and habits. The methods applied are varied and may include storytelling, discussion, teaching, training, and direct research. It is often done under the guidance of teachers, but students can also learn by themselves. It can take place in formal or informal settings and can embrace every experience that has a formative effect. Education is commonly organized into stages: preschool, primary school, secondary school, and after that higher education level. See the International Standard Classification of Education (ISCED, 2011) for a more technical presentation. Research According to the OECD’s Frascati Manual (2002), research and development (R&D) is the “creative work undertaken on a systematic basis in order to increase the stock of knowledge, including knowledge of man, culture, and society, and the use of this stock of knowledge to devise new applications.” The term R&D covers three activities: “basic research, applied research and experimental development. Basic research is experimental or theoretical work undertaken primarily to acquire new knowledge of the underlying foundation of phenomena and observable facts, without any particular application or use in view. Applied research is also original investigation undertaken in order to acquire new knowledge. It is, however, directed primarily toward a specific practical aim or objective. Experimental development is systematic work, drawing on existing knowledge gained from research and/or practical experience, which is directed to producing new materials, products, or devices, to installing new processes, systems, and services, or to improving substantially those already produced or installed. R&D covers both formal R&D in R&D units and informal or occasional R&D in other units.” See also the more recent Frascati Manual (OECD, 2015b). Innovation According to the OECD (2005), an innovation is “the implementation of a new or significantly improved product (good or service), or process, a new marketing method, or a new organizational method in business practices, workplace organization or external relations. The minimum requirement for an innovation is that the product, process, marketing method or organizational method must be new (or significantly improved) to the firm. Innovation activities are all scientific, technological, organizational, financial and commercial steps which actually, or are intended to, lead to the implementation of innovations. Innovation activities also include R&D that is not directly related to the development of a specific innovation.”
The main building blocks of
The problem of evaluation of the research activities, in our set-up, is framed in a
The three main
The more we are able to go to the deep, fine-grain of the most atomic level-unit of analysis, the higher the level of tailorability, the higher the level of transparency and openness may be, and the better will be the conceptualization and formalization of quality within a model.
In this paper, we assert that the ability of developing (and afterward understanding and effectively using) models for the assessment of research is linked and depends, among other factors, on the degree or depth of the
The level of conceptualization and formalization of Quality, however, is neither objective nor unique. It depends on the purposes and the subject or unit of the analysis (e.g. scholars, groups, institutions, up to meso or macro aggregated units, as regional or national entities) and it relates, in the end, to the specific evaluation problem under investigation.
We propose, finally, three
We maintain that these three enabling conditions contribute to the conceptualization and formalization of the idea of Quality that is related and fosters the overlap of the different perspectives, namely modeling world, empirical world, and policy world (see Section 4 and Figure 2 in Section 5).
An illustration of the relationship between modeling world, empirical world, and policy world: they are all somewhat overlapping visions or projections of the real worlds.Figure 2
Summing up, evaluating research and its impacts is a real complex task. Perhaps the key problem is that research performance is not fully quantifiable. Hence, research assessment has to deal with non-fully quantifiable concepts.
There are several approaches to evaluating research. In order to adopt and use our framework, the following three postulates, intended as general validity conditions or principles, have to be accepted.
This postulate is a proposition that we assume to be true because it is obvious. The implication of Postulate 1 is that if the model underlying the metric is not described, this does not mean that it is more robust to modeling choice. It simply means that you do not state clearly and in detail and account for the underlying theoretical choices, methodological assumptions, and data limits considerations. Put in other words, the metric cannot be more robust than the model, and it is possible to assess the robustness of the model only if it is explicitly described.
This is the cornerstone postulate of our framework. The accuracy, completeness, and consistency of the research assessment depends upon and is limited by, among other factors, the complexity of the research evaluation. A further discussion on this issue can be found in Daraio (2017a).
A metric developed according to a model that conceptualizes and formalizes in an unambiguous way the idea of Quality in its different layers and meanings is able to substantiate and give content to the concept of “responsible” metrics.
Postulate 3 should be considered to be an
The main contributions of the paper are:
To introduce a simple framework that could be helpful in developing models for metrics of research assessment (e.g. a kind of To propose a basis for the research of the To outline directions for further research.
Our framework acts as a
For
Table 2 reports some streams of literature which have considered research and innovation, which are somewhat overlapping, as the main interplay of science and society together with education.
A non-exhaustive overview of the literature on the
From the economics of education we know that education is an investment in human capital analogous to an investment in physical capital.
People represent the link between all these streams of literature. People in fact attend schools and higher education institutions, acquiring competences and skills. People are educated first and after that do research and carry out innovative activities during which they continue to learn, acquiring/extending their competences and skills and so on.
Moreover, higher education systems are increasingly expanding their interplay with society moving toward markets in higher education systems or going beyond. There are some science and public policy studies that have analyzed the elements of societal impact, mostly rooting it into universities and public research characteristics (Bornmann, 2013) whilst others, mostly refer to approaches developed by practitioners (Ebrahim & Rangan, 2014) An interesting survey on university-industry relations can be found in Perkmann et al. (2013).
The existing literature, summarized in Table 2, can be systematized around the
Research and teaching institutions provide their environment infrastructural and knowledge assets. These act as resources in the assessment of the impact of those institutions on the innovation of the economic system. The transmission channels of the impact which emerge from previous literature are, just to cite a few, mobility of researchers, career of alumni, applied research contracts, and joint use of infrastructures. In this context, different theories and models of the system of knowledge production and allocation could be developed and tested. According to Gibbons et al. (1994), knowledge is produced by configuring human capital that is more malleable than physical capital. Indeed human capital can be configured in different ways to generate new forms of specialized knowledge. The new economics of production can be interpreted as a shift from search for economies of scale to economies of scope where the latter arise from the ability to re-configurate human resources and particularly knowledge in new ways (Gibbons et al., 1994, p. 63). Traditional and new forms of knowledge creation (Mode 1 and Mode 2 according to Gibbons et al. (1994) definitions) co-exist and dynamically evolve. The dynamics of knowledge production, distribution, co-creation, and evolution obviously matter for the assessment of research and its impact.
The assessment of research cannot be addressed in isolation without education and innovation. It requires the specification of variables and indicators consistent with a systemic view.
Results can widely differ at different levels of aggregation, for instance at the public research organization and higher education institution level or individual university/research center, or faculty or team down to individual scholar. At these different levels, the possible moderating variables or causes of different performances may change too. Examples of possible moderating variables are: the legislation and regulation, public funding, teaching fees, and duties; geography, characteristics of the local economic and cultural system, effectiveness of research and recruiting strategy, budgeting, and infrastructures; intellectual ability of researchers, historical paths, ability to recruit doctoral students, world-wide network of contacts, and the like.
A quite complete comparison of the main advantages and disadvantages of quantitative approaches, such as citation based indicators,
Evidently, the means should be identified in accordance with the subject of the assessment. The organization and the communication aspects of the evaluation, however, fall within the sphere of policy and governance.
We propose three building blocks for
Dimensions of methodology: subject and means in our framework.Dimension Type/category Content Output (baseline) Result of a transformation process which uses inputs to produce products or services Productivity and efficiency Partial or total factor productivity with respect to a reference Effectiveness Considering inputs and outputs, and accounting for the aims of the activities Impact All contributions of research outside academia Quantitative approaches Qualitative approaches Quali-quantitative approaches
A distinction between productivity and efficiency is in order. Productivity is the ratio of the outputs over the inputs. Efficiency, in the broad sense, is defined as the output/input relation with respect to an estimated reference frontier, or frontier of the best practices (Daraio & Simar, 2007, p. 14). The econometrics of production functions is different from that of production frontiers as the main objective of their analysis differs: production functions look at average behavior whilst production frontiers analyze the whole distribution, taking into account the best/worst behavior (Bonaccorsi & Daraio, 2004). Obviously, assessing the impact on the average performance is different from assessing the impact on the best/worst performance. Accounting for inequality and diversity is much more natural in a model based on best/worst performance frontiers than in a standard average or representative behavior model. This is because in the former case the whole distribution is considered instead of only the central tendency. This distinction between “average”
As far as quantitative methods are concerned, different approaches, both parametric (Galán, Veiga, & Wiper, 2014) and non-parametric (Bădin, Daraio, & Simar, 2012; 2014; Daraio & Simar, 2014) have been proposed, highlighting the changes required by the attempt to disentangle the impact of external-heterogeneity factors on the efficient frontier from that on the distribution of inefficiency. This trend witnesses the need to move from the assessment of efficiency toward the assessment of impacts. Some precursors of methodological challenges and changes within the frontier approach may be identified, without being complete, in:
statistical approach to non-parametric frontier estimation (Daraio, Simar, & Wilson, 2016; Simar & Wilson, 2015): trend toward a data-driven modeling; Models averaging in stochastic frontier estimation (Parmeter, Wan, & Zhang, 2016): trend toward robustness of modeling; Using information about technologies, markets, and behavior of institutions in productivity indices (O’Donnell, 2016); trend toward more comprehensive informational setup; and From an implementation point of view, interactive benchmarking (e.g. Bogetoft, Fried, & Eeckaut 2007); trend toward developing analytics for policy decision making support.
Moving from efficiency to effectiveness is an important step. At this purpose, the inclusion of managerial and more qualitative aspects in the quantitative benchmarking models could be beneficial. According to Drucker (1967),
The methodological dimension should handle how to evaluate what, providing an appropriate account of reliability and robustness (Glänzel, 2010; Glänzel & Moed, 2013), and uncertainty.
An interesting distinction exists between uncertainty and sensitivity analysis.
These are all considerations which refer to the
Use models Adopt an Detect Find Aim for Do not do the sums right but
We should move on from efficiency to effectiveness, and then toward impact, shifting our current paradigm, including quality indicators to assess effectiveness instead of efficiency; considering the quality of the applied method and the overall quality of the model.
The
Besides this positive view on data, data has a
The main building blocks we identify to characterize the
A characterization of the data dimension in our framework.Dimension Characterization usability sampling freely, controlled or undisclosed consumption open, institutional provided commercial privacy/confidentiality (see Ekbia et al. (2015)) a very high level is obtained by an OBDM approach (see Daraio et al. (2016b)) independence of the data from the unit of analysis
A relevant connection, also for the following developments of modeling is the relationship between data and information. According to Floridi (2014), information and communication technologies (ICT) have brought new opportunities as well as new challenges for human development and have led to a revolutionary shift in our understanding of humanity’s nature and its role in the universe, the “fourth revolution” according to which “we are now slowly accepting the idea that we might be informational organisms among many agents…,
An interesting and perhaps connected change, due to the developments introduced in information processing including novel algorithms, protocols, and properties of information brings to shift from the classical to the quantum computation paradigm and recently leads to derive quantum theory as a special theory of information (D’Ariano & Perinotti, 2016. For an introduction, see Nielsen & Chuang (2010).).
Within this context emerged the
Our general framework is derived integrating relevant dimensions, grounded in existing approaches, according to the three main dimensions illustrated in Figure 1. This framework could allow for combining the fine-grained results of case studies, with the ability to replicate and route them, taking them to a higher level, thanks to an integrated view, which maps the interfaces, interdependencies, complementarities among the three dimensions and allows for analyzing the constraints on the three dimensions that may make analysis difficult.
Concerning Quality, in the field of education, much progress has been made. The quality of education has been demonstrated as relevant for research and innovation.
Several contributions have analyzed the impact of education quality on economic growth (e.g. Aghion, 2009; Hanushek & Woessmann, 2007; Hanushek et al., 2008).
Much more work is needed for research and innovation due to the inherent difficulties that arise for their specific content, context, and complexity. The main object of a research evaluation is represented by the results of given research activities, which can be considered the research effort (Hemlin, 1996). The outputs of a given research activity are the result of a complex set of interacting characteristics and activities that involve, but are not limited to: ability, talents, social aspects, luck, incentives, motivations, trade-offs, commitment, financial resource, efforts, infrastructure, education, personality skills, network, organization, curiosity, communication skills, and contextual and institutional factors. These all interact dynamically, giving rise to complex processes. The evaluation of research is done in a context characterized by many more different factors that interact as well. Hemlin (1996, p. 210) points out that “all evaluation of research quality must be based on an idea of the meaning of this concept. […] The variety in meaning of scientific quality reflects the fact that research evaluations are being made in a context in which a number of different factors interact and where the interplay between these factors is essential to the concept of quality in science … not only the real interplay between factors is important, but also the evaluators conceptions of this interplay is crucial.”
The meaning of scientific quality and its difficulties in delimiting what is meant by it are related to the nature of research itself. The conception of what is good or bad research varies between different research areas and periods, constantly changing as the result of an interactive process between scientific development and events in the world outside the scientific community.
All these aspects show the complexity of the evaluation of research.
Issues of uncertainty are closely related to those of quality of information. Problems of quality of information are involved whenever policy related research is utilized in the policy process (Funtowicz & Ravetz, 1990, p. 11). In assessing research, it is important also to consider the interactions of quality with uncertainty and policy, “in a situation where major decisions, on the most complex and uncertain issues, must frequently be made under conditions of urgency” (Funtowicz & Ravetz, 1990, p. 13).
From a methodological point of view, the inclusion of quality indicators in the analysis, may allow us to move from efficiency to effectiveness. Effectiveness can be captured then by using in the analysis “qualitative-adjusted” quantitative measures. In the end, maybe, although difficult to assess, it is the quality of education, research, and innovation, which has an impact on the development of the society.
Finally, it is on the
Data quality according to the OECD (2011) Quality Framework is defined with respect to user needs, and it has seven dimensions: relevance (“degree to which data serves to address their purposes”); accuracy (“how the data correctly describes the features it is designed to measure”); credibility (“confidence of users in the data products and trust in the objectivity of the data”); timeliness (“length of time between its availability and the phenomenon it describes”); accessibility (“how readily the data can be located and accessed”); interpretability (“the ease with which the user may understand and properly use and analyze the data”); coherence (“the degree to which it is logically connected and mutually consistent”).
Quality of available data is crucial; in data quality there have been relevant advances, going from data quality to
Quality as acceptability (suitability) for application (fitness for purpose) is the overarching concept, which keeps together the building blocks of the three dimensions of our framework. It is a characteristic in all the three dimensions.
The nine building blocks are attributes of Quality. The quality of
From the description so far, it emerges that the assessment of the research activity is indeed a complex task. Now, our finding could be interpreted in two ways:
Due to the complexity of the evaluation of research described so far, it is more appropriate to talk about
In our framework we identify three implementation factors and three enabling conditions that may be helpful to monitor the model development.
We highlight again that our framework is able to act as a
In
In a
Transparency and openness are two implementation factors that can be detailed along the main building blocks of our framework and have a self-evident importance.
For
According to OECD (2015a), Open Science refers to “efforts by researchers, governments, research funding agencies or the scientific community itself to make the primary outputs of publicly funded research results publications and the research data publicly accessible in digital format with no or minimal restriction as a means for accelerating research; these efforts are in the interest of enhancing transparency and collaboration, and fostering innovation.” “[…] Three main aspects of open science are: open access, open research data, and open collaboration enabled through ICT. Other aspects of open science post-publication peer review, open research notebooks, open access to research materials, open source software, citizen science, and research crowdfunding are also part of the architecture of an open science system” (OECD, 2015a, p. 7).
Nielsen (2012) develops the concept of open research a bit further, talking about “data-driven intelligence” controlled by human intelligence which amplifies collective intelligence: “To amplify cognitive intelligence, we should scale up collaborations, increasing cognitive diversity and the range of available expertise as much as possible. Ideally, the collaboration will achieve designed serendipity…” According to Nielsen (2012) this could be achieved by conversational critical mass and collaboration which becomes self-stimulating with online tools, which may establish architecture of attention that directs each participant where it is best suited. This collaboration may follow the patterns of open source software: commitment to working in a modular way; encouraging small contributions; allowing easy reuse of earlier work; using signaling mechanisms (e.g. scores) to help people to decide where to direct attention.
The exponential increase and development of information availability and the development of the information society is leading us toward an open innovation society (see e.g. Chesbrough (2012)) West et al. (2014) in reviewing the open innovation literature identify three main directions of research: better measurement, resolving the role of appropriability, and linking open innovation to the management and economics literature
In this model, government, industry, academia, and civil participants work together to
Although the Quadruple Helix model gives emphasis to the broad idea of cooperation in innovation, it is not a very well established and much used concept in research and innovation studies, because of its conceptual and practical elusiveness. We argue here that our framework could be a valid support for the conceptualization and the implementation of a Quadruple Helix model.
The first enabling condition is
The second enabling condition refers to
The third enabling condition refers to the Within some research projects funded by Sapienza University in 2013 and 2015 we did an experiment of a knowledge infrastructure, a case of an “open science of science” exercise, around Sapientia: The Ontology of Multi-Dimensional Research Assessment (Daraio et al., 2016a; 2016b). Sapientia represents an effort of going toward a common platform which can show which data has to be collected; by offering the opportunity of making analysis under different perspectives, testing different models, but sharing the same common conceptual characterization.
In the next section, Figure 2 illustrates the connections of our modeling framework with the empirical, policy, and real world. The enabling conditions foster these connections.
The discussion so far seems incomplete: what is missing? Perhaps much, but we identify two things at least: the connection to the real world and a “reference” against which to monitor the development of the model of research evaluation. We try to illustrate the contribution of our framework with respect to the different “representations” of the real world involved in research evaluation processes. Figure 2 shows the interconnections between the different views of the real world, made by the policy world, the modeling world, and the empirical world. The illustration of the different representations as concentric ellipses denotes the fact that each world is perceived differently from other worlds.
Figure 2 shows the role of our modeling framework in its interplay with the empirical and policy world for the understanding of the real world. We claim that the more the Quality is conceptually and formally specified, the more the overlapping area among modeling, policy, and empirical worlds is, and closer to the real world the model is.
This statement is basically Postulate 2 of our framework (see Section 1). It is linked to the second missing item introduced before, namely the need to have a “reference” for checking the development of the model. It also calls for the introduction of the third postulate which is the monitoring of the developments and the evolutions of the modeling activity can be carried out in relation to the “responsibility” of the metrics proposed and involved.
But what does being a “responsible metric” mean in an evaluation process? According to Cambridge Dictionary, to be responsible could be defined as “be responsible for something or someone” that means “to have control and authority over someone or something and the duty of taking care of it;” or as “be responsible to something or someone” that means “to be controlled by someone or something.” Does “responsible” relate to metric itself or to its use, or both? Wilsdon et al. (2015, p. x) propose the notion of responsible metrics as “a way of framing appropriate uses of quantitative indicators in the governance, management and assessment of research […]”:
“Responsible metrics can be understood in terms of the following dimensions: Robustness: basing metrics on the best possible data in terms of accuracy and scope; Humility: recognizing that quantitative evaluation should support but not supplant qualitative, expert assessment; Transparency: keeping data collection and analytical processes open and transparent, so that those being evaluated can test and verify the results; Diversity: accounting for variation by field, and using a range of indicators to reflect and support a plurality of research and researcher career paths across the system; Reflexivity: recognizing and anticipating the systemic and potential effects of indicators, and updating them in response” (Wilsdon et al., 2015, p. x).
Interestingly, also Benessia et al. (2016) propose “responsible” metrics at the end of their discussion on the crisis of science.
After the publication of From the website
Coming back to our framework, we identify some connections of its enabling conditions with the oeuvre of Alasdair MacIntyre See for instance Lutz (2017) and Ballard (2000) that describes an overview on MacIntyre’s oeuvre, reporting a rich bibliography on his works.
Toward an ethics of research assessment? Some connections of our framework with MacIntyre’ oeuvre.Enabling condition Potential connection MacIntyre work Invitation to overcome the fragmentation of knowledge and excessive specialization. The need to go beyond a pure quantitative approach (abstract representation of the reality) and include qualitative cases (narratives and storytelling). Retrieve the values of tradition in communities of practice that regulate themselves by defining their own standards.
The third postulate of our framework, reported in the Introduction, gives the ability to give content to the concept of “responsible metrics” to the grade (level) of conceptualization and formalization, in an unambiguous way, of the different layers/meanings of “Quality.”
This could permit to give content to the somewhat “vague” idea of “excellence” (Moore et al., 2017) as well. These activities of conceptualization and formalization of Quality are strictly linked to the production, use, and effects of “standards.” It is useful to recall here a precursor paper on the need for standards in bibliometrics. We refer to the work of Glänzel (1996), still relevant today, more than 20 years after its publication. As clearly illustrated by Brunsson and Jacobsson (2002a), standardization may be a valid alternative to market forces and to organizational forms as an institutional arrangement for coordinating and controlling complex exchanges. Brunsson and Jacobsson (2002b) summarize the arguments in favor of standardization in “more effective use of information, better coordination of activities, simplification, and the advantages of large-scale production” (Brunsson & Jacobsson (2002b, p. 170). On the other hand, they summarize the arguments against standardization in those similar to the objections against rules and regulation in general, lack of trust in the expertise and goodwill of those who set the rules, critics of those that prefer markets to standards, or of those that want, on the other hand, a stronger formal coordination way (such as directives) (Brunsson & Jacobsson, 2002b, pp. 171–172). In concluding their essay and the entire book, Brunsson and Jacobsson (2002b, p. 172) state that “Standardization deserves to be paid a good deal more attention than it has received up to now,” and “… we may have something to learn from the old Greek myths. In a way, standardizing is the art of constructing a Procrustean bed. Procrustes was a legendary bandit in Greek mythology, a bandit who placed his victims on a specially constructed bed. The bed was a pattern and a yardstick intended to create conformity… (p. 173).” We share their conclusions, and believe that their reference to the procrustean heritage could be an interesting starting point to further explore and develop the connections of our framework with MacIntyre’s oeuvre (Table 5). Further research on the connections with MacIntyre’s oeuvre could help to fill an existing gap providing new tools to assess efficiency together with More reading can be found in Furner (2014).
The main objective of this paper is to provide a
Our framework may be particularly useful to develop models of research assessment, to frame the traditional problems of evaluation in a wider perspective and to facilitate the introduction of new methods for the assessment of research relevant to support their governance. The framework introduced has the ambition of being general and valid for different units and layers of analysis. For this reason it needs to be corroborated, tested, and extended to different specific evaluation cases.
This paper may open the way to many extensions and further research:
Testing the proposed framework for developing effective checklists for designing and implementing policy monitoring mechanisms on the assessment of research activities along the lines of Daraio (in press); Running additional research for providing a systematic review, analysis, and classification of the existing literature, having our framework as a Corroborating the framework facing the problem of the democratization of the evaluation (Daraio, 2017b); Extending the proposed framework to the characterization of different governance systems (Capano, Howlett, & Ramesh, 2015) for analyzing their systemic connection with their performance; Applying the framework, Investigating the Corroborating the framework for the regulation of the evaluation of research.
Finally, our framework may pave the way for new