Selecting among alternative projects is a core management task in all innovating organizations. In this paper, we focus on the evaluation of frontier scientific research projects. We argue that the “intellectual distance” between the knowledge embodied in research proposals and an evaluator’s own expertise systematically relates to the evaluations given. To estimate relationships, we designed and executed a grant proposal process at a leading research university in which we randomized the assignment of evaluators and proposals to generate 2,130 evaluator–proposal pairs. We find that evaluators systematically give lower scores to research proposals that are closer to their own areas of expertise and to those that are highly novel. The patterns are consistent with biases associated with boundedly rational evaluation of new ideas. The patterns are inconsistent with intellectual distance simply contributing “noise” or being associated with private interests of evaluators. We discuss implications for policy, managerial intervention, and allocation of resources in the ongoing accumulation of scientific knowledge.This paper was accepted by Lee Fleming, entrepreneurship and innovation.
The science of art advancement: Art appreciation is highly subjective. Fraiberger et al. used an extensive record of exhibition and auction data to study and model the career trajectory of individual artists relative to a network of galleries and museums. They observed a lock-in effect among highly reputed artists who started their career in high-prestige institutions and a long struggle for access to elite institutions among those who started their career at the network periphery. Science , this issue p. 825
17 May 2021 in Proceedings of the National Academy of Sciences
Significance: Collective intelligence (CI) is critical to solving many scientific, business, and other problems. We find strong support for a general factor of CI using meta-analytic methods in a dataset comprising 22 studies, including 5,279 individuals in 1,356 groups. CI can predict performance in a range of out-of-sample criterion tasks. CI, in turn, is most strongly predicted by group collaboration process, followed by individual skill and group composition. The proportion of women in a group is a significant predictor of group performance, mediated by social perceptiveness.
We study how contributors to innovation contests improve their performance through direct experience and by observing others as they synthesize learnable signals from different sources. Our research draws on a 10-year panel of more than 55,000 individuals participating in a firm-hosted online innovation community sponsoring creative t-shirt design contests. Our data set contains almost 180,000 submissions that reflect signals of direct performance evaluation from both the community and the firm. Our data set also contains almost 150 million ratings that reflect signals for learning from observing the completed work of others. We have three key findings. First, we find a period of initial investment with decreased performance. This is because individuals struggle to synthesize learnable signals from early performance evaluation. This finding is contrary to other studies that report faster learning from early direct experience when improvements are easiest to achieve. Second, we find that individuals consistently improve their performance from observing others’ good examples. However, whether they improve from observing others’ bad examples depends on their ability to correctly recognize that work as being of low quality. Third, we find that individuals can successfully integrate signals about what is valued by the firm hosting the community, not just about what is valued by the community. We thus provide important insights into the mechanisms of how individuals learn in crowdsourced innovation and provide important qualifications for the often-heralded theme of “learning from failures.”The online appendix is available at https://doi.org/10.1287/orsc.2018.1219 .
1 December 2017 in Academy of Management Discoveries
Organizations are increasingly turning to crowdsourcing to solve difficult problems. This is often driven by the desire to find the best subject matter experts, strongly incentivize them, and engage them, with as little coordination cost as possible, to pool their knowledge. A growing number of authors, however, are calling for increased collaboration in crowdsourcing settings, hoping to draw upon the advantages of teamwork observed in traditional settings. The question is how to effectively incorporate team-based collaboration in a setting that has traditionally been individual-based. We report on a large field experiment of team collaboration on an online platform, in which incentives and team membership were randomly assigned, to evaluate the influence of exogenous inputs (member skills and incentives) and emergent collaboration processes on performance of crowd-based teams. Building on advances in machine learning and complex systems, we leverage new measurement techniques to examine the content and timing of team collaboration. We find that temporal "burstiness" of team activity and the diversity of information exchanged among team members are strong predictors of performance, even when inputs such as incentives and member skills are controlled. We discuss implications for research on crowdsourcing and team collaboration.
28 February 2013 in International Journal of Human–Computer Interaction
Social connectedness is an indicator of the extent to which people can realize various network benefits and is therefore a source of social capital. Using the case of Twitter, a theoretical model of social connectedness based on the functional and structural characteristics of people's communication behavior within an online social network is developed and tested. The study investigates how social presence, social awareness, and social connectedness influence each other, and when and for whom the effects of social presence and social awareness are most strongly related to positive outcomes in social connectedness. Specifically, the study looks at the concurrent direct and moderating effect of two structural constructs characterizing people's online social network: network size and frequency of usage. The research model is tested using data (n = 121) collected from two sources: (a) an online survey of Twitter users and (b) their usage data collected directly from Twitter. Results indicate that social awareness, social presence, and usage frequency have a direct effect on social connectedness, whereas network size has a moderating effect. Social presence is found to partially mediate the relationship between social awareness and social connectedness. The findings of the analysis are used to outline design implications for online social networks from a human–computer interaction perspective.