Doing Things Twice: Strategies to Identify Studies for Targeted Validation

Abstract: The “reproducibility crisis” has been a highly visible source of scientific controversy and dispute. Here, I propose and review several avenues for identifying and prioritizing research studies for the purpose of targeted validation. Of the various proposals discussed, I identify scientific data science as being a strategy that merits greater attention among those interested in reproducibility. I argue that the tremendous potential of scientific data science for uncovering high-value research studies is a significant and rarely discussed benefit of the transition to a fully open-access publishing model.

  • Gopal P. Sarma, “Doing Things Twice: Strategies to Identify Studies for Targeted Validation” (2017). [Preprint]

Scientific Data Science and the Case for Open Access

Abstract: “Open access” has become a central theme of journal reform in academic publishing. In this article, I examine the consequences of an important technological loophole in which publishers can claim to be adhering to the principles of open access by releasing articles in proprietary or “locked” formats that cannot be processed by automated tools, whereby even simple copy and pasting of text is disabled. These restrictions will prevent the development of an important infrastructural element of a modern research enterprise, namely, scientific data science, or the use of data analytic techniques to conduct meta-analyses and investigations into the scientific corpus. I give a brief history of the open access movement, discuss novel journalistic practices, and an overview of data-driven investigation of the scientific corpus. I argue that particularly in an era where the veracity of many research studies has been called into question, scientific data science should be one of the key motivations for open access publishing. The enormous benefits of unrestricted access to the research literature should prompt scholars from all disciplines to reject publishing models whereby articles are released in proprietary formats or are otherwise restricted from being processed by automated tools as part of a data science pipeline..

  • Gopal P. Sarma, “Scientific Data Science and the Case for Open Access” (2016). [Preprint]

Collecting Systematic, Introspective Reports of Pharmacological Effects and Side-Effects

Abstract: The study of subjective, first-person experience is a topic with both philosophical and practical implications. In this article, I discuss the value of collecting a critical mass of prose or verbal descriptions of introspectively determined, subjective effects of pharmacological agents. I suggest that datasets of introspective reports fit in the modern research landscape at the intersection of biomedical informatics and the emerging discipline of contemplative neuroscience. I compare the current proposal to Descriptive Experience Sampling (DES), discuss relevant methodological and conceptual issues in the study of introspection, and provide a list of questions for directing future investigation.

  • Gopal P. Sarma, “Collecting Systematic, Introspective Reports of Pharmacological Effects and Side-Effects” (2016). [Preprint]

Mammalian Value Systems

Abstract: Characterizing human values is a topic deeply interwoven with the sciences, humanities, political philosophy, art, and many other human endeavors. In recent years, a number of thinkers have argued that accelerating trends in computer science, cognitive science, and related disciplines foreshadow the creation of intelligent machines which meet and ultimately surpass the cognitive abilities of human beings, thereby entangling an understanding of human values with future technological development. Contemporary research accomplishments suggest increasingly sophisticated AI systems becoming widespread and responsible for managing many aspects of the modern world, from preemptively planning users’ travel schedules and logistics, to fully autonomous vehicles, to domestic robots assisting in daily living. The extrapolation of these trends has been most forcefully described in the context of a hypothetical “intelligence explosion,” in which the capabilities of an intelligent software agent would rapidly increase due to the presence of feedback loops unavailable to biological organisms. The possibility of superintelligent agents, or simply the widespread deployment of sophisticated, autonomous AI systems, highlights an important theoretical problem: the need to separate the cognitive and rational capacities of an agent from the fundamental goal structure, or value system, which constrains and guides the agent’s actions. The “value alignment problem” is to specify a goal structure for autonomous agents compatible with human values. In this brief article, we suggest that recent ideas from affective neuroscience and related disciplines aimed at characterizing neurological and behavioral universals in the mammalian kingdom provide important conceptual foundations relevant to describing human values. We argue that the notion of “mammalian value systems” points to a potential avenue for fundamental research in AI safety and AI ethics.

  • Gopal P. Sarma and Nick J. Hay, “Mammalian Value Systems” (2016). [Preprint]

Reductionism and the Universal Calculus

Abstract: In the seminal essay, “On the unreasonable effectiveness of mathematics in the physical sciences,” physicist Eugene Wigner poses a fundamental philosophical question concerning the relationship between a physical system and our capacity to model its behavior with the symbolic language of mathematics. In this essay, I examine an ambitious 16th and 17th-century intellectual agenda from the perspective of Wigner’s question, namely, what historian Paolo Rossi calls “the quest to create a universal language.” While many elite thinkers pursued related ideas, the most inspiring and forceful was Gottfried Leibniz’s effort to create a “universal calculus,” a pictorial language which would transparently represent the entirety of human knowledge, as well as an associated symbolic calculus with which to model the behavior of physical systems and derive new truths. I suggest that a deeper understanding of why the efforts of Leibniz and others failed could shed light on Wigner’s original question. I argue that the notion of reductionism is crucial to characterizing the failure of Leibniz’s agenda, but that a decisive argument for the why the promises of this effort did not materialize is still lacking.

  • Gopal Sarma, “Reductionism and the Universal Calculus”, submitted (2016). [Preprint]

Scientific Auditing Firms

Abstract: Recent analyses have brought to light a startling reality about contemporary science, namely, low rates of reproducibility in research studies across many disciplines. On the other hand, the legitimately world-changing advances that have taken place in the last half-century have also resulted in theoretical knowledge and experimental capacity so advanced that outstanding and meticulously performed science can often be difficult to understand and to interpret to all but a few specialists in a field. In anticipating the future needs of a scientific establishment whose complexity is only increasing, I propose the concept of scientific auditing firms, independent organizations whose primary responsibility is to conduct random audits of the scientific literature. In addition to creating a disincentive for those who might otherwise engage in scientific misconduct, these firms could play the role of global monitors of broad scientific trends as well as provide consulting services for difficult technical and conceptual problems in academia and industry. I use the idea of a scientific auditing firm as a thought experiment with which to focus a discussion on reproducibility, peer review, and the tradeoff between incentive structures for novelty versus correctness. I then sketch an outline of how such an entity might be constructed in practice and the skills that would be required of its personnel. Finally, I suggest that a “mock” trial run of a simplified firm consisting of a few researchers over a short-time period could be conducted today with minimal resources and would provide valuable insight into the feasibility of this proposal.

  • Gopal Sarma, “Scientific Auditing Firms”, Progress 05(2016). [Journal]
    (A shorter version of this article is available: [Preprint])

Training Scientific Generalists: Response to Comments and Additional Thoughts

Abstract: In several recent articles, I proposed the creation of new graduate programs aimed at training scientific generalists. Here, I collect and respond to a number of comments and criticisms raised in response to these proposals.

  • Gopal Sarma, “Training Scientific Generalists: Response to Comments and Additional Thoughts”, The Winnower 05(2016). [Journal]

Is There Value in Training Scientific Generalists For Positions at the Edge of Academia?

Abstract: Contemporary scientific research faces major cultural and institutional hurdles. Some of the primary challenges include an exploding knowledge base and organizational complexity of many scientific projects, the overproduction of PhDs relative to the availability of faculty positions, and protracted educational trajectories for many aspiring researchers. Perhaps the most serious set of consequences caused by the fierce competition of modern science are low rates of reproducibility in research studies across many disciplines, a startling reality which undermines the scientific process and institutional authority itself. In an increasingly interconnected intellectual world, where fundamental and applied research are deeply interwoven, the implications of this state of affairs extend well beyond the research laboratory. In this article, I explore one possible strategy among the many necessary interventions for addressing these critical global issues, namely, new graduate programs to train scientific generalists. Rather than focus on developing niche technical skills, these programs would train outstanding communicators and decision makers who have been exposed to multiple subjects at the graduate level. The motivation for creating such programs is to introduce a large number of exceptionally trained individuals across all industries and organizations who have been encouraged to think critically about the practical realities and contemporary cultural trends of scientific research. I suggest possible avenues for structuring such programs and examine the roles that generalists might play in the modern research, policy, and industrial landscape.

  • Gopal Sarma, “Is There Value in Training Scientific Generalists For Positions at the Edge of Academia?”, The Winnower 03(2016). [Journal]

Unit Testing, Model Validation, and Biological Simulation

Abstract: The growth of the software industry has gone hand in hand with the development of tools and cultural practices for ensuring the reliability of complex pieces of software. These tools and practices are now acknowledged to be essential to the management of modern software. As computational models and methods have become increasingly common in the biological sciences, it is important to examine how these practices can accelerate biological software development and improve research quality. In this article, we give a focused case study of our experience with the practices of unit testing and test-driven development in OpenWorm, an open-science project aimed at modeling Caenorhabditis elegans. We identify and discuss the challenges of incorporating test-driven development into a heterogeneous, data-driven project, as well as the role of model validation tests, a category of tests unique to software which expresses scientific models.

  • Gopal P. Sarma, Travis W. Jacobs, Mark D. Watts, S. Vahid Ghayoomie, Stephen D. Larson, and Rick C. Gerkin, “Unit Testing, Model Validation, and Biological Simulation”, F1000Research 5:1946 (2016). [Journal][Preprint]

A Simple Technique to Record Mental Events

Abstract: In recent years, there has been growing interest in bridging bodies of knowledge from introspective and contemplative traditions with modern neuroscience. By making the primary object of study an individual’s subjective experience, scientists are then confronted with the challenging problem of how to record a given mental state at a given point in time. For simple experiences, such as in facial recognition tasks, an external recording device such as a button box or computer keyboard is adequate. However, these devices pose fundamental problems when attempting to record more subtle states of mind, or to do so in situations where the presence of an external input device would disturb the subject, such as in the case of studying the evolution of internal states of highly advanced meditators. Here, I suggest that variants of the mnemonic techniques used in memory competitions can be used to make measurements in experiments where an external device would be impractical to use, or would unnecessarily disturb the subject. Demographic and performance data from memory competitions suggest that these techniques can be learned by ordinary subjects participating in multi-session experiments, and that, for highly trained individuals, events can be recorded at roughly 400 millisecond intervals for timescales on the order of tens of seconds, and at roughly one-second intervals for sustained periods of one hour or longer. Finally, I discuss the relationship of the current proposal to descriptive experience sampling (DES) and the visualization techniques of the Tibetan meditative tradition.

  • Gopal Sarma, “A simple technique to record mental events,” Journal of Consciousness Studies 22(7-8) (2015). [Journal]