Scientific Auditing Firms

Abstract: Recent analyses have brought to light a startling reality about contemporary science, namely, low rates of reproducibility in research studies across many disciplines. On the other hand, the legitimately world-changing advances that have taken place in the last half-century have also resulted in theoretical knowledge and experimental capacity so advanced that outstanding and meticulously performed science can often be difficult to understand and to interpret to all but a few specialists in a field. In anticipating the future needs of a scientific establishment whose complexity is only increasing, I propose the concept of scientific auditing firms, independent organizations whose primary responsibility is to conduct random audits of the scientific literature. In addition to creating a disincentive for those who might otherwise engage in scientific misconduct, these firms could play the role of global monitors of broad scientific trends as well as provide consulting services for difficult technical and conceptual problems in academia and industry. I use the idea of a scientific auditing firm as a thought experiment with which to focus a discussion on reproducibility, peer review, and the tradeoff between incentive structures for novelty versus correctness. I then sketch an outline of how such an entity might be constructed in practice and the skills that would be required of its personnel. Finally, I suggest that a “mock” trial run of a simplified firm consisting of a few researchers over a short-time period could be conducted today with minimal resources and would provide valuable insight into the feasibility of this proposal.

  • Gopal Sarma, “Scientific Auditing Firms”, Progress 05(2016). [Journal]
    (A shorter version of this article is available at Economic and Social Impacts of Innovation 05(2016). [Journal][Preprint])

Training Scientific Generalists: Response to Comments and Additional Thoughts

(See ScientificGeneralists.org for additional discussion.)

Abstract: In several recent articles, I proposed the creation of new graduate programs aimed at training scientific generalists. Here, I collect and respond to a number of comments and criticisms raised in response to these proposals.

  • Gopal Sarma, “Training Scientific Generalists: Response to Comments and Additional Thoughts”, The Winnower 05(2016). [Journal]

Is There Value in Training Scientific Generalists For Positions at the Edge of Academia?

(See ScientificGeneralists.org for additional discussion.)

Abstract: Contemporary scientific research faces major cultural and institutional hurdles. Some of the primary challenges include an exploding knowledge base and organizational complexity of many scientific projects, the overproduction of PhDs relative to the availability of faculty positions, and protracted educational trajectories for many aspiring researchers. Perhaps the most serious set of consequences caused by the fierce competition of modern science are low rates of reproducibility in research studies across many disciplines, a startling reality which undermines the scientific process and institutional authority itself. In an increasingly interconnected intellectual world, where fundamental and applied research are deeply interwoven, the implications of this state of affairs extend well beyond the research laboratory. In this article, I explore one possible strategy among the many necessary interventions for addressing these critical global issues, namely, new graduate programs to train scientific generalists. Rather than focus on developing niche technical skills, these programs would train outstanding communicators and decision makers who have been exposed to multiple subjects at the graduate level. The motivation for creating such programs is to introduce a large number of exceptionally trained individuals across all industries and organizations who have been encouraged to think critically about the practical realities and contemporary cultural trends of scientific research. I suggest possible avenues for structuring such programs and examine the roles that generalists might play in the modern research, policy, and industrial landscape.

  • Gopal Sarma, “Is There Value in Training Scientific Generalists For Positions at the Edge of Academia?”, The Winnower 03(2016). [Journal]

Unit Testing, Model Validation, and Biological Simulation

Abstract: The growth of the software industry has gone hand in hand with the development of tools and cultural practices for ensuring the reliability of complex pieces of software. These tools and practices are now acknowledged to be essential to the management of modern software. As computational models and methods have become increasingly common in the biological sciences, it is important to examine how these practices can accelerate biological software development and improve research quality. In this article, we give a focused case study of our experience with the practices of unit testing and test-driven development in OpenWorm, an open-science project aimed at modeling Caenorhabditis elegans. We identify and discuss the challenges of incorporating test-driven development into a heterogeneous, data-driven project, as well as the role of model validation tests, a category of tests unique to software which expresses scientific models.

  • Gopal P. Sarma, Travis W. Jacobs, Mark D. Watts, S. Vahid Ghayoomie, Stephen D. Larson, and Rick C. Gerkin, “Unit Testing, Model Validation, and Biological Simulation”, F1000Research 5:1946 (2016). [Journal][Preprint]

A Simple Technique to Record Mental Events

Abstract: In recent years, there has been growing interest in bridging bodies of knowledge from introspective and contemplative traditions with modern neuroscience. By making the primary object of study an individual’s subjective experience, scientists are then confronted with the challenging problem of how to record a given mental state at a given point in time. For simple experiences, such as in facial recognition tasks, an external recording device such as a button box or computer keyboard is adequate. However, these devices pose fundamental problems when attempting to record more subtle states of mind, or to do so in situations where the presence of an external input device would disturb the subject, such as in the case of studying the evolution of internal states of highly advanced meditators. Here, I suggest that variants of the mnemonic techniques used in memory competitions can be used to make measurements in experiments where an external device would be impractical to use, or would unnecessarily disturb the subject. Demographic and performance data from memory competitions suggest that these techniques can be learned by ordinary subjects participating in multi-session experiments, and that, for highly trained individuals, events can be recorded at roughly 400 millisecond intervals for timescales on the order of tens of seconds, and at roughly one-second intervals for sustained periods of one hour or longer. Finally, I discuss the relationship of the current proposal to descriptive experience sampling (DES) and the visualization techniques of the Tibetan meditative tradition.

  • Gopal Sarma, “A simple technique to record mental events,” Journal of Consciousness Studies 22(7-8) (2015). [Journal]

Reconsidering Written Language

Abstract: A number of elite thinkers in Europe during the 16th and 17th centuries pursued an agenda which historian Paolo Rossi calls the “quest for a universal language,” a quest which was deeply interwoven with the emergence of the scientific method. From a modern perspective, one of the many surprising aspects of these efforts is that they relied on a diverse array of memorization techniques as foundational elements. In the case of Leibniz’s universal calculus, the ultimate vision was to create a pictorial language that could be learned by anyone in a matter of weeks and which would contain within it a symbolic representation of all domains of contemporary thought, ranging from the natural sciences, to theology, to law. In this brief article, I explore why this agenda might have been appealing to thinkers of this era by examining ancient and modern memory feats. As a thought experiment, I suggest that a society built entirely upon memorization might be less limited than we might otherwise imagine, and furthermore, that cultural norms discouraging the use of written language might have had implications for the development of scientific methodology. Viewed in this light, the efforts of Leibniz and others seem significantly less surprising. I close with some general observations about cross-cultural origins of scientific thought.

  • Gopal Sarma, “Reconsidering Written Language”, Interdisciplinary Description of Complex Systems 13(3) (2015). [Journal][Preprint]

Should We Train Scientific Generalists?

(See ScientificGeneralists.org for additional discussion.)

An explosion of scientific knowledge has led to the current age of specialization. Is it possible that there are negative consequences to the inevitable stratification of knowledge that has resulted from this growth? In the essay below, I argue that to successfully navigate the complexities of the modern scientific establishment and to create a solid foundation for future generations, we need new graduate programs specifically aimed at training a critical mass of scientific generalists. In addition to the specialized knowledge of an ordinary PhD, these programs would train researchers who have substantial exposure to multiple subjects at the graduate level. I explore possibilities for how to structure such programs, and suggest potential roles that generalists might play in the contemporary scientific landscape.

  • Gopal Sarma, “Should We Train Scientific Generalists?”, The Winnower 01(2016). [Journal][Preprint]

The Art of Memory and the Growth of the Scientific Method

Imagine a distant future where the biomedical sciences have grown to subsume nearly the entirety of scientific research. Arriving on campus on the first day of graduate school, you see an old friend from college. “Hey man, what’s going on? And what brings you to this neck of the woods? I thought for sure you would be studying the biochemistry.” “Great to see you too,” your friend says. “Yeah, I really was into the biochemistry in college wasn’t I? But I always had this nagging feeling that I wanted to study something with a more theoretical bent, so I decided to apply for PhD programs in the biophysics instead.”

Would this discussion not sound strange? Why the weird “the” in front of “biochemistry” and “biophysics?”

The thing is, we are using an analogous construction whenever we speak of “the” scientific method. Just as biochemistry and biophysics are specific subcategories of chemistry and physics, several centuries ago, “scientific method” was a subcategory of the broader notion of “method,” and there were other competing ideas about method such as “mnemonic method” and “Lullian method” that represented the forefront of intellectual thought and which were critical scaffolding ideas in the emergence of the modern scientific establishment.

What is even more surprising is that the unifying theme in the diverse notions of “method” were techniques for memorization. Whereas in the modern scientific world, memorization is thought of as being antithetical to conceptual thinking, it was only several centuries ago that leading scientific thinkers, including Renes Descrates, Francis Bacon, and Gottfried Leibniz were pursuing an ambitious agenda to systematize all human knowledge using techniques for memorization as foundational elements.  In the essay below, I describe this vision and the broader cultural milieu of the day, and attempt to discern the role that these ideas may have played in the growth of the scientific method and the emergence of the modern scientific establishment.  Not being a professional historian, I hope that this writing will stimulate the interest of those who are in a position to more fully explore this fascinating aspect of scientific history.

  • Gopal Sarma, “The Art of Memory and the Growth of the Scientific Method”, Interdisciplinary Description of Complex Systems 13(3) (2015). [Journal][Preprint]