As Lewin (1943) already noted, “there is nothing as practical as a good theory”. However, how do we determine which theories are good and which are bad? It is hard to improve theory quality without a tool to assess it in practice. In psychology, most subfields are characterized by weak theories or a complete lack of theories. Even though problems of bad theory have been discussed with clockwork regularity, little progress has been made so far (e.g., Borsboom et al., 2021; Gigerenzer, 1991; Meehl, 1978). A potential reason is that the discipline lacks the tools to assess the quality of theories systematically. Therefore, we (Maier et al., 2021) proposed a computational model for theory evaluation. Specifically, we implement Thagard’s (1989) theory of explanatory coherence (TEC) in the Ising model. The Ising model, originally developed in statistical mechanics to describe the polarization of ferromagnetic materials (Ising, 1925), is a network model that has found broad application in psychological research. We showed that a) hypotheses provided by a scientific theory and phenomena explained by theories can be expressed by the nodes of the Ising model; b) empirical evidence for (against) the phenomena can be expressed by positive (negative) threshold on the phenomena; and c) explanatory and contradictory relations between hypotheses and phenomena can be expressed by positive and negative edges. The Ising Model of Explanatory Coherence (IMEC) incorporates the TEC principles of symmetry, explanation, data, priority, contradiction, and acceptability. Unlike previous implementations of TEC, IMEC allows researchers to evaluate individual theories and is available in an R package. Maier et al. (2021) showed that this simple computational meta-theory could successfully reproduce a variety of examples from the history of science. However, there is room for extension. In this talk, I will briefly introduce IMEC and demonstrate how it integrates considerations of explanatory breadth, refutation, simplicity, and downplaying potentially irrelevant evidence with respect to the hypotheses of the theory and other phenomena. In addition, I will demonstrate how to think through hypothetical scenarios and identify critical experiments using examples of theories in psychological science. Further, I will extend the methodology employed in Maier et al. (2021) by adding sensitivity analyses to IMEC. For instance, by examining the sensitivity of theory evaluations to variations of the edge weights between hypotheses and phenomena, it is possible to improve the robustness of applied theory comparison. However, considerations about the range of possible values under which sensitivity needs to be assessed are fundamentally intertwined with fundamental questions in the philosophy of science, such as the following: How can we quantify (the strength of) evidence (for a phenomenon)? To what extent is a theory supported (refuted) by making a correct (wrong) prediction or explanation? How can we determine the number of elemental propositions that a theory consists of? I hope this talk will spark a debate around these considerations and later allow me to incorporate them in the proposed sensitivity analysis.

In current practice, psychological explanations typically present a narrative in which a theory renders a putative empirical phenomenon intuitively likely. However, whether the theory actually implies the phenomenon in question is also left to this intuition. To design a test for such a theory, different experts iterate through possible experimental setups until they agree that a particular manipulation should show the effect. The fact that this crucial link has to be fleshed out by polling experts, reveals an Achilles’ heel in current psychological theories. Nobody ever had to ask Einstein what would happen to light in the famous eclipse that Eddington observed (Dyson et al., 1920), because Einstein’s opinion was irrelevant. The reason for this is that Einstein’s theory can be and is implemented in a formal model, which means that every competent researcher can check whether the theory does or does imply a given phenomenon. That such independent verification of theoretical implications is not possible in many cases in psychology has direct consequences for the evaluation of the evidence for and against theories. For example, Vohs et al. (2021) suggest that the empirical phenomena associated with the theory of ego-depletion are not robust, as the experimental tasks used did not produce these phenomena. However, it is difficult to gauge whether or not this constitutes evidence against ego-depletion, because in the absence of an unambiguous formalization we cannot even be sure that the theory implies the anticipated phenomena. This points to an important desideratum for explanatory systems, namely that they should be (specific enough to be) encoded in a formal system (e.g., a set of mathematical equations, logical formalisms, or model simulations). We contribute to this task by proposing an account of productive explanation, in which the theory specifies a formal model that produces statistical patterns that reflect empirical phenomena that are purportedly explained by the theory. Expressing the theory in a formal model, and showing how that formal model produces patterns in data, brings transparency to the relation between the theory and the empirical phenomenon. To achieve this aim, we combine insights taken from recent discussions on theory construction (Borsboom et al., 2021; van Rooij and Baggio, 2021) and philosophical considerations (e.g., Cummins, 2000; Haig, 2005) with existing approaches to indirect inference in system dynamics (Haslbeck et al., 2021; Hosseinichimeh et al., 2016) to arrive at a workable methodology for establishing empirical implications. This productive explanation methodology involves a) translating a verbal theory into a set of model equations, b) representing empirical phenomena as statistical patterns in putative data, c) assessing whether the formal model actually produces the targeted phenomenon. In addition, we explicate a number of important criteria for evaluating the goodness of this explanatory relation between theory and empirical phenomenon.

Standard methodological and statistical texts divide research methodology into two strictly separated categories: confirmatory and exploratory research. In some fields, like scientific psychology, almost all research reports are written up as if they are confirmatory, i.e., involve rigorous tests of an antecedent theory. In reality, however, typical research in psychology involves an iterative procedure in which theory is adapted in view of the data, and new data are gathered to further investigate the adequacy of these theory changes. This has led several critics to lament the exploratory aspects of psychological research, as it leads to the possibility of hypothesizing after the facts are known: HARKing. HARKing is a problem because it involves generating hypotheses post hoc, while presenting these as prior to the research project. Thus, it presents research that is exploratory as if it is confirmatory. This practice has been argued to generate an excess of false positive findings, and as such is suspected to lie at the basis of the replication crisis in psychology. Accordingly, in response to that crisis, there has been a rapid surge in the development of methodological tools designed to make the theory testing process more rigorous: from preregistration to blinded data analysis and from many-labs paradigms to reproducibility projects. I argue that this response puts the horse behind the cart, because most psychological research should not be characterized as confirmatory or exploratory, but as aimed at theory construction. This diagnosis has direct implications for the organization of psychological science and the methodological education of psychologists. First, I will argue that even though theory construction has a creative dimension, there is also much logic to the process; as such, the process can be systematized and structured in the same way that we systematize other research processes. This invites the development of techniques and tools that can be used to support theory formation; an example is our recently introduced theory construction methodology, which is a structured series of steps that can be followed to develop theory. Second, theory construction is not covered by standard research methodology and is not taught in psychology curricula. Instead, it is treated as an almost mystical process by which a researcher is supposed to conjure theories out of thin air. However, I argue that theory construction is a skill like any other, and it should be practiced and taught. Third, reports of theory construction research do not fit current reporting standards in scientific publishing, which are almost entirely structured to present either empirical discoveries or tests of scientific theories. Thus, we need new reporting formats to allow such research to be reported truthfully. I will argue that, together, these elements of theory construction define a methodological agenda that has the potential to significantly advance psychological science.

Current approaches to resolving psychology’s theoretical problems converge in their call for the further formalization of psychological theory (e.g., Fried, 2020; Van Rooij & Baggio, 2021; Borsboom et al., 2021; Robinaugh et al., 2021; Guest & Martin, 2021). In contrast, we have argued that psychology’s theoretical problems are in large part caused by issues independent from whether theories are represented verbally, formally or mathematically (e.g., Eronen & Bringmann, 2021; Oude Maatman, 2021). In this talk, we focus on the most fundamental of these issues, which has received little attention in the debate so far: that psychological theory generally is ontologically unspecific. More specifically, it often remains unclear how the constructs and processes described in psychological theories could be realized in the world, or even what the referents of key theoretical concepts are – despite their being treated as real causes. Concepts and constructs are often defined either operationally (e.g., intelligence; ego depletion; Lurquin & Miyake, 2017), functionally (e.g., creativity; Runco & Jaeger, 2012; implicit attitudes; Greenwald & Banaji, 1995; Greenwald & Lai, 2020) or by simply adopting their lexical, folk psychological definition (e.g., in emotion research; Fiske, 2020). Furthermore, psychological theorizing is often completely independent from any foundational theory of the nature of human cognition or approach to the mind-body problem, instead consisting of positing folk-psychologically intuitive causal relationships and mechanisms with few further constraints (see also Danziger, 1997). It thus often remains unclear what the exact ontological commitments of psychological theories are in terms of what particular entities or processes they posit to exist, how hypothesized causal relationships among them are assumed to be realized, or how they fit into a scientific picture of human cognition as a whole. In our talk, we show that this lack of ontological commitment is highly problematic for scientific practice in psychology (cf. Hochstein, 2019). Without a clear ontology, it becomes impossible to delimit the set of causally relevant variables for any to be explained process or phenomenon. Yet, without such delimitation one cannot determine under which conditions an effect or phenomenon should occur or not, which heavily complicates the design of experiments, the interpretation of (non-)replications, and any claims about the generalizability of experimentally identified effects (e.g., Cartwright, 2009). Such delimitation is also necessary to create theory-derived models for prediction or causal inference; if relevant causes are not included, these after all would fail. Without a clear ontology and well-delineated referents for concepts, one also cannot argue or determine whether psychological interventions only affect the intended concept (i.e., the problem of fat-handedness; Eronen, 2020) or that conceptually similar experiments or measurement techniques indeed tap into the same phenomenon or construct. Despite its potential benefits, formalization cannot resolve these issues; the only solution lies in conceptual work, in the form of creating or adopting an ontology. Given the broad scope of the aforementioned problems, we conclude that theory construction (method) in psychology needs to engage with ontology and ontological commitments if psychological science is to advance.

In this paper, we consult the philosophical literature to improve suboptimal practices in psychology. We discuss several practices in psychology that in our view hamper its scientific development. We then argue that these practices are rooted in certain methodological and philosophical commitments. We suggest that psychological science updates some of these commitments with more recent debates in the philosophy of science, in particular those on theory formation, epistemic iteration and the use of models. Our primary concern is with three research practices in psychology, which we term epistemic freezing, empirical myopia, and data fixation. We discuss these practices and identify a common thread of logical empiricism and hypothetico-deductivism among them. First, many concepts in psychology are operationalized by standardized instruments and then get stuck in their operationalization, resisting changes in theories about these concepts. For example, if one compares current intelligence tests to the original setup of e.g., Wechsler in 1955, or one compares the current version of the Beck Depression Inventory to the original from 1961, changes are only marginal and rarely informed by theoretical advances. We call this “epistemic freezing”, as opposed to “epistemic iteration” which refers to the idea that measurement and theories about the measured attribute iteratively improve each other (Chang, 2004). A possible explanation for this tendency to “freeze” concepts is that this would give psychological science a shared empirical basis. Second, it is common practice to stipulate hypotheses before collecting data, and then proceed with confirmatory testing of these hypotheses. Most of the research methodology concerns the testing of a given hypothesis and ignores the research part in which ideas are generated and theories are built, so that testable hypotheses can be formulated. We call this singular focus on hypothesis testing “empirical myopia”, because by focusing only on the testing part, psychology loses sight of the more speculative and exploratory process of theory construction. This practice clearly reflects the confirmatory practice of science along hypothetico-deductivist lines. Third, in psychology, observation has become almost synonymous to ‘data’. For example, in methodology textbooks, theories are said to predict and explain data, where explanation is more or less synonymous to accounting for variance of a dependent variable. In addition, for each new hypothesis to test, one should collect new data. We call this practice “data fixation”, as the focus is on explaining data as opposed to phenomena. Again, we recognize an empiricist streak: the basis for our claim to knowledge is to be found in observed data, and anything that moves us away from direct contact with these observations presumably weakens this basis. Summing up, our diagnosis is that many current practices in psychology are still committed to logical empiricism and hypothetico-deductivism. To help psychological science move away from its somewhat outdated philosophical and methodological commitments, we suggest that it re-evaluates the role of psychological theory. Next to the methodological norms that govern data handling and hypotheses testing, psychological science is in need of norms for the construction and use of theory.