This paper argues that there are two different types of causes that we can wish to understand when we talk about wanting machine learning models to be explainable. The first are causes in the features that a model uses to make its predictions. The second are causes in the world that have enabled those features to carry out the model’s predictive function. I argue that this difference should be seen as giving rise to two distinct types of explanation and explainability and show how the proposed distinction proves useful in a number of applications.

The volume and variety of data in astrophysics creates a need for efficient heuristics to automate the discovery of novel phenomena. Moreover, data-driven practices suggest a role for machine-led exploration in conceptual development. I argue that philosophical accounts of exploratory experiments should be amended to include the characteristics of cases involving machine learning, such as the use of automation to vary experimental parameters and the prevalence of idealized and abstracted representations of data. I consider a case study that applies machine learning to develop a novel galaxy classification scheme from a dataset of ‘low-level’ but idealized observables.

I draw attention to an under-theorized problem for the application of machine learning models in science, which I call semantic opacity. Semantic opacity occurs when the knowledge needed to translate the output of an unsupervised model into scientific concepts depends on theoretical assumptions about the same domain of inquiry into which the model purports to grant insight. Semantic opacity is especially likely to occur in exploratory contexts, wherein experimentation is not strongly guided by theory. I argue that techniques in explainable AI (XAI) that aim to make these models more interpretable are not well suited to address semantic opacity.

Philosophical concern with epistemological challenges presented by opacity in deep neural networks does not align with the recent boom in optimism for AI in science and recent scientific breakthroughs driven by AI methods. I argue that the disconnect between philosophical pessimism and scientific optimism is driven by a failure to examine how AI is actually used in science. I argue, using cases from the scientific literature, that examination of the role played by deep learning as part of a wider process of discovery shows that epistemic opacity need not diminish AI’s capacity to lead scientists to significant and justifiable breakthroughs.