Abstract
I draw attention to an under-theorized problem for the application of machine learning models in science, which I call semantic opacity. Semantic opacity occurs when the knowledge needed to translate the output of an unsupervised model into scientific concepts depends on theoretical assumptions about the same domain of inquiry into which the model purports to grant insight. Semantic opacity is especially likely to occur in exploratory contexts, wherein experimentation is not strongly guided by theory. I argue that techniques in explainable AI (XAI) that aim to make these models more interpretable are not well suited to address semantic opacity.
