Over the past decade, scientific researchers and applied data scientists have steadily adopted machine learning (ML) techniques, particularly highly parameterized deep neural networks, Deep Learning (DL). Trained estimators resulting from such ML processes, referred to as models, are now commonly used to either better estimate particular unknown features given a particular context or to improve understanding of said features given their respective contexts. Recently philosophical work has investigated the nature of such understanding from ML models. Sullivan (2020) argues that the complexity of DL trained models means that they can be contrasted with the traditional use of idealization models, which ostensibly enable explanation or understanding by reducing complexity. In this work we explore the strength of this contrast, arguing that while the explicit functional form of particular highly parameterized DL trained models can be quite complex, such complexities are irrelevant to gains in explanation or understanding generated by DL models. We observe that framing the form of understanding gained from ML models as in Tamir & Shech (2022) enables an account of understanding from ML models that consequently illuminates both the nuances and failures of this contrast. Specifically, we propose that individual parameter instantiations resulting from ML training of particular models are best understood as approximations of the more general target phenomenon to be understood. We demonstrate that a proper analysis in which the contexts where approximation relationships break down are distinguished from those in which it can be sustained, enables us identify both sort of details irrelevant to understanding and the sort of higher level representations often captured by hidden layers of deep neural networks which may be leveraged for explanation or improved understanding. We show that hindrances to understanding from ML models due to parametrization complexity are analogous to infinite idealization dilemmas found in the philosophy of physics literature (Batterman 2002, Shech 2013). Drawing on Norton’s (2012) distinction between idealizations and approximations, we argue that our resolution of understanding from ML models despite parameterization complexity has important parallels with resolutions of said infinite idealization dilemmas, viz., Butterfield (2011), Norton (2012). We conclude with a unifying framework under which the success of accounts of understanding from highly parameterized ML models as well as understanding from some idealized models (including problematic infinite models) can be properly assessed.

AI systems are being used for a rapidly increasing number of important decisions. Many of these systems are “black boxes”: their functioning is opaque both to the people affected by them and to those developing them. This opacity is often due to the complexity of the model used by the AI system, and to the fact that these models are using machine learning techniques (Burrell 2016, Sullivan 2020). Black box AI systems are difficult to evaluate for accuracy and fairness, seem less trustworthy, and make it more difficult for affected individuals to seek recourse for undesirable decisions. Explainable AI (XAI) methods aim to alleviate the opacity of complex AI systems (Lakkaraju et al. 2020). These methods typically involve approximating the original black box system with a distinct “explanation model”. The original opaque model is used for actual recommendations or decision-making. Then, the explanation model provides an explanation for the original model’s output. However, there is debate about whether such methods can provide adequate explanations for the behavior of black box AI systems. This debate is made difficult by lack of agreement in the literature concerning what it means to give an adequate explanation. I argue that the goal of XAI methods should be to produce explanations that promote understanding for stakeholders. That is, a good explanation of an AI system is one that places relevant stakeholders in a position to understand why the system made a particular decision or recommendation. Moreover, I suggest that XAI methods can achieve this goal because (when things go well) the explanation models they produce serve as idealized representations of the original black box model. An idealization is an aspect of a scientific model that deliberately misrepresents its target to enable better understanding of that target (Elgin 2017). Even though idealizations are false, they can promote understanding by conferring a variety of benefits on a model (Potochnik 2017). An idealized model can be simpler, can leave out unimportant information, and can highlight specific causal patterns that might otherwise be obscured by the complexity of the system being represented. Recognizing that XAI methods produce idealized models can help illuminate how these methods function. This recognition can also guide decisions on when and whether specific methods should be employed. Certain kinds of idealizations will be apt for explaining a particular black box model to a particular audience. This in turn will help determine which XAI methods should be employed for providing those explanations. Whether an idealization is appropriate will depend on what benefits it will confer on an idealized model. For instance, consider feature importance methods that use linear equation models, such as LIME (Ribeiro et al 2016). These XAI methods employ idealizations that confer simplicity and legibility on the resulting explanation model. They eliminate information about causally unimportant features, while highlighting relevant causal patterns that are important for determining the original model’s output. These idealizations serve to promote understanding for non-technical stakeholders affected by an XAI system.

A central problem for machine learning (ML) models is that they are “black boxes” and epistemically opaque. This means the inner workings of these models—how the model internally represents the data to reach a certain decision—are opaque or a “black box” to experts. This is concerning in health-care settings where such models are increasingly being used autonomously for high-stakes decision making. These concerns have led to a growing legal and ethical demand that the ML models be explainable if used in safety-critical domains. Explanations often require describing how the model represents the data or what the machine "sees" when it uses data to make a prediction. However, it is widely accepted that ML models are subject to an inherent and general tradeoff between predictive performance and explainability. The argument for the Tradeoff Thesis is based on model complexity. A more complex model is more accurate because of its complexity: it can train on, represent, and learn from a larger body of complex data. A more complex model (like a neural network) is less explainable because it combines that data using nonlinear functions, over multiple layers, and iteratively updates its outputs to optimize predictive skill. In contrast, a simpler model (like a decision tree) is more explainable in virtue of the rules encoded by human scientists but exhibits poorer predictive performance because of its rigidity. This Tradeoff Thesis reflects a long-standing philosophical position that describes prediction and explanation as two distinct, and often competing, theoretical virtues or epistemic goals. I challenge the Tradeoff Thesis using a case study of two deep learning systems that diagnose eye disease using retinal images. I then use my study of how explanation facilitates improved predictions in medical AI to support Heather Douglas’s (2009) argument for the tight practical and functional relation between prediction and explanation. In a case study, I demonstrate that improvements in the explainability of a deep learning system that uses representations of retinal lesions to detect diabetic retinopathy leads to improvements in predictive skill when compared to earlier studies that used simpler and more opaque models. I argue that the improved explainability facilitates improved predictive performance and that increased complexity is compatible with explainability. Furthermore, I compare explanations of DeepDR and its predictions with those of human ophthalmologists. I show how the explainability of DeepDR is on par with medical explanations provided by human doctors. An important consequence of my findings is that the Tradeoff Thesis must be proven to hold within a circumscribed set of models and cannot be presumed to hold rather generically “for all current and most likely future approaches to using ML for medical decision-making” (Heinrichs & Eickhoff 2020, 1437). Furthermore, this case illustrates how, in practice, prediction and explanation are deeply connected. This poses a challenge for philosophical models which construe the relation between prediction and explanation as one of epistemic rivals. Therefore, complex ML algorithms may still hold promise for reliable and ethical deployment in safety-critical fields like medicine. .

One way machine learning (ML) modeling is different from more traditional modeling methods is that they are data-driven, instead of what Knüsel and Baumberger (2020) call process driven. Moreover, ML models suffer from a higher degree of model opacity compared to more traditional modeling methods. Despite these differences, modelers and philosophers (e.g. Sullivan 2020, Meskhidze 2021) have claimed that ML models can still provide understanding of phenomena. However, before the epistemic consequences of opacity become salient, there is an underexplored prior question of representation. If ML models do not represent their targets in any meaningful sense, how can ML models provide understanding? The problem is that it does in fact seem as though ML models do not represent their targets in any meaningful sense. For example, the similarity view of representation seems to exclude the possibility that ML models can represent phenomena. ML models use methods of finding feature relationships that are highly divorced from their target systems, such as relying on decision-rules and loose correlations instead of causal relationships. Moreover, the data that models are trained on can be manipulated by modelers in a way that reduces similarity. For example, the well-known melanoma detection ML model (Esteva et al. 2017) augments the RBG spectrum of dermatologist images (Tamir and Shech 2022). Thus, if the similarity view is right, then even if model opacity qua opacity does not get in the way of understanding, ML models may still fail to enable understanding of phenomena because they fail to represent phenomena. Contra to the similarity view, I argue that ML models are in fact able to represent phenomena, under specific conditions. Drawing on the literature of how highly idealized models represent their targets, and the interpretative view of representation (Nguyen 2020), a strong case can be made that ML models can accurately represent their targets. Even though ML models seem to be the opposite of highly idealized simple models, there are a number of representational similarities between them. Thus, if we accept that highly idealized models can represent phenomena, then so can ML models. References Knüsel, B., and Baumberger, C. (2020): Understanding climate phenomena with data-driven models. Studies in History and Philosophy of Science Part A, 84, 46-56. Meskhidze, H. (2021). Can Machine Learning Provide Understanding? How Cosmologists Use Machine Learning to Understand Observations of the Universe. Erkenntnis, 1-15. Nguyen, J. (2020). It’s not a game: Accurate representation with toy models. The British Journal for the Philosophy of Science, 71(3), 1013-1041. Sullivan, E.(2020): Understanding from Machine Learning Models. In The British Journal for the Philosophy of Science. DOI: 10.1093/bjps/axz035. Esteva, A.; Kuprel, B.; Novoa, R. A.; Ko, J.; Swetter, S. M.; Blau, H. M.; Thrun, Seb. (2017): Dermatologist-level classification of skin cancer with deep neural networks. In Nature 542 (7639), pp. 115–118. DOI: 10.1038/nature21056.