AI Explainability for Evidence Generation

In summary

• Interpretable and explainable AI are not the same. The former is transparent by design; the latter uses post-hoc methods to approximate the behaviour of an otherwise opaque model.
• A plausible explanation can support scrutiny, but it does not prove that a model is valid, fair or accountable.
• Fairness must be tested across data, populations and outcomes, rather than inferred from explanations alone.
• For evidence generation, trust should rest on a reproducible and risk-proportionate evidence package: context of use, data lineage, verification, validation, subgroup analysis, uncertainty assessment and audit trails.
• Explainability remains valuable as a diagnostic and hypothesis-generation instrument. It should open an investigation, not close it.

Why “black box” persists

The term black box is borrowed from aviation. An aircraft’s flight recorder does not explain why a pilot took a particular action. It records what happened so that, if something goes wrong, investigators can reconstruct the sequence of events from reliable evidence.

Aviation did not become safe because every component could narrate its internal reasoning. Safety emerged from the infrastructure surrounding the aircraft: certification standards, maintenance records, operating limits, redundant systems, simulator training, continuous monitoring and structured investigation. Trust was created through demonstrable assurance, not through an intuitively satisfying explanation.

Artificial intelligence raises a related question. If we cannot directly observe how a model transforms thousands or millions of interacting signals into an output, can we still trust what it produces?

The instinctive response is to ask the model for an explanation. Yet a fluent rationale, a feature ranking or a convincing saliency map can make an output feel understandable without demonstrating that the explanation faithfully represents the process that generated it. Some explanation methods can remain visually plausible even after meaningful changes are made to the underlying model, showing why plausibility and fidelity must not be confused.

NIST draws a similar distinction in its principles for explainable AI. Providing an explanation is not enough: that explanation must be meaningful to its intended audience, accurately reflect the system’s process and remain within clearly defined knowledge limits.

For evidence generation, the more useful question is therefore not simply whether a model can explain its output. It is whether the organisation can demonstrate that the output has been validated for a clearly defined use, on appropriate and representative data, with explicit knowledge of its assumptions and limitations.

The challenge is to move from plausible explanation to demonstrable assurance.

What explanations tell us

Although terminology is not entirely standardised, it is useful to distinguish between an interpretable model and the post-hoc explanation of an opaque one.

An interpretable model is transparent by construction. A linear score, a short decision tree or a concise set of predefined rules can be inspected directly. Its relevant structure is itself the explanation.

Consider a familiar example from preventive cardiology: estimating a patient’s ten-year risk of a cardiovascular event. An interpretable approach might assign points for age, blood pressure, cholesterol, smoking status and diabetes, then convert the total into a risk estimate. The calculation can be followed directly. In this case, the explanation and the model are effectively the same object.

A more complex model might instead process a 12-lead ECG, a retinal image or a longitudinal clinical record. It may identify interactions across thousands of waveform, image or clinical features that were never encoded as explicit rules.

A post-hoc method can then highlight parts of the ECG or assign importance scores to variables in the clinical record. This may help a researcher or clinician investigate the model’s behaviour, but it remains an estimate of that behaviour. The explanation is another analytical output with its own assumptions, instability and potential errors. It is not necessarily a causal mirror of the underlying computation.

This does not make post-hoc explanation worthless. It changes the weight that should be placed on it. A good explanation may help detect an unexpected dependency, reveal possible data leakage or identify a pattern deserving further investigation. What it cannot do on its own is establish that the model is scientifically valid or appropriate for the decision at hand.

The distinction becomes particularly important when explainability is expected to deliver fairness and accountability.

Fairness cannot be inferred from how convincingly a model’s behaviour can be narrated. One of the most widely discussed examples of bias in healthcare algorithms involved a model that used healthcare expenditure as a proxy for healthcare need. Historical differences in access to care meant that Black patients had often generated lower healthcare costs than equally ill white patients. The logic of the model could be understood, but the objective it had been asked to optimise encoded an existing inequality.

The problem became visible by comparing needs and outcomes across patient groups, not by producing a better explanation of the model. This is because unfairness may enter through the population represented in the data, the people excluded from them, the choice of target, an unsuitable proxy, unequal measurement or the way an output is translated into action.

Fairness must therefore be tested empirically throughout the data-to-evidence chain. Relevant subgroup and intersectional analyses should examine performance, calibration, uncertainty and error patterns rather than treating a plausible explanation as proof that the model behaves equitably.

There are nevertheless situations in which interpretability-by-design should be preferred. If a system directly determines access to a scarce resource or inclusion in a programme, procedural transparency may be part of the objective itself. In those cases, being able to understand the rule in advance can be valuable independently of any difference in performance.

Accountability presents a related problem. With an interpretable model, retracing the explanation may mean retracing the actual calculation. With a complex model examined through a post-hoc method, the explanation remains an approximation. Investigating a failure solely through that approximation risks investigating the wrong process.

Modern AI pipelines can make this even harder. A single output may depend on data extraction, harmonisation, cohort selection, language processing, prediction, simulation and statistical analysis. Each component can have different versions, assumptions and failure modes. Combining a series of approximate explanations does not reconstruct the history of the final result. Accountability must instead be grounded in documented process: versioned data and model lineage, predefined analysis plans, recorded transformations and exclusions, validation reports, audit logs, reproducible outputs and explicit responsibility for every stage of the pipeline.

Bias can also arise in the way explanations are received. People tend to interpret a clinically plausible heat map or feature ranking as confirmation that a model reached the right answer for the right reason. Studies of human-AI interaction suggest that explanations can sometimes increase confidence or reliance without consistently improving users’ ability to detect erroneous recommendations.

Selection bias and confirmation bias can then reinforce one another. Incomplete data produce a limited model; the model generates an output; a post-hoc method creates a plausible explanation; and the explanation reassures the reviewer, reducing the scrutiny that might otherwise have exposed the original limitation.

An explanation should therefore be treated as an object of evaluation, not as evidence that evaluation is no longer necessary.

Evidence changes the question

The alternative is not to accept black-box models automatically. For structured data containing a manageable number of meaningful variables, an interpretable model may perform as well as a more complex one. In high-stakes settings, researchers have therefore argued that opaque models should not be used when equally effective interpretable alternatives are available.

That principle does not, however, translate into a requirement to simplify every model regardless of its purpose. In medical imaging, multi-omics, unstructured clinical text and longitudinal health records, relevant signals may be distributed across very large numbers of interacting variables. Restricting the analysis to a directly human-readable form may sometimes prevent the model from capturing relationships that matter.

Complexity must nevertheless earn its place. The relevant principle is not “black box first” or “interpretability at any cost”. It is to use the simplest model that is fit for purpose, require additional complexity to demonstrate incremental value, and apply scrutiny proportional to the model’s influence and the consequences of error.

This becomes especially important in evidence generation, where the central output may not be an individual prediction. It may instead be a cohort-feasibility assessment, a synthetic population, a synthetic comparator arm, an estimate of treatment effect, a subgroup hypothesis or a real-world evidence analysis.

These outputs raise questions that no feature-attribution chart can answer. Is the scientific question correctly defined? Does the target population correspond to the decision being informed? Are causal assumptions explicit? Have confounding, selection and missingness been addressed? Is uncertainty represented adequately? Do conclusions remain stable under alternative specifications? Are clinically relevant relationships preserved in the synthetic data? Does the result apply to the intended population?

For synthetic data and cohorts, fitness for purpose also requires a joint assessment of analytical utility and privacy. A dataset may reproduce aggregate distributions while failing to preserve the relationships required for a particular analysis. Conversely, high fidelity without sufficient protection may leave an unacceptable residual privacy risk. Neither dimension can be inferred from the model’s explanation alone.

The relevant contrast for evidence generation is therefore not simply explainability versus accuracy. It is plausible explanation versus validated evidence fitness.

From explanation to assurance

ICH M15 offers a useful way to structure this transition. The guideline anchors model-informed evidence to a defined question of interest and context of use. It then considers model influence, the consequences of a wrong decision and the resulting level of model risk, connecting that risk to proportionate verification, validation and applicability assessment.

This approach changes the starting point. Rather than asking first whether a model is interpretable, we begin by asking what the output will be used for and what would happen if it were wrong.

A model supporting an exploratory hypothesis does not require the same evidence as one materially influencing a clinical-development or regulatory decision. Likewise, a synthetic dataset used for software testing should not be evaluated in exactly the same way as a synthetic comparator cohort used to support an estimate of treatment effect.

The evidence required should therefore be proportional to context and consequence. At a minimum, the process should define the question, the intended use and the target population; document the origin and transformations of the data; verify the implementation; validate performance against predefined technical and clinical criteria; test robustness, uncertainty and subgroup behaviour; assess privacy and applicability; and record the limitations under which the result should not be used.

This is also where explainability retains genuine value. Explanation methods can surface unexpected dependencies, identify candidate signals, compare behaviour across groups and guide sensitivity analyses. In high-dimensional clinical or biological data, they can direct researchers towards relationships that deserve independent confirmation.

But this is their appropriate role: the explanation generates a hypothesis or focuses an investigation. It does not replace statistical testing, external validation, scientific review or prospective confirmation.

The explanation opens the investigation; it does not close it.

A validation-first position is not an argument against transparency or human oversight. The EU AI Act requires high-risk AI systems to provide deployers with sufficiently transparent information to interpret and use outputs appropriately, including intended purpose, expected performance, known limitations, relevant input characteristics, foreseeable risks and logging capabilities.

Its human-oversight provisions also require authorised reviewers to understand relevant capabilities and limitations, remain alert to automation bias, recognise anomalous behaviour and, where appropriate, disregard, override or stop a system.

These obligations do not necessarily require every internal weight to be narrated in plain language. They require operationally meaningful transparency: what the system is designed to do, how it has been evaluated, where it may fail, and how its outputs should—and should not—be used.

Where this leaves Aindo

At Aindo, we do not treat explainability as an inherent certificate of trust. Nor do we consider model opacity a reason to relax scrutiny. For evidence generation, the primary object of validation is the evidence package itself: the provenance and representativeness of the data, the integrity of the analytical method, the preservation of clinically relevant relationships, performance across relevant subgroups, robustness to alternative assumptions, uncertainty, privacy risk, applicability and reproducibility.

This requires a multi-layered assurance infrastructure. Data, models and methodologies must be versioned. Inclusion, exclusion and transformation rules must be documented. Validation benchmarks should be defined before results are interpreted. Utility and privacy should be assessed against the intended context of use. Assumptions, uncertainty and knowledge limits must be made explicit, and outputs must be supported by reproducible records and audit trails.

Where post-hoc explanations are used, their fidelity and stability should themselves be tested. Where procedural transparency is a central requirement, interpretable-by-design methods should be preferred. Where complex models provide demonstrated incremental value, they should be governed through risk-proportionate verification, validation and applicability assessment.

Our position is therefore straightforward: Validation first. Explanation where useful. Interpretability where required. Auditability throughout.

Returning to aviation, we do not trust an aircraft because it can explain itself. We trust the system because its design, operating limits, maintenance, monitoring and performance have been rigorously assessed—and because, if something goes wrong, the event can be reconstructed from reliable evidence.

AI used for evidence generation should be held to the same underlying standard. The decisive question is not simply whether a model can explain its output. It is whether the organisation can demonstrate, with reproducible evidence, that the output is valid, robust and fit for the purpose for which it will be used.