Deep Learning Electrocardiograms Underestimate Mortality Risk Under Clinical Case-Mix Shift

A deep learning survival model trained on digital electrocardiograms accurately ranked patient mortality risk but severely underestimated absolute death rates when evaluated in a different clinical population, according to an investigation in Frontiers in Cardiovascular Medicine. Led by researchers across institutions in Saudi Arabia, Turkey, and Azerbaijan, the retrospective study evaluated how artificial intelligence forecasts time-to-death across changing clinical environments. The team developed a five-seed ensemble of deep neural networks using 183,689 training electrocardiograms from the Brazilian CODE-15 percent cohort and evaluated it externally on 1,627 patients from the SaMi-Trop study. Although the artificial intelligence system maintained substantial discrimination when ordering patients from lowest to highest risk across internal and external tests, its absolute probability predictions deteriorated under clinical case-mix shift. The researchers demonstrated that relying on relative risk ranking alone can obscure dangerous underestimations of absolute mortality, showing that strong discrimination does not guarantee reliable clinical risk prediction across shifting patient populations.

Cardiovascular diseases remain the primary contributor to global mortality, generating sustained clinical interest in prognostic tools capable of detecting hidden risk during routine checkups before overt symptoms emerge. Standard twelve-lead electrocardiograms provide an accessible, non-invasive recording of cardiac electrophysiology widely gathered across healthcare systems, making them an attractive substrate for automated screening algorithms. While earlier research established that deep neural networks can extract subtle morphological patterns linked to structural heart disease, myocardial infarction, biological age, and mortality, those studies largely evaluated whether models could correctly separate high-risk individuals from low-risk individuals. However, clinical decision-making requires accurate absolute risk probabilities rather than mere ordering metrics, as physicians depend on calibrated survival estimates to recommend therapies or determine follow-up frequency. The investigative team addressed this critical gap by examining how discrimination, demographic integration, probability calibration, and model recalibration behave when an algorithm encounters marked changes in baseline disease burden within shared technical infrastructure.

To establish an evaluation pipeline, the investigators analyzed two large public datasets originating from the Minas Gerais telehealth network in Brazil. The primary development resource was the CODE-15 percent dataset, encompassing 345,779 twelve-lead electrocardiograms sampled at 400 hertz across 4,096 time points per lead. Restricting the cohort to adult patients aged 18 years or older with complete records and non-zero follow-up yielded 230,036 eligible participants. The team partitioned CODE at the patient level into 183,689 training participants with 6,603 recorded deaths, 22,974 validation participants with 828 deaths, and 22,959 internal test participants with 825 deaths, stratifying by mortality status, sex, and four age bands. For external evaluation, the team selected the SaMi-Trop cohort, a longitudinal study of individuals with self-reported Chagas disease and prior electrocardiographic abnormalities. Unlike the general ambulatory population in CODE, SaMi-Trop featured an older, predominantly female demographic with substantially higher baseline mortality and shorter follow-up, providing a defined case-mix shift sharing identical recording hardware and sampling parameters.

Because both datasets were gathered within the same telehealth network during overlapping calendar windows, the authors instituted an exhaustive cross-release audit to verify participant separation before model evaluation. They calculated secure SHA-256 cryptographic hashes across full numerical arrays and padding-trimmed signals, screening all 345,779 CODE recordings against the 1,631 released SaMi-Trop files. The team expanded a normalized cross-correlation screen across 326,200 candidate pairs over half-second lag windows, verifying the detection pipeline against 800 positive-control transformations involving synthetic noise, scaling, and cropping. This audit uncovered four identical waveform recordings masquerading under conflicting external identification numbers across the two public releases. Source metadata revealed that two matching participants existed in the CODE training set and one appeared in the internal test split. The researchers conservatively excluded all four duplicate patients from SaMi-Trop without modifying CODE splits, establishing a final external evaluation cohort of 1,627 participants with 104 deaths, while noting that remaining records cannot be guaranteed fully disjoint across independent encounters.

The primary predictive architecture comprised a discrete-time survival ensemble utilizing a one-dimensional residual convolutional neural network and demographic embeddings. The convolutional trunk processed raw twelve-lead voltages through six residual blocks with channel expansion up to 256 dimensions, applying Gaussian error linear unit activations and adaptive global average pooling. This representation was concatenated with a 128-dimensional multilayer perceptron embedding derived from standardized age and binary sex, yielding a 384-dimensional fused latent vector containing 2,368,656 trainable parameters. Rather than imposing proportional hazards assumptions like continuous Cox regression, the network output sixteen discrete half-year interval hazard logits covering an eight-year follow-up grid. The team trained five independent seeds using AdamW optimization on Apple Metal Performance Shaders, averaging interval hazards across frozen models to construct smooth cumulative survival curves. Parallel comparator models were evaluated alongside the primary concatenation network, including an electrocardiogram-only residual network, an age-and-sex neural baseline, an adaptive gated fusion ablation, and training-derived linear and spline Cox reference models.

Evaluation on the internal test cohort confirmed strong predictive performance for the primary concatenation model, achieving a Harrell concordance index of 0.842 and time-dependent area under the receiver operating characteristic curve values of 0.862 at one year and 0.844 at two years. When evaluated on the external SaMi-Trop cohort, the primary model maintained respectable ranking ability, yielding a concordance index of 0.797, a one-year receiver operating characteristic curve of 0.818, and a two-year curve of 0.800. However, the comparative contribution of demographic information shifted dramatically between cohorts. Internally, fusing age and sex with electrocardiograms produced a distinct concordance improvement of 0.0256 over the waveform-only architecture. Externally, this incremental benefit dropped to an unresolved 0.0109 with a ninety-five percent bootstrap confidence interval spanning negative 0.0246 to positive 0.0489. Similar statistical uncertainty characterized one-year and two-year discrimination gains, demonstrating that adding patient demographics to complex neural representations provided robust internal gains that failed to achieve statistical resolution under external case-mix shift.

The investigation revealed substantial deterioration in absolute probability accuracy despite preserved external discrimination. In the external cohort at two years, the primary neural model predicted an average mortality risk of 3.21 percent, whereas observed Kaplan-Meier mortality was 6.27 percent, representing a near two-fold underestimation of actual deaths. The two-year calibration-in-the-large metric reached 0.703, where positive values denote systematic risk underprediction. At one year, the algorithm predicted an average risk of 1.81 percent compared to an observed mortality rate of 2.83 percent, generating a calibration-in-the-large value of 0.474. In decile calibration analyses, the highest predicted risk group in SaMi-Trop exhibited a predicted two-year mortality of 11.53 percent against an observed mortality of 25.92 percent, producing a 14.38 percentage point underprediction gap. The authors warned that using such uncalibrated absolute probabilities in clinical practice could provide false reassurance to patients facing high mortality hazards, potentially leading clinicians to withhold essential therapeutic monitoring.

To determine whether this miscalibration could be mitigated without architectural retraining, the authors evaluated fivefold out-of-fold recalibration directly within SaMi-Trop. Adjusting the baseline risk intercept alone increased the mean two-year predicted risk to 6.07 percent, reducing the Brier prediction error score from 0.05558 to 0.05337 and increasing the index of prediction accuracy from 5.42 percent to 9.18 percent. Jointly recalibrating both the intercept and the slope yielded a comparable Brier reduction to 0.05324, representing an absolute error decline of 0.00234 and raising the prediction accuracy index to 9.40 percent. However, at the one-year evaluation horizon, recalibration produced smaller and statistically unresolved improvements in Brier accuracy. The authors emphasized that these out-of-fold adjustments quantify within-cohort probability correction rather than independently validated clinical transportability. Because the recalibration parameters were derived within the same relatively small Chagas cohort through cross-validation, the findings confirm that local baseline calibration can improve mathematical accuracy while cautioning that clinical utility requires testing across distinct external healthcare systems.

The researchers probed the internal representations of the frozen ensemble through patient-bootstrap sensitivity analyses, single-lead occlusion, and numerically verified integrated gradients across 256 matched participants per cohort. Single-lead occlusion demonstrated a high cross-cohort rank correlation of 0.979, with lead I identified as the most influential lead in both populations. However, the absolute relative importance of lead I dropped from 0.213 internally to 0.166 externally, demonstrating that stable relative rankings can conceal significant shifts in input sensitivity. Evaluating integrated gradients across 16 to 512 integration steps confirmed numerical convergence, yielding a rank correlation of 0.944 while meeting strict completeness residual criteria. Grouped masking revealed that suppressing all limb leads simultaneously altered two-year predicted mortality by a mean absolute value of 0.01696 internally and 0.01798 externally, whereas precordial masking produced smaller shifts. The authors noted that because peripheral limb leads share mathematical linear dependencies, individual lead attributions reflect algorithmic sensitivity rather than invariant, identifiable physiological mechanisms of disease.

The study authors highlighted several structural limitations that constrain broader clinical generalization of their findings. Although SaMi-Trop introduced significant clinical shift through its Chagas-enriched, older population, both cohorts shared the identical Brazilian telehealth acquisition network and national mortality linkage infrastructure. Consequently, the study evaluated case-mix variation rather than true transportability across distinct international healthcare institutions, novel digital recording devices, or unselected populations. Furthermore, the external cohort included only 104 observed deaths across 1,627 individuals, creating wide confidence intervals around incremental gains and recalibration metrics. Because the primary neural network possesses over 2.3 million parameters relative to 6,603 training events, subtle developmental overfitting remains plausible despite validation stopping criteria. The authors concluded that future clinical deployment of artificial intelligence electrocardiogram models demands rigorous evaluation of absolute calibration alongside discrimination metrics, separate derivation and testing cohorts for recalibration models, and prospective clinical validation before automated survival predictions can safely guide cardiovascular patient management.

Subject of Research: Deep learning survival prediction from electrocardiograms under case-mix shift

Article Title: Deep ECG survival prediction for all-cause mortality under case-mix shift: discrimination, risk underestimation, and recalibration

Article References: Ganie, S. M., Burukanli, M., Turkmen, H., & Pacal, I. (2026). Deep ECG survival prediction for all-cause mortality under case-mix shift: discrimination, risk underestimation, and recalibration. Frontiers in Cardiovascular Medicine, 13, 1993391. https://doi.org/10.3389/fcvm.2026.1993391

Image Credits: AI Generated

DOI: 10.3389/fcvm.2026.1993391

Keywords: deep learning, electrocardiography, survival analysis, case-mix shift, mortality prediction, risk calibration, calibration-in-the-large, Brier score, Chagas disease, CODE-15% cohort, SaMi-Trop cohort, discrete-time survival, Harrell concordance index, integrated gradients, recalibration, waveform hashing, cardiovascular prognosis, external evaluation

Tags:

 

Share Story:

Facebook
X
LinkedIn