Insulin resistance is a major but often underdiagnosed driver of modern metabolic disease: it can precede clinical type 2 diabetes by years and progressively impair vascular health, liver function and energy metabolism. The Homeostasis Model Assessment for Insulin Resistance (HOMA‑IR) models fasting‑state feedback between hepatic glucose production and insulin secretion; epidemiological reviews consider HOMA‑IR > 2.9 indicative of insulin resistance.
Objective measures of body composition complement wearable sensor data. Wearables capture daily physiological behavior, while body composition reveals structural patterns of adiposity. Total body fat percentage (BF%) is a useful baseline, but further biomarkers such as the Android‑to‑Gynoid fat ratio (A/G) and the Visceral‑to‑Subcutaneous fat area ratio (V/S) provide deeper clinical insight: higher A/G and greater visceral fat correlate strongly with insulin resistance. Dual‑Energy X‑Ray Absorptiometry (DXA) remains the gold standard for measuring true body composition, but DXA is expensive, requires clinical infrastructure and involves low‑dose radiation, limiting scalability for routine screening.
The PhotoScan approach
PhotoScan is an investigational deep learning framework that estimates BF%, A/G ratio and V/S ratio directly from standard 2D smartphone photographs. Development and evaluation proceeded in three phases:
-
Pre‑training (UK Biobank, N = 35,323): the team used a UK Biobank subset containing MRI images and DXA ground truth. A ResNet‑50 backbone (initialized with ImageNet weights) was trained to predict composition metrics from 2D frontal and lateral projection images generated from 3D MRI scans. The model fused image features with participant sex, height, weight and BMI through a final dense layer to output probability distributions for the target metrics.
-
Fine‑tuning (PhotoBIA cohort, N = 677): the model was fine‑tuned on real smartphone photos paired with DXA ground truth using 5‑fold cross‑validation. An automated landmark detection pipeline selected optimal frontal and lateral pose frames from 360° participant videos to augment the training data.
-
Validation (MetabolicMosaic cohort, N = 132): an independent cohort from a 30‑week longitudinal trial in San Francisco was used for validation. Participants in this study had paired DXA, PhotoScan, BIA, anthropometrics, 12‑hour fasting blood labs (fasting glucose and insulin, full lipid panel, etc.) and passive continuous Fitbit tracking.
Key results
Body composition accuracy
- In the PhotoBIA cohort (5‑fold cross‑validation), the fine‑tuned PhotoScan model achieved an average mean absolute error (MAE) of 2.15 for BF% prediction. The BIA‑based model’s MAE in the same cohort was 2.91.
- Averaged MAE values for A/G and V/S in PhotoBIA were 0.107 and 0.094, respectively.
- In the independent MetabolicMosaic cohort the MAE values were comparable: BF% 2.13, A/G 0.085 and V/S 0.085. The modest reduction in A/G and V/S error in MetabolicMosaic is partly explained by that cohort’s higher proportion of female participants (67% vs. 57% in PhotoBIA), since women typically have more gynoid, subcutaneous fat which reduces regional ratio variance and prediction error.
Insulin resistance classification
- The researchers tested a gradient boosting classifier on the MetabolicMosaic cohort to identify subjects with insulin resistance, ensuring leak‑free testing and balanced groups by BMI and insulin resistance status.
- Five feature sets were compared: baseline demographics (age, sex, BMI), standard tape anthropometrics, smartwatch BIA, PhotoScan metrics, and clinical DXA scans.
- Performance was evaluated using AUROC (Area Under the Receiver Operating Characteristic) and Net Reclassification Index (NRI). The demographic baseline model achieved an AUROC of 0.692. Adding PhotoScan‑based body composition features (demo + photoscan) improved AUROC to 0.760 and NRI to 0.593 — nearly matching the performance of DXA (AUROC 0.773, NRI 0.748). Adding BIA to demographics produced no improvement in AUROC or NRI, because BIA supplies only BF%, whose feature importance was substantially lower than A/G and V/S in the demo + photoscan model.
Conclusions and future directions
The findings indicate that smartphone‑based optical estimation of body composition is a feasible, scalable approach for cardiometabolic research and risk screening. Clinical DXA remains the most accurate method but lacks scalability; wearable BIA is convenient but limited to basic BF% estimates. PhotoScan offers an intermediate solution by deriving granular, region‑specific body composition metrics from standard smartphone imagery with near‑DXA accuracy.
This work highlights how digital phenotyping can overcome limitations of simple anthropometrics like BMI, which often miss clinically relevant body composition differences. While PhotoScan is still a research prototype, the results suggest a pathway to accessible, non‑invasive screening for insulin resistance risk. Future research aims to integrate multimodal signals — body composition estimation, continuous wearable data, glucose dynamics and blood biomarkers — to support more holistic and accessible metabolic health management.
