Medical Informatics

Research Article

DOI 10.25209/2079-3316-2026-17-3-105-126

Approaches to Assessing the Accuracy of Measurements Performed by AI-Services in Radiology

Evgeniya Alexandrovna Krylova¹, Yuriy Alexandrovich Vasilev², Vera Vladimirovna Soboleva³, Irina Andreevna Raznitsyna⁴, Tatiana Mikhailovna Bobrovskaya⁵, Kirill Mikhailovich Arzamasov⁶

1,4-6	Scientific and Practical Clinical Center for Diagnostics and Telemedicine Technologies of the Moscow Department of Health, Moscow, Russia
3	Pirogov Russian National Research Medical University, Moscow, Russia
6	MIREA - Russian Technological University, Moscow, Russia
1	KrylovaEA13@zdrav.mos.ru

Abstract. The importance of quantitative measurements in radiology is increasing, as is the number of artificial intelligence (AI) services being introduced into the field. However, assessment of their measurement accuracy remains insufficiently standardized and relies on statistical metrics that are not always applicable to domain-specific tasks. This study aimed to develop and conduct a preliminary evaluation of an automated method for assessing the quality of quantitative measurements performed by AI-services in radiology, using linear measurements of adrenal lesions on chest CT as an example.
The dataset comprised 90 chest CT examinations with annotations including a binary indicator of lesion presence and its measurement. The approach was evaluated using the outputs of two AI-services. Each measurement was scored according to its proximity to the reference standard (0, 0.5, or 1 point). The overall score was calculated as the mean score, with 0.81 adopted as the threshold for sufficient accuracy. Acceptable deviations were determined empirically: 2 (± 0.2) and 4 (± 0.4) mm defined the boundaries for correct and partially correct measurements, respectively.
The overall score was 0.914 (0.853–0.966) for AI-1 and 0.913 (0.857–0.968) for AI-2, while the intraclass correlation coefficient was 0.94 and 0.93, Gwet’s AC2 was 0.97 and 0.96, respectively. No systematic bias was identified. The proposed approach enabled assessment of individual measurements as well as the overall performance quality of an AI-service and proved applicable for the task of measuring adrenal lesions on chest CT. (In Russian).

Keywords: Artificial intelligence, quality control, radiology, statistical analysis

MSC-2020

68U10; 62P10, 92C50

Acknowledgments: This paper was prepared by a group of authors as a part of the research and development effort titled “Artificial intelligence for synchronous evaluation of images and EHR (multimodal AI)”, in accordance with the Order No. 1213 dated November 27, 2025 "On approval of state assignments funded by means of allocations from the budget of the city of Moscow to the state budgetary (autonomous) institutions subordinate to the Moscow Health Care Department, for 2026 and the planned period of 2027 and 2028" issued by the Moscow Health Care Department.

For citation: Evgeniya A. Krylova, Yuriy A. Vasilev, Vera V. Soboleva, Irina A. Raznitsyna, Tatiana M. Bobrovskaya, Kirill M. Arzamasov. Approaches to Assessing the Accuracy of Measurements Performed by AI-Services in Radiology. Program Systems: Theory and Applications, 2026, 17:3, pp. 105–126. (In Russ.). https://psta.psiras.ru/2026/3_105-126.

Full text of article (PDF): https://psta.psiras.ru/read/psta2026_3_105-126.pdf.

The article was submitted 16.07.2026; approved after reviewing 16.08.2026; accepted for publication 09.09.2026; published online 12.09.2026.