Potential of machine learning to predict early ischemic events after carotid endarterectomy or stenting: a comparison with surgeon predictions

17

1. Brott TG, Howard G, Roubin GS, Meschia JF, Mackey A, Brooks W, et al (2016) Long-Term

18

Results of Stenting versus Endarterectomy for Carotid-Artery Stenosis. N Engl J Med

18

374:1021-1031. doi:10.1056/NEJMoa1505215

2. Buchlak QD, Esmaili N, Leveque JC, Farrokhi F, Bennett C, Piccardi M, et al (2019) Machine

learning applications to clinical decision support in neurosurgery: an artificial intelligence

augmented systematic review. Neurosurg Rev. doi:10.1007/s10143-019-01163-8

3. Chen T, Guestrin C (2016) XGBoost: A Scalable Tree Boosting System. Paper presented at the

Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery

and Data Mining,

10

11

4. Choi JC, Johnston SC, Kim AS (2015) Early outcomes after carotid artery stenting compared

with

endarterectomy

for

asymptomatic

carotid

stenosis.

Stroke

46:120-125.

doi:10.1161/STROKEAHA.114.006209

5. Dua A, Romanelli M, Upchurch GR, Jr., Pan J, Hood D, Hodgson KJ, et al (2016) Predictors

12

of

poor

outcome

after

13

doi:10.1016/j.jvs.2016.03.428

carotid

intervention.

Vasc

Surg

64:663-670.

14

6. Favre JP, Nourissat A, Duprey A, Nourissat G, Albertini JN, Becquemin JP (2008)

15

Endovascular treatment for carotid artery stenosis after neck irradiation. J Vasc Surg 48:852-

16

858. doi:10.1016/j.jvs.2008.05.069

17

7. Ferguson GG, Eliasziw M, Barr HW, Clagett GP, Barnes RW, Wallace MC, et al (1999) The

18

North American Symptomatic Carotid Endarterectomy Trial Surgical Results in 1415 Patients.

19

Stroke 30:1751-1758. doi: 10.1161/01.str.30.9.1751

8. Figueroa RL, Zeng-Treitler Q, Kandula S, Ngo LH (2012) Predicting sample size required for

classification performance. BMC Med Inform Decis Mak 12:8. doi: 10.1186/1472-6947-12-8

9. Friedell ML, Clark JM, Graham DA, Isley MR, Zhang XF (2008) Cerebral oximetry does not

correlate with electroencephalography and somatosensory evoked potentials in determining

the need for shunting during carotid endarterectomy. J Vasc Surg 48:601-606.

doi:10.1016/j.jvs.2008.04.065

10. Gurm HS, Yadav JS, Fayad P, Katzen BT, Mishkel GJ, Bajwa TK, et al (2008) Long-term

results of carotid stenting versus endarterectomy in high-risk patients. N Engl J Med 358:1572-

10

1579.

11

11. Halm EA, Tuhrim S, Wang JJ, Rockman C, Riles TS, Chassin MR (2009) Risk factors for

12

perioperative death and stroke after carotid endarterectomy: results of the new york carotid

13

artery surgery study. Stroke 40:221-229. doi:10.1161/STROKEAHA.108.524785

14

12. Heo J, Yoon JG, Park H, Kim YD, Nam HS, Heo JH (2019) Machine Learning-Based Model

15

for

Prediction

of

Outcomes

16

doi:10.1161/STROKEAHA.118.024293

in

Acute

Stroke.

Stroke

50:1263-1265.

17

13. Kernan WN, Ovbiagele B, Black HR, Bravata DM, Chimowitz MI, Ezekowitz MD, et al

18

(2014) Guidelines for the prevention of stroke in patients with stroke and transient ischemic

20

attack:

Association/American

doi:10.1161/STR.0000000000000024

guideline

for

healthcare

Stroke

professionals

Association.

from

the

Stroke

American

Heart

45:2160-2236.

14. Koch M (2018) Artificial Intelligence Is Becoming Natural. Cell 173:531-533.

doi:10.1016/j.cell.2018.04.007

15. Lammers RL, Hudson DL, Seaman ME (2003) Prediction of traumatic wound infection with

neural

network-derived

doi:10.1053/ajem.2003.50026

decision

model.

Am

Emerg

Med

21:1-7.

16. Li L, Chen Y, Shen Z, Zhang X, Sang J, Ding Y, et al (2020) Convolutional neural network

10

for the diagnosis of early gastric cancer based on magnifying narrow band imaging. Gastric

11

Cancer 23:126-132. doi:10.1007/s10120-019-00992-2

12

17. Maekawa K, Shibata M, Nakajima H, Kitano Y, Seguchi M, Kobayashi K, et al (2018)

13

Cholesterol Crystals in Embolic Debris are Associated with Postoperative Cerebral Embolism

14

after Carotid Artery Stenting. Cerebrovasc Dis 46:242-248. doi:10.1159/000495795

15

18. Mantese VA, Timaran CH, Chiu D, Begg RJ, Brott TG, Investigators C (2010) The Carotid

16

Revascularization Endarterectomy versus Stenting Trial (CREST): stenting versus carotid

17

endarterectomy

18

doi:10.1161/STROKEAHA.110.595330

for

carotid

disease.

21

Stroke

41:S31-34.

19. Moons KG, Altman DG, Reitsma JB, Ioannidis JP, Macaskill P, Steyerberg EW, et al (2015)

Transparent Reporting of a multivariable prediction model for Individual Prognosis or

Diagnosis (TRIPOD): explanation and elaboration. Ann Intern Med 162:W1-73.

doi:10.7326/M14-0698

20. Moresoli P, Habib B, Reynier P, Secrest MH, Eisenberg MJ, Filion KB (2017) Carotid Stenting

Versus Endarterectomy for Asymptomatic Carotid Artery Stenosis: A Systematic Review and

Meta-Analysis. Stroke 48:2150-2157. doi:10.1161/STROKEAHA.117.016824

21. Muller MD, von Felten S, Algra A, Becquemin JP, Brown M, Bulbulia R, et al (2018)

Immediate and Delayed Procedural Stroke or Death in Stenting Versus Endarterectomy for

10

Symptomatic

Carotid

11

doi:10.1161/STROKEAHA.118.020684

Stenosis.

Stroke

49:2715-2722.

12

22. Oermann EK, Rubinsteyn A, Ding D, Mascitelli J, Starke RM, Bederson JB, et al (2016)

13

Using a Machine Learning Approach to Predict Outcomes after Radiosurgery for Cerebral

14

Arteriovenous Malformations. Sci Rep 6:21161. doi:10.1038/srep21161

15

23. Rosenfield K, Matsumura JS, Chaturvedi S, Riles T, Ansel GM, Metzger DC, et al (2016)

16

Randomized Trial of Stent versus Surgery for Asymptomatic Carotid Stenosis. N Engl J Med

17

374:1011-1020. doi:10.1056/NEJMoa1515706

18

24. Setacci C, Chisci E, Setacci F, Iacoponi F, de Donato G, Rossi A (2010) Siena carotid artery

22

stenting score: a risk modelling study for individual patients. Stroke 41:1259-1265.

doi:10.1161/STROKEAHA.110.578583

25. Skjelland M, Krohg-Sørensen K, Tennøe B, Bakke SJ, Brucher R, Russell D (2009) Cerebral

microemboli and brain injury during carotid artery endarterectomy and stenting. Stroke

40:230-234. doi:10.1161/STROKEAHA.107.513341

26. Taussky P, Hanel RA, Meyer FB (2011) Clinical considerations in the management of

asymptomatic

doi:10.3171/2011.9.FOCUS11222

10

carotid

artery

stenosis.

Neurosurg

Focus

31:E7.

27. The Japan Stroke Society SGC (2015) Japanese guidelines for the management of stroke 2015.

Kyowa Kikaku, Tokyo

11

28. Theiss W, Hermanek P, Mathias K, Bruckmann H, Dembski J, Hoffmann FJ, et al (2008)

12

Predictors of death and stroke after carotid angioplasty and stenting: a subgroup analysis of

13

the Pro-CAS data. Stroke 39:2325-2330. doi:10.1161/STROKEAHA.108.514356

14

29. Traenka C, Engelter ST, Brown MM, Dobson J, Frost C, Bonati LH (2019) Silent brain infarcts

15

on diffusion-weighted imaging after carotid revascularisation: A surrogate outcome measure

16

for procedural stroke? A systematic review and meta-analysis. Eur Stroke J 4:127-143.

17

doi:10.1177/2396987318824491

18

30. Troyanskaya O, Cantor M, Sherlock G, Brown P, Hastie T, Tibshirani R, et al (2001) Missing

23

value estimation methods for DNA microarrays. Bioinformatics 17:520-525

31. Tu JV, Wang H, Bowyer B, Green L, Fang J, Kucey D (2003) Risk factors for death or stroke

after carotid endarterectomy: observations from the Ontario Carotid Endarterectomy Registry.

Stroke 34:2568-2573. doi:10.1161/01.STR.0000092491.45227.0F

32. van Os HJA, Ramos LA, Hilbert A, van Leeuwen M, van Walderveen MAA, Kruyt ND, et al

(2018) Predicting Outcome of Endovascular Treatment for Acute Ischemic Stroke: Potential

Value of Machine Learning Algorithms. Front Neurol 9:784. doi:10.3389/fneur.2018.00784

33. Xie Y, Jiang B, Gong E, Li Y, Zhu G, Michel P, et al (2019) Use of Gradient Boosting Machine

Learning to Predict Patient Outcome in Acute Ischemic Stroke on the Basis of Imaging,

10

Demographic,

and

Clinical

11

doi:10.2214/AJR.18.20260

Information.

AJR

Am

Roentgenol

212:44-51.

12

34. Ye H, Gao F, Yin Y, Guo D, Zhao P, Lu Y, et al (2019) Precise diagnosis of intracranial

13

hemorrhage and subtypes using a three-dimensional joint convolutional and recurrent neural

14

network. Eur Radiol 29:6191-6201. doi:10.1007/s00330-019-06163-2

15

35. Yoo Y (2019) Hyperparameter optimization of deep neural network using univariate dynamic

16

encoding

17

doi:10.1016/j.knosys.2019.04.019

18

algorithm

for

searches.

Knowledge-Based

Systems

178:74-83.

36. Zhou W, Baughman BD, Soman S, Wintermark M, Lazzeroni LC, Hitchner E, et al (2017)

24

Volume of subclinical embolic infarct correlates to long-term cognitive changes after carotid

revascularization. J Vasc Surg 65:686-694. doi:10.1016/j.jvs.2016.09.057

Figure Captions

Fig. 1 Flow diagram describing the general framework of the study. Models were built using the

training dataset. The test dataset was used for measuring the predictive performance and

comparison with the surgeons. CAS, carotid artery stenting; CEA, carotid endarterectomy; ICA-

PSV, internal carotid artery peak systolic velocity; kNN, k-nearest-neighbours; LDL, low density

lipoprotein; PPV, positive predictive value

10

11

Fig. 2 Importance values of the clinical factors measured using the total gain of the XGBoost

12

algorithm. CAS, carotid artery stenting; CEA carotid endarterectomy; DM, diabetes mellitus;

13

ICA-PSV, internal carotid artery peak systolic velocity; LDL, low density lipoprotein; mRS,

14

modified Rankin scale

15

16

Supplementary Information

17

Supplemental Table 1 Patient characteristics of CEA group and CAS group.

18

Supplemental Table 2 Optimized hyperparameters of five machine learning models.

25

Table 1. Patient characteristics

Before

imputation

Variable

After imputation

Total

(n=165)

Training

data

(n=143)

Test data

(n=22)

p value

74.2 (7.61)

73.8 (7.82)

76.7 (5.62)

0.04

141 (85)

122 (85)

19 (86)

0 [0-1]

0 [0-1]

0.5 [0-1.75]

0.55

90.0 (29.5)

90.4 (30.1)

87.1 (23.3)

0.56

Hypertension

136 (82)

118 (83)

18 (82)

Diabetes mellitus

58 (35)

50 (35)

8 (36)

Acute coronary syndrome

53 (32)

42 (29)

11 (50)

0.08

Peripheral artery disease

28 (17)

24 (17)

4 (18)

0.77

Contralateral occlusion, n (%)

14 (8.5)

11 (7.7)

3 (14)

0.40

Stenosis at a high position, n (%)

13 (7.9)

12 (8.4)

1 (4.5)

Type III Aorta, n (%)

64 (39)

56 (39)

8 (36)

276 (130)

272 (126)

291 (147)

0.56

Mobile plaque, n (%)

19 (12)

15 (10)

4 (18)

0.29

Plaque ulceration, n (%)

39 (24)

36 (25)

3 (14)

0.29

61 (37)

54 (38)

7 (32)

0.64

Previous neck irradiation, n (%)

15 (9.1)

13 (9.1)

2 (9.1)

Symptomatic, n (%)

64 (39)

56 (39)

8 (36)

Crescendo TIA or stroke in evolution, n (%)

10 (6.1)

8 (5.6)

2 (9.1)

0.62

95 (58)

85 (59)

10 (45)

0.25

45 (27)

42 (29)

3 (14)

0.13

Major ischemic stroke

3 (1.8)

2 (1.4)

1 (4.5)

0.36

Minor ischemic stroke

3 (1.8)

3 (2.1)

0 (0)

Asymptomatic DWI hyperintense lesions

39 (24)

37 (26)

2 (9.1)

0.11

Age, year, mean (SD)

Male sex, n (%)

pre-treatment mRS, median [IQR]

LDL-cholesterol, mg/dL, mean (SD)

Prior medical histories, n (%)

Anatomical and pathophysiological features

ICA-PSV, cm/sec, mean (SD)

plaque with hyperintense signal on TOF, n

(%)

Treatment, CEA, n (%)

Outcome, n (%)

Ischemic stroke within 30 days

Follow-up duration, days, mean (SD)

833 (565)

921 (555)

259 (259)

<.0001

CEA = carotid endarterectomy; DWI = diffusion weighted imaging; IQR = interquartile range; LDL = low

density lipoprotein; mRS = modified Rankin scale; ICA-PSV = internal carotid artery-peak systolic velocity;

TIA = transient ischemic attack; TOF = time-of-flight.

Table 2. Prediction results on the training dataset evaluated by repeated 5-fold cross validation and sorted by

ROC AUC

Model

Ensemble model *

XGBoost

Logistic regression

Neural network

Random forest

SVM

ROC AUC

Sensitivity (%)

Specificity (%)

PPV (%)

Accuracy (%)

mean (95%CI)

mean (95%CI)

mean (95%CI)

mean (95%CI)

mean (95%CI)

0.739

15.1

97.4

75.1

72.7

(0.714 - 0.764)

(11.3 – 18.8)

(96.2 – 98.6)

(65.6 – 84.7)

(71.3 – 74.0)

0.719

14.6

94.9

54.5

70.8

(0.692 - 0.746)

(10.3 – 19.0)

(93.1 – 96.7)

(44.6 – 64.3)

(69.1 – 72.4)

0.702

26.5

95.5

71.2

74.8

(0.671 - 0.732)

(22.4 – 30.5)

(94.2 – 96.8)

(62.7 – 79.7)

(73.1 – 76.4)

0.692

29.0

91.2

61.3

72.5

(0.659 - 0.726)

(23.8 – 34.1)

(89.1 – 93.3)

(54.3 – 68.3)

(70.8 – 74.3)

0.683

22.0

98.3

78.9

75.4

(0.653 - 0.712)

(18.0 – 26.0)

(97.4 – 99.2)

(69.2 – 88.7)

(74.0 – 76.8)

0.680

36.2

85.3

51.7

70.6

(0.650 - 0.711)

(32.2 – 40.3)

(83.5 – 87.1)

(46.6 – 56.8)

(68.8 – 72.4)

* Ensemble model is created by using XGBoost, neural network, and logistic regression, which are the three

most highest ROC AUC models.

ROC AUC = area under the receiver operating characteristic curve; PPV = positive predictive value; SVM =

support vector machine.

Table 3. Prediction results on the test dataset evaluated using the bootstrap technique and sorted by accuracy.

Model

XGBoost

Neural network

Ensemble model *

Random forest

SVM

Logistic regression

Surgeons **

Sensitivity (%)

Specificity (%)

PPV (%)

Accuracy (%)

mean (95%CI)

mean (95%CI)

mean (95%CI)

mean (95%CI)

31.9

94.6

47.2

86.2

(30.0 – 33.8)

(94.2 – 94.9)

(44.9 – 49.6)

(85.7 – 86.6)

4.1

95.4

12.4

83.3

(3.3 – 4.8)

(95.3 – 95.5)

(10.4 – 14.4)

(82.9 – 83.8)

31.8

89.6

32.0

82.0

(29.9 – 33.7)

(89.2 – 90.0)

(30.0 – 33.9)

(81.5 – 82.5)

34.0

84.3

25.1

77.7

(32.1 – 35.9)

(83.8 – 84.8)

(23.6 – 26.6)

(77.1 – 78.2)

34.0

79.1

19.9

73.2

(32.1 – 35.9)

(78.6 – 79.7)

(18.7 – 21.1)

(72.6 – 73.7)

34.0

79.0

20.2

73.0

(32.1 –35.9]

(78.4 – 79.6)

(18.9 – 21.5)

(72.4 – 73.6)

41.7

75.0

20.1

70.5

(0 – 92.4)

(60.7 – 89.3)

(0 – 42.8)

(57.9 – 83.0)

* Ensemble model is created by using XGBoost, neural network, and logistic regression, which are the three

most highest ROC AUC models on the training dataset.

** The average of 4 surgeon’s prediction results with 95% confidence interval.

PPV = positive predictive value; SVM = support vector machine.

Supplementary Information

Article title

Potential of machine learning to predict early ischemic events after carotid endarterectomy or stenting: A

comparison with surgeon predictions

Journal name

Neurosurgical Review

Author names

Kazuya Matsuo, Atsushi Fujita, Kohkichi Hosoda, Jun Tanaka, Taichiro Imahori, Taiji Ishii, Masaaki Kohta,

Kazuhiro Tanaka, Yoichi Uozumi, Hidehito Kimura, Takashi Sasayama, Eiji Kohmura.

Affiliation and e-mail address of the corresponding author

Kazuya Matsuo, MD, PhD, Department of Neurosurgery, Kobe University Graduate School of Medicine, Kobe,

Japan. E-mail: kkmatsuo@outlook.jp

Supplemental Table 1. Patient characteristics of CEA group and CAS group.

CEA group

(n=95)

CAS group

(n=70)

p value

74.0 (8.22)

74.5 (6.74)

0.70

0 [0-1]

0 [0-2]

0.41

89.7 (32.7)

90.7 (23.9)

0.83

Hypertension

84 (88)

52 (74)

0.02

Diabetes mellitus

30 (32)

28 (40)

0.32

Arteriosclerotic disease

36 (38)

35 (50)

0.15

Contralateral occlusion, n (%)

9 (9.5)

5 (7.1)

0.78

Stenosis at a high position, n (%)

8 (8.4)

5 (7.1)

Type III Aorta, n (%)

33 (35)

31 (44)

0.26

259 (125)

294 (132)

0.09

Mobile plaque, n (%)

14 (15)

5 (7.1)

0.15

Plaque ulceration, n (%)

33 (35)

6 (8.6)

<0.0001

Plaque with hyperintense signal on TOF, n (%)

43 (45)

18 (26)

0.01

Previous neck irradiation, n (%)

7 (7.4)

8 (11)

0.42

Symptomatic, n (%)

45 (47)

19 (27)

0.01

Crescendo TIA or stroke in evolution, n (%)

3 (3.2)

7 (10)

0.10

20 (21)

26 (37)

0.03

Major ischemic stroke

3 (3.2)

0 (0)

0.27

Minor ischemic stroke

0 (0)

3 (4.3)

0.07

17 (18)

23 (33)

0.04

Variable

Age, year, mean (SD)

pre-treatment mRS, median [IQR]

LDL-cholesterol, mg/dL, mean (SD)

Prior medical histories, n (%)

Anatomical and pathophysiological features

ICA-PSV, cm/sec, mean (SD)

Outcome, n (%)

Ischemic stroke within 30 days

Asymptomatic DWI hyperintense lesions

CAS = carotid artery stenting; CEA = carotid endarterectomy; DWI = diffusion weighted imaging; IQR = interquartile

range; LDL = low density lipoprotein; mRS = modified Rankin scale; ICA-PSV = internal carotid artery-peak systolic

velocity; TIA = transient ischemic attack; TOF = time-of-flight.

Supplemental table 2. Optimized hyperparameters of five machine learning models

Model

XGBoost

Hyperparameters

learning_rate = 0.3

n_estimators = 360

max_depth = 1

min_child_weight = 6.5

gamma = 0.95

subsample = 0.65

colsample_bytree = 0.65

objective = 'binary:logistic'

reg_alpha = 0.001

reg_lambda = 0.1

max_delta_step = 1

early_stopping_rounds = 20

Random forest

criterion = 'entropy'

max_depth = 4

max_features = 4

min_samples_leaf = 0.0001

min_samples_split = 0.0001

n_estimators = 40

Logistic regression

penalty = 'l2'

C = 0.05

solver = 'saga'

SVM

kernel = 'linear'

C = 2.5

coef0 = 0

degree = 1

gamma = 0.0001

probability = True

Neural Network

models.Sequential()

model.add(layers.Dense(16, activation ='relu', input_shape=(17,)))

model.add(layers.Dropout(0.2))

model.add(layers.BatchNormalization())

model.add(layers.Dense(16, activation ='relu'))

model.add(layers.Dropout(0.2))

model.add(layers.BatchNormalization())

model.add(layers.Dense(1, activation ='sigmoid'))

EarlyStopping (patience=20, restore_best_weights=True)

optimizer=Adam (lr=0.01)

batch_size=8

epochs=41

SVM = support vector machine

* A subset of the program code generated for this study is available at GitHub and can be accessed at [BLINDED

FOR REVIEW].

...