Clinical Prediction Models

Preface vii<div>Acknowledgements xi</div><div>Chapter 1 Introduction 1</div><div>1.1 Diagnosis, prognosis and therapy choice in medicine 1</div><div>1.1.1 Predictions for personalized evidence-based medicine 1</div><div>1.2 Statistical modeling for prediction 5</div><div>1.2.1 Model assumptions 5</div><div>1.2.2 Reliability of predictions: aleatory and epistemic uncertainty 6</div><div>1.2.3 Sample size 6</div>1.3 Structure of the book 8<div>1.3.1 Part I: Prediction models in medicine 8</div><div>1.3.2 Part II: Developing internally valid prediction models 8</div><div>1.3.3 Part III: Generalizability of prediction models 9</div><div>1.3.4 Part IV: Applications 9</div><div>Part I: Prediction models in medicine 11</div><div>Chapter 2 Applications of prediction models 13</div><div>2.1 Applications: medical practice and research 13</div><div>2.2 Prediction models for Public Health 14</div>2.2.1 Targeting of preventive interventions 14<div>*2.2.2 Example: prediction for breast cancer 14</div><div>2.3 Prediction models for clinical practice 17</div><div>2.3.1 Decision support on test ordering 17</div><div>*2.3.2 Example: predicting renal artery stenosis 17</div><div>2.3.3 Starting treatment: the treatment threshold 20</div><div>*2.3.4 Example: probability of deep venous thrombosis 20</div><div>2.3.5 Intensity of treatment 21</div><div>*2.3.6 Example: defining a poor prognosis subgroup in cancer 22</div><div>2.3.7 Cost-effectiveness of treatment 23</div><div>2.3.8 Delaying treatment 23</div><div>*2.3.9 Example: spontaneous pregnancy chances 24</div><div>2.3.10 Surgical decision-making 26</div><div>*2.3.11 Example: replacement of risky heart valves 27</div><div>2.4 Prediction models for medical research 28</div><div>2.4.1 Inclusion and stratification in a RCT 28</div><div>*2.4.2 Example: selection for TBI trials 29</div><div>2.4.3 Covariate adjustment in a RCT 30</div><div>2.4.4 Gain in power by covariate adjustment 31</div><div>*2.4.5 Example: analysis of the GUSTO-III trial 32</div><div>2.4.6 Prediction models and observational studies 32</div><div>2.4.7 Propensity scores 33</div>*2.4.8 Example: statin treatment effects 34<div>2.4.9 Provider comparisons 35</div><div>*2.4.10 Example: ranking cardiac outcome 35</div><div>2.5 Concluding remarks 35</div><div>Chapter 3 Study design for prediction modeling 37</div><div>3.1 Studies for prognosis 37</div><div>3.1.1 Retrospective designs 37</div><div>*3.1.2 Example: predicting early mortality in esophageal cancer 37</div><div>3.1.3 Prospective designs 38</div>*3.1.4 Example: predicting long-term mortality in esophageal cancer 39<div>3.1.5 Registry data 39</div><div>*3.1.6 Example: surgical mortality in esophageal cancer 39</div><div>3.1.7 Nested case-control studies 40</div><div>*3.1.8 Example: perioperative mortality in major vascular surgery 40</div><div>3.2 Studies for diagnosis 41</div><div>3.2.1 Cross-sectional study design and multivariable modeling 41</div><div>*3.2.2 Example: diagnosing renal artery stenosis 41</div><div>3.2.3 Case-control studies 41</div><div>*3.2.4 Example: diagnosing acute appendicitis 42</div><div>3.3 Predictors and outcome 42</div><div>3.3.1 Strength of predictors 42</div><div>3.3.2 Categories of predictors 42</div><div>3.3.3 Costs of predictors 43</div><div>3.3.4 Determinants of prognosis 44</div><div>3.3.5 Prognosis in oncology 44</div><div>3.4 Reliability of predictors 45</div><div>3.4.1 Observer variability 45</div><div>*3.4.2 Example: histology in Barrett’s esophagus 45</div><div>3.4.3 Biological variability 46</div><div>3.4.4 Regression dilution bias 46</div><div>*3.4.5 Example: simulation study on reliability of a binary predictor 46</div><div>3.4.6 Choice of predictors 47</div><div>3.5 Outcome 47</div><div>3.5.1 Types of outcome 47</div><div>3.5.2 Survival endpoints 48</div><div>*3.5.3 Examples: 5-year relative survival in cancer registries 48</div><div>3.5.4 Composite endpoints 49</div><div>*3.5.5 Example: composite endpoints in cardiology 49</div><div>3.5.6 Choice of prognostic outcome 49</div><div>3.5.7 Diagnostic endpoints 49</div><div>*3.5.8 Example: PET scans in esophageal cancer 50</div><div>3.6 Phases of biomarker development 50</div><div>3.7 Statistical power and reliable estimation 51</div><div>3.7.1 Sample size to identify predictor effects 51</div><div>3.7.2 Sample size for reliable modeling 53</div><div>3.7.3 Sample size for reliable validation 55</div><div>3.8 Concluding remarks 55</div><div>Chapter 4 Statistical models for prediction 57</div><div>4.1 Continuous outcomes 57</div><div>*4.1.1 Examples of linear regression 58</div><div>4.1.2 Economic outcomes 58</div><div>*4.1.3 Example: prediction of costs 58</div><div>4.1.4 Transforming the outcome 58</div><div>4.1.5 Performance: explained variation 59</div><div>4.1.6 More flexible approaches 60</div><div>4.2 Binary outcomes 61</div><div>4.2.1 R2 in logistic regression analysis 62</div><div>4.2.2 Calculation of R2 on the log likelihood scale 63</div><div>4.2.3 Models related to logistic regression 65</div><div>4.2.4 Bayes rule 65</div><div>4.2.5 Prediction with Naïve Bayes 66</div><div>4.2.6 Calibration and Naïve Bayes 67</div><div>*4.2.7 Logistic regression and Bayes 67</div><div>4.2.8 Machine learning: more flexible approaches 68</div><div>4.2.9 Classification and regression trees 69</div><div>*4.2.10 Example: mortality in acute MI patients 69</div><div>4.2.11 Advantages and disadvantages of tree models 70</div><div>4.2.12 Trees versus logistic regression modeling 70</div><div>*4.2.13 Other methods for binary outcomes 71</div><div>4.2.14 Summary on binary outcomes 72</div><div>4.3 Categorical outcomes 73</div><div>4.3.1 Polytomous logistic regression 73</div><div>4.3.2 Example: histology of residual masses 73</div><div>*4.3.3 Alternative models 75</div><div>*4.3.4 Comparison of modeling approaches 76</div><div>4.4 Ordinal outcomes 77</div><div>4.4.1 Proportional odds logistic regression 77</div><div>* 4.4.2 Relevance of the proportional odds assumption in RCTs 78</div><div>4.5 Survival outcomes 80</div><div>4.5.1 Cox proportional hazards regression 80</div><div>4.5.2 Prediction with Cox models 81</div><div>4.5.3 Proportionality assumption 81</div><div>4.5.4 Kaplan-Meier analysis 81</div><div>*4.5.5 Example: impairment after treatment of leprosy 82</div><div>4.5.6 Parametric survival 82</div><div>*4.5.7 Example: replacement of risky heart valves 83</div><div>4.5.8 Summary on survival outcomes 83</div><div>4.6 Competing risks 84</div><div>4.6.1 Actuarial and actual risks 84</div><div>4.6.2 Absolute risk and the Fine&Gray model 84</div><div>4.6.3 Example: Prediction of coronary heart disease incidence 85</div><div>4.6.4 Multi-state modeling 86</div><div>4.7 Dynamic predictions 87</div><div>4.7.1 Multi-state models and landmarking 87</div>4.7.2 Joint models 87<div>4.8 Concluding remarks 88</div><div>Chapter 5 Overfitting and optimism in prediction models 91</div><div>5.1 Overfitting and optimism 91</div><div>5.1.1 Example: surgical mortality in esophagectomy 92</div><div>5.1.2 Variability within one center 92</div><div>5.1.3 Variability between centers: noise vs. true heterogeneity 93</div><div>5.1.4 Predicting mortality by center: shrinkage 94</div><div>5.2 Overfitting in regression models 95</div><div>5.2.1 Model uncertainty and testimation bias 95</div><div>5.2.2 Other modeling biases 97</div><div>5.2.3 Overfitting by parameter uncertainty 97</div><div>5.2.4 Optimism in model performance 98</div><div>5.2.5 Optimism-corrected performance 99</div><div>5.3 Bootstrap resampling 100</div><div>5.3.1 Applications of the bootstrap 101</div><div>5.3.2 Bootstrapping for regression coefficients 102</div><div>5.3.3 Bootstrapping for prediction: optimism correction 102</div><div>5.3.4 Calculation of optimism-corrected performance 103</div><div>*5.3.5 Example: Stepwise selection in 429 patients 104</div><div>5.4 Cost of data analysis 105</div><div>*5.4.1 Degrees of freedom of a model 105</div><div>5.4.2 Practical implications 105</div><div>5.5 Concluding remarks 106</div><div>Chapter 6 Choosing between alternative models 109</div><div>6.1 Prediction with statistical models 109</div><div>6.1.1 Testing of model assumptions and prediction 110</div><div>6.1.2 Choosing a type of model 110</div><div>6.2 Modeling age – outcome relations 111</div><div>*6.2.1 Age and mortality after acute MI 111</div><div>*6.2.2 Age and operative mortality 112</div><div>*6.2.3 Age – outcome relations in other diseases 115</div><div>6.3 Head-to-head comparisons 116</div><div>6.3.1 StatLog results 116</div><div>*6.3.2 Cardiovascular disease prediction comparisons 117</div><div>*6.3.3 Traumatic brain injury modeling results 119</div><div>6.4 Concluding remarks 120</div><div>Part II: Developing valid prediction models 123</div><div>Checklist for developing valid prediction models 124</div><div>Chapter 7 Missing values 125</div><div>7.1 Missing values and prediction research 125</div><div>7.1.1 Inefficiency of complete case analysis 126</div><div>7.1.2 Interpretation of CC Analyses 127</div><div>7.1.3 Missing data mechanisms 127</div><div>7.1.4 Missing outcome data 128</div>7.1.5 Summary points 129<div>7.2 Prediction under MCAR, MAR and MNAR mechanisms 130</div><div>7.2.1 Missingness patterns 130</div><div>7.2.2 Missingness and estimated regression coefficients 132</div><div>7.2.4 Missingness and estimated performance 134</div><div>7.3 Dealing with missing values in regression analysis 135</div><div>7.3.1 Imputation principle 135</div><div>7.3.2 Simple and more advanced single imputation methods 136</div><div>7.3.3 Multiple imputation 137</div><div>7.4 Defining the imputation model 138</div><div>7.4.1 Types of variables in the imputation model 138</div><div>*7.4.2 Transformations of variables 139</div><div>7.4.3 Imputation models for SI 139</div><div>7.4.4 Summary points 139</div><div>7.5 Success of imputation under MCAR, MAR and MNAR 140</div><div>7.5.1 Imputation in a simple model 140</div><div>7.5.2 Other simulation results 140</div><div>* 7.5.3 Multiple predictors 140</div><div>7.6 Guidance to dealing with missing values in prediction research 142</div><div>7.6.1 Patterns of missingness 142</div><div>7.6.2 Simple approaches 143</div><div>7.6.3 More advanced approaches 143</div><div>7.6.4 Maximum fraction of missing values before omitting a predictor 143</div><div>7.6.5 Single or multiple imputation for predictor effects? 144</div><div>7.6.6 Single or multiple imputation for deriving predictions? 145</div><div>7.6.7 Missings and predictions for new patients 145</div><div>*7.6.8 Performance across multiple imputed data sets 146</div><div>7.6.9 Reporting of missing values in prediction research 146</div><div>7.7 Concluding remarks 148</div><div>7.7.1 Summary statements 148</div><div>*7.7.2 Available software and challenges 149</div><div>Chapter 8 Case study on dealing with missing values 151</div><div>8.1 Introduction 151</div><div>8.1.1 Aim of the IMPACT study 151</div><div>8.1.2 Patient selection 152</div><div>8.1.3 Potential predictors 152</div><div>8.1.4 Coding and time dependency of predictors 153</div><div>8.2 Missing values in the IMPACT study 153</div><div>8.2.1 Missing values in outcome 153</div><div>8.2.2 Quantification of missingness of predictors 154</div><div>8.2.3 Patterns of missingness 156</div><div>8.3 Imputation of missing predictor values 159</div><div>8.3.1 Correlations between predictors 159</div><div>8.3.2 Imputation model 160</div><div>8.3.3 Distributions of imputed values 160</div><div>*8.3.4 Multilevel imputation 161</div><div>8.4 Predictor effect: adjusted analyses 162</div><div>8.4.1 Adjusted analysis for complete predictors: age and motor score 163</div><div>8.4.2 Adjusted analysis for incomplete predictors: pupils 165</div><div>8.5 Predictions: multivariable analyses 165</div><div>*8.5.1 Multilevel analyses 166</div><div>8.6 Concluding remarks 166</div><div>Chapter 9 Coding of categorical and continuous predictors 169</div>9.1 Categorical predictors 169<div>9.1.1 Examples of categorical coding 170</div><div>9.2 Continuous predictors 171</div><div>*9.2.1 Examples of continuous predictors 171</div><div>9.2.2 Categorization of continuous predictors 172</div><div>9.3 Non-linear functions for continuous predictors 173</div><div>9.3.1. Polynomials 173</div><div>9.3.2. Fractional polynomials (FP) 174</div><div>9.3.3 Splines 175</div><div>*9.3.4 Example: functional forms with RCS or FP 176</div><div>9.3.5 Extrapolation and robustness 176</div><div>9.3.5 Preference for FP or RCS? 176</div><div>9.4 Outliers and winsorizing 177</div><div>9.4.1 Example: glucose values and outcome of TBI 178</div><div>9.5 Interpretation of effects of continuous predictors 180</div><div>*9.5.1 Example: predictor effects in TBI 181</div><div>9.6 Concluding remarks 182</div><div>9.6.1 Software 183</div><div>Chapter 10 Restrictions on candidate predictors 185</div><div>10.1 Selection before studying the predictor – outcome relation 185</div><div>10.1.1 Selection based on subject knowledge 185</div><div>*10.1.2 Examples: too many candidate predictors 185</div><div>10.1.3 Meta-analysis for candidate predictors 186</div><div>*10.1.4 Example:&nbsp; predictors in testicular cancer 186</div><div>10.1.5 Selection based on distributions 186</div><div>10.2 Combining similar variables 187</div><div>10.2.1 Subject knowledge for grouping 187</div><div>10.2.2 Assessing the equal weights assumption 188</div><div>10.2.3 Biologically motivated weighting schemes 189</div><div>10.2.4 Statistical combination 189</div><div>10.3 Averaging effects 190</div>*10.3.1 Example: Chlamydia trachomatis infection risks 190<div>*10.3.2 Example: acute surgery risk relevant for elective patients? 190</div><div>*10.4 Case study: family history for prediction of a genetic mutation 191</div><div>10.4.1 Clinical background and patient data 191</div>10.4.2 Similarity of effects 191<div>10.4.3 CRC and adenoma in a proband 194</div><div>10.4.5 Full prediction model for mutations 196</div><div>10.5 Concluding remarks 197</div><div>Chapter 11 Selection of main effects 199</div><div>11.1 Predictor selection 199</div><div>11.1.1 Reduction before modeling 199</div><div>11.1.2 Reduction while modeling 200</div><div>11.1.3 Collinearity 200</div><div>11.1.4 Parsimony 200</div><div>11.1.5 Non-significant candidate predictors 201</div><div>11.1.6 Summary points on predictor selection 201</div><div>11.2 Stepwise selection 202</div><div>11.2.1 Stepwise selection variants 202</div><div>11.2.2 Stopping rules in stepwise selection 202</div><div>11.3 Advantages of stepwise methods 203</div><div>11.4 Disadvantages of stepwise methods 204</div><div>11.4.1 Instability of selection 204</div><div>11.4.2 Testimation: Biased in selected coefficients 206</div><div>*11.4.3 Testimation: empirical illustrations 207</div><div>11.4.4 Misspecification of variability and p-values 208</div><div>11.5 Influence of noise variables 210</div><div>11.6 Univariate analyses and model specification 211</div><div>11.6.1 Pros and cons of univariate pre-selection 211</div><div>*11.6.2 Testing of predictors in a domain 212</div><div>11.7 Modern selection methods 212</div><div>*11.7.1 Bootstrapping for selection 212</div><div>*11.7.2 Bagging and boosting 212</div><div>*11.7.3 Bayesian model averaging (BMA) 213</div><div>11.7.4 Shrinkage of regression coefficients to zero 213</div><div>11.8 Concluding remarks 214</div><div>Chapter 12 Assumptions in regression models: Additivity and linearity 217</div><div>12.1 Additivity and interaction terms 217</div><div>12.1.1 Potential interaction terms to consider 218</div><div>12.1.2 Interactions with treatment 218</div><div>12.1.3 Other potential interactions 219</div><div>*12.1.4 Example: time and survival after valve replacement 220</div><div>12.2 Selection, estimation and performance with interaction terms 220</div><div>12.2.1 Example: age interactions in GUSTO-I 220</div><div>12.2.2 Estimation of interaction terms 221</div><div>12.2.3 Better prediction with interaction terms? 222</div><div>12.2.4 Summary points 223</div><div>12.3 Non-linearity in multivariable analysis 223</div><div>12.3.1 Multivariable restricted cubic splines (rcs) 224</div><div>12.3.2 Multivariable fractional polynomials (FP) 225</div><div>12.3.3 Multivariable splines in gam 225</div><div>12.4 Example: non-linearity in testicular cancer case study 226</div><div>*12.4.1 Details of multivariable FP and gam analyses 227</div><div>*12.4.2 GAM in univariate and multivariable analysis 228</div><div>*12.4.3 Predictive performance 229</div><div>*12.4.4 R code for non-linear modeling in testicular cancer example 230</div><div>12.5 Concluding remarks 230</div><div>12.5.1 Recommendations 231</div><div>Chapter 13 Modern estimation methods 233</div><div>13.1 Predictions from regression and other models 233</div><div>*13.1.1 Estimation with other modeling approaches 234</div><div>13.2 Shrinkage 234</div><div>13.2.1 Uniform shrinkage 235</div><div>13.2.2 Uniform shrinkage: illustration 236</div><div>13.3 Penalized estimation 236</div><div>*13.3.1 Penalized maximum likelihood estimation 237</div><div>13.3.2 Penalized ML: illustration 238</div><div>*13.3.3 Optimal penalty by bootstrapping 238</div><div>13.3.4 Firth regression 239</div><div>*13.3.5 Firth regression: illustration 239</div><div>*13.4.1 Estimation of a LASSO model 240</div><div>13.5 Elastic net 241</div><div>*13.5.1 Estimation of Elastic Net model 241</div><div>13.6 Performance after shrinkage 242</div><div>13.6.1 Shrinkage, penalization, and model selection 242</div><div>13.7 Concluding remarks 244</div><div>Chapter 14 Estimation with external information 247</div><div>Background 247</div><div>14.1 Combining literature and individual patient data (IPD) 247</div><div>14.1.1 A global prediction model 248</div><div>*14.1.2 A global model for traumatic brain injury 249</div><div>14.1.3 Developing a local prediction model 249</div><div>14.1.4 Adaptation of univariate coefficients 250</div><div>*14.1.5 Adaptation method 1 250</div><div>*14.1.6 Adaptation method 2 251</div><div>*14.1.7 Estimation of adaptation factors 251</div><div>*14.1.8 Simulation results 252</div><div>14.1.9 Performance of the adapted model 253</div><div>14.2 Case study: prediction model for AAA surgical mortality 254</div><div>14.2.1 Meta-analysis 254</div><div>14.2.2 Individual patient data analysis 255</div><div>14.2.3 Adaptation and clinical presentation 256</div><div>14.3 Alternative approaches 257</div><div>14.3.1 Overall calibration 257</div><div>14.3.2 Stacked regressions 257</div><div>14.3.3 Bayesian methods: using data priors to regression modeling 257</div><div>14.3.4 Example: predicting neonatal death 258</div><div>*14.3.5 Example: aneurysm study 258</div><div>14.4 Concluding remarks 258</div>Chapter 15 Evaluation of performance 261<div>15.1 Overall performance measures 261</div><div>15.1.1 Explained variation: R2 261</div><div>15.1.2 Brier score 262</div><div>15.1.3 Performance of testicular cancer prediction model 263</div><div>15.3.4 Assessment of moderate calibration 283</div><div>15.3.5 Assessment of strong calibration 283</div><div>15.3.6 Calibration of survival predictions 284</div><div>15.3.7 Example: calibration in testicular cancer prediction model 285</div><div>*15.3.8 R code for assessing calibration 286</div><div>15.3.9 Calibration and discrimination 286</div><div>15.4 Concluding remarks 287</div><div>15.4.1 Bibliographic notes 287</div><div>Chapter 16 Evaluation of clinical usefulness 289</div><div>16.1 Clinical usefulness 289</div><div>16.1.1 Intuitive approach to the cutoff 290</div><div>16.1.2 Decision-analytic approach: benefit vs harm 290</div><div>16.1.3 Accuracy measures for clinical usefulness 291</div><div>16.1.4 Decision curve analysis 292</div><div>16.1.5 Interpreting net benefit in decision curves 293</div><div>16.1.6 Example: clinical usefulness of prediction in testicular cancer 295</div><div>16.1.7 Decision curves for testicular cancer example 296</div><div>16.1.8 Verification bias and clinical usefulness 297</div><div>*16.1.9 R code 298</div><div>16.2 Discrimination, calibration, and clinical usefulness 300</div><div>16.2.1 Discrimination, calibration, and Net Benefit in the testicular cancer case study 300</div>16.2.2 Aims of prediction models and performance measures 301<div>16.2.2 Summary points 302</div><div>16.3 From prediction models to decision rules 303</div><div>16.3.1 Performance of decision rules 303</div><div>16.3.2 Treatment benefit in prognostic subgroups 305</div><div>16.3.3 Evaluation of classification systems 305</div><div>16.4 Concluding remarks 306</div><div>Chapter 17 Validation of prediction models 309</div><div>17.1 Internal versus external validation, and validity 309</div><div>17.1.1 Assessment of internal and external validity 310</div><div>17.2 Internal validation techniques 311</div><div>17.2.1 Apparent validation 311</div><div>17.2.3 Cross-validation 313</div><div>17.2.4 Bootstrap validation 314</div><div>17.2.5 Internal validation combined with imputation 315</div><div>17.3 External validation studies 315</div><div>17.3.1 Temporal validation 316</div><div>*17.3.2 Example: validation of a model for Lynch syndrome 316</div><div>17.3.3 Geographic validation 317</div><div>17.3.4 Fully independent validation 319</div><div>17.3.5 Reasons for poor validation 320</div><div>17.4 Concluding remarks 321</div><div>Chapter 18 Presentation formats 323</div><div>18.1 Prediction models versus decision rules 323</div><div>18.2 Clinical prediction models 325</div><div>18.2.1 Regression formulas 325</div><div>18.2.2 Confidence intervals for predictions 326</div><div>18.2.3 Nomograms 327</div><div>18.2.4 Score chart 329</div><div>18.2.5 Tables with predictions 330</div><div>18.2.6 Specific formats 331</div><div>18.2.7 Black box presentations 331</div><div>18.3 Case study: clinical prediction model for testicular cancer model 333</div><div>18.3.1 Regression formula from logistic model 333</div><div>18.3.2 Nomogram 334</div><div>*18.3.3 Score chart 334</div><div>18.3.4 Summary points 335</div><div>18.4 Clinical decision rules 335</div><div>18.4.1 Regression tree 335</div><div>18.4.2 Score chart rule 335</div><div>18.4.3 Survival groups 336</div><div>18.4.4 Meta-model 337</div><div>18.5 Concluding remarks 338</div><div>Part III: Generalizability of prediction models 341</div><div>Chapter 19 Patterns of external validity 343</div><div>19.1 Determinants of external validity 343</div><div>19.1.1 Case-mix 343</div><div>19.1.2 Differences in case-mix 343</div><div>19.1.3 Differences in regression coefficients 344</div><div>19.2.1 Simulation set-up 345</div><div>19.2.2 Performance measures 347</div><div>19.3 Distribution of predictors 348</div><div>19.3.1 More or less severe case-mix according to X 348</div><div>*19.3.2 Interpretation of testicular cancer validation 349</div><div>19.3.3 More or less heterogeneous case-mix according to X 349</div><div>19.3.4 More or less severe case-mix according to Z 350</div><div>19.3.5 More or less heterogeneous case-mix according to Z 351</div><div>19.4 Distribution of observed outcome y 353</div><div>19.5 Coefficients β 354</div><div>19.5.1 Coefficient of linear predictor &lt; 1 354</div><div>19.5.2 Coefficients β different 355</div><div>19.6 Summary of patterns of invalidity 356</div><div>19.6.1 Other scenarios of invalidity 357</div>19.7 Reference values for performance 358<div>19.7.1 Model-based performance: performance if the model is valid 358</div><div>19.7.2 Performance with refitting 358</div><div>*19.7.3 Examples: testicular cancer and TBI 359</div><div>*19.7.4 R code 360</div><div>19.8 Limited validation sample size 361</div><div>19.8.1 Uncertainty in validation of performance 361</div><div>*19.8.2 Estimating standard errors in validation studies 363</div><div>19.8.3 Summary points 363</div><div>19.9 Design of external validation studies 363</div><div>19.9.1 Power of external validation studies 364</div><div>*19.9.2 Calculating sample sizes for validation studies 365</div><div>19.9.3 Rules for sample size of validation studies 366</div><div>19.9.4 Summary points 367</div><div>19.10 Concluding remarks 368</div><div>Chapter 20 Updating for a new setting 371</div><div>20.1 Updating only the intercept 372</div><div>20.1.1 Simple updating methods 372</div><div>20.2 Approaches to more extensive updating 372</div><div>20.2.1 Eight updating methods for predicting binary outcomes 373</div><div>20.3 Validation and updating in GUSTO-I 375</div><div>20.3.1 Validity of TIMI-II model for GUSTO-I 376</div>20.3.2 Updating the TIMI-II model for GUSTO-I 377<div>20.3.3 Performance of updated models 378</div><div>*20.3.4 R code for updating methods 379</div><div>20.4 Shrinkage and updating 379</div><div>20.4.1 Shrinkage towards recalibrated values in GUSTO-I 380</div><div>*20.4.2 R code for shrinkage and penalization in updating 381</div><div>20.4.4 Bayesian updating 382</div><div>20.5 Sample size and updating strategy 383</div><div>*20.5.1 Simulations of sample size, shrinkage, and updating strategy 384</div><div>20.5.2 A closed test for the choice of updating strategy 386</div><div>20.6 Validation and updating of tree models 386</div><div>20.7 Validation and updating of survival models 388</div><div>*20.7.1 Validation of a simple index for non-Hodgkin's lymphoma 388</div><div>20.7.2 Updating the prognostic index 389</div><div>20.7.3 Recalibration for groups by time points 389</div><div>20.7.4 Recalibration with a Cox or Weibull regression model 390</div><div>20.7.6 Summary points 391</div><div>20.8 Continuous updating 392</div><div>*20.8.1 Precision and updating strategy 392</div><div>*20.8.2 Continuous updating in GUSTO-I 393</div><div>*20.8.3 Other dynamic modeling approaches 394</div><div>20.9 Concluding remarks 396</div><div>*20.9.1 Further illustrations of updating 397</div><div>Chapter 21 Updating for multiple settings 401</div><div>21.1 Differences in outcome 401</div><div>21.1.1 Testing for calibration-in-the large 401</div>*21.1.2 Illustration of heterogeneity in GUSTO-I 402<div>21.1.3 Updating for better calibration-in-the large 403</div><div>21.1.4 Empirical Bayes estimates 403</div><div>*21.1.5 Illustration of updating in GUSTO-I 404</div><div>21.1.6 Testing and updating of predictor effects 405</div><div>*21.1.7 Heterogeneity of predictor effects in GUSTO-I 405</div><div>*21.1.8 R code for random effect analyses in GUSTO-I 405</div><div>21.2 Provider profiling 406</div><div>21.2.1 Ranking of centers: the expected rank 407</div><div>*21.2.2 Example: provider profiling in stroke 408</div><div>*21.2.4 Estimation and interpreting differences between centers 409</div><div>*21.2.5 Ranking of centers 410</div><div>*21.2.6 R code for provider profiling 411</div>21.3 Concluding remarks 412<div>*21.3.1 Further literature 413</div><div>Part IV: Applications 415</div><div>Chapter 22 Case study on a prediction of 30-day mortality 417</div><div>22.1 GUSTO-I study 417</div><div>22.1.1 Acute myocardial infarction 417</div><div>*22.1.2 Treatment results from GUSTO-I 418</div><div>22.1.3 Prognostic modeling in GUSTO-I 418</div><div>22.2 General considerations of model development 421</div><div>22.2.1 Research question and intended application 421</div><div>22.2.2 Outcome and predictors 421</div><div>22.2.3 Study design and analysis 421</div><div>22.3 Seven modeling steps in GUSTO-I 423</div><div>22.3.1 Preliminary 423</div><div>22.3.2 Coding of predictors 423</div><div>22.3.3 Model specification 423</div><div>22.3.4 Model estimation 423</div><div>22.3.5 Model performance 424</div><div>22.3.6 Model validation 424</div><div>22.3.7 Presentation 425</div><div>22.3.8 Predictions 426</div><div>22.4 Validity 428</div><div>22.4.1 Internal validity: overfitting 428</div>22.4.2 External validity: generalizability 428<div>22.4.3 Summary points 429</div><div>22.5 Translation into clinical practice 429</div><div>22.5.1 Score chart for choosing thrombolytic therapy 429</div><div>22.5.2 From predictions to decisions 430</div><div>22.6 Concluding remarks 432</div><div>Chapter 23 Case study on survival analysis: prediction of cardiovascular events 435</div><div>23.1 Prognosis in the SMART study 435</div><div>*23.1.1 Patients in SMART 436</div><div>23.2 General considerations in SMART 438</div><div>23.2.1 Research question and intended application 438</div><div>23.2.2 Outcome and predictors 438</div><div>23.2.3 Study design and analysis 438</div><div>23.3 Preliminary modeling steps in the SMART cohort 440</div><div>23.3.1 Patterns of missing values 440</div><div>23.3.2 Imputation of missing values 441</div><div>23.3.3 R code 442</div><div>23.4 Coding of predictors 443</div><div>23.4.1 Extreme values 443</div><div>23.4.2 Transforming continuous predictors 444</div><div>23.4.3 Combining predictors with similar effects 445</div><div>23.4.4 R code 446</div><div>23.5.1 A full model 447</div><div>23.5.2 Impact of imputation 449</div><div>23.5.3 R code for full model and imputation variants 449</div><div>23.6 Model selection and estimation 451</div><div>23.6.1 Stepwise selection 451</div><div>23.6.2 LASSO for selection with imputed data 452</div><div>23.7 Model performance and internal validation 453</div><div>23.7.1 Estimation of optimism in performance 453</div><div>23.7.2 Model presentation 456</div><div>23.7.3 R code for presentations 457</div><div>23.8 Concluding remarks 458</div><div>Chapter 24 Overall lessons and data sets 461</div><div>24.1 Sample size 461</div><div>24.1.1 Model selection, estimation, and sample size 462</div><div>24.1.2 Calibration improvement by penalization 463</div><div>24.1.3 Poorer performance with more predictors 464</div><div>24.1.4 Model selection with noise predictors 465</div><div>24.1.5 Potential solutions 466</div><div>24.1.6 R code for model selection and penalization 466</div><div>24.2 Validation 467</div><div>24.2.1 Examples of internal and external validation 467</div><div>24.3 Subject matter knowledge versus machine learning 468</div><div>24.3.1 Exploiting subject matter knowledge 468</div><div>24.3.2 Machine learning and Big Data 470</div><div>24.4 Reporting of prediction models and risk of bias assessments 470</div><div>24.4.1 Reporting guidelines 470</div><div>24.4.2 Risk of bias assessment 472</div>24.5 Data sets 473<div>24.5.1 GUSTO-I prediction models 473</div><div>24.5.2 SMART case study 475</div><div>24.5.3 Testicular cancer case study 476</div><div>24.5.4 Abdominal aortic aneurysm case study 478</div><div>24.6 Concluding remarks 481</div><div>References 483</div><div>&nbsp;</div><div><br></div>