Independent validation and testing for predictive models — yours or anyone’s. Eight test families locate where, when and why a model fails, and produce evidence a validator can stand behind.
Metrics across datasets, segments and slices — beyond a single headline number.
Locate the regions of the input space where the model underperforms.
Structure in the errors is a finding. Examine residuals against features and segments.
Identify where train–test gaps concentrate rather than averaging them away.
Prediction uncertainty: how much should any single prediction be trusted?
Behavior under input perturbation and noise.
Behavior under distribution shift between environments.
Group-level performance and error analysis for review by your compliance function.
Challenger and benchmark models compared side by side, sliced the same way.
Validation teams rarely choose the training stack. Model wrappers bring externally built models under the same test suites — adopting Model Studio is not a prerequisite.
ts.diagnose_residual_analysis(features="hr", dataset="test") ts.explain_pfi() # permutation feature importance ts.explain_pdp(features="hr") # partial dependence ts.interpret_fi() # inherent feature importance tsc.compare_accuracy_table(train_dataset="train", test_dataset="test", metric=("MAE", "R2")) tsc.compare_slicing_accuracy(features="hr", method="uniform", bins=5, metric="MAE")
Every test returns a structured validation result: tables and plots that drop straight into a validation report, carrying the configuration that produced them. When data or models move, ongoing validation re-runs the same suite.
How MRM teams use it →Model Assurance ships inside the modeva package. Install it and everything below works now.
# the TestSuite works on wrapped models from any framework ts.diagnose_residual_analysis(features="hr", dataset="test") tsc.compare_accuracy_table(train_dataset="train", test_dataset="test", metric=("MAE", "R2"))
