Modelscape Validate

Modelscape™ Validate™ is a model validation and documentation workflow. Use this workflow if you review business line data, models, assumptions, or if you build challenger models. Use Modelscape Validate to validate models written in any programming language.

You can record anomalies or observations related to models and the appropriate closure measures or actions and export the findings to a PDF file using the Review Editor app. The app also logs and stores all activity with the model, which ensures traceability and reproducibility.

Modelscape Validate comprises these features:

• Interactive apps for data preparation, model construction, model testing, and validation

• A comprehensive library of validation statistics, machine and deep learning, financial, risk, and economic algorithms

• Customizable and reusable model development and documentation templates

• Automated model documentation generation

You can use Modelscape Validate to perform these tasks:

• Build, test, experiment with, and validate multiple models in parallel.

• Create models from validated pre-built functions instead of writing code or building your own libraries.

• Automate iterative workflows through code generation and reuse.

• Integrate algorithms and internal IP.

• Generate live, auditable, and traceable documentation for model validation and governance.

• Preserve model development history for auditability, transparency, and knowledge transfer.

Modelscape Validate Workflow

This figure shows how to use the Modelscape Develop™ workflow in parallel with the Modelscape Validate workflow. The Develop workflow comprises the white boxes and the Validate workflow comprises the orange boxes. You can also perform the validation workflow independently after the development workflow.

After you develop a model version, you can propose to deploy it. To deploy the model version, you must first validate it.

Raise Review Request

To begin validation for a model version, lock the commit version of the Modelscape model in the GitHub^® repository and raise a review request for that model version. This action sends a review request to the Validator team.

Open Review Editor App

Open the Modelscape Home page, look for the model version, and open a review. Opening a review opens the Review Editor app. Use the Review Editor app to perform model validation on that model version.

Analyze Model Version

Create live scripts using the Review Editor app and use them to explore and analyze data. You can run the models and perform what-if analyses, which include adjusting the parameters of models, using different methods for fitting models, and observing any consequent changes in the model performance.

You can use the metrics in Modelscape to analyze the bias in your models. For more details, see Fairness Metrics in Modelscape.

You can write scripts to implement model validation suites. For an example script that shows how to validate a probability of default model, see Credit Scorecard Validation Metrics. This example shows how to use the techniques in the BCBS Working Paper 14.

If your model version uses a programming language other than MATLAB^®, you can still use the validation tools from Modelscape. For more information, see Validation of External Models.

For an example that shows how to validate credit models using the European Central Bank template, see Validation of Credit Models in ECB Templates.

Attach Documents to Review

You can attach documents to the Review Editor app. These documents include recommendations for improvement or evidence of model performance reports outside the Review Editor app. Use the app to attach files such as detailed model validation documents and scripts supporting such documents. By default, the Review Editor attaches the files to the project or model repository.

You can customize the location to which you save documents. See Extensibility for more details.

Make Review Decision

Finish your review by using the submit button and sign-off your form in the app. You can customize the Submit Review dropdown and sign-off forms based on the needs of your organization. For more details, see Extensibility.

By default, the Review Editor app provides a full review form and a reduced review form. Use the reduced review form to approve the model if the changes are trivial. If the changes are not trivial and have a significant material impact, for example on the value of trade, use the full review form.

You can choose to approve or reject the model version. The next stages depend on the model lifecycle. If you do not approve the model version, it could be sent back to the developer. If you approve the model version, it could be sent to production and deployment.