How YourCloudGroup approaches auditable, measurable models
This lesson shows how one product, YourCloudGroup Model Manager, applies the ideas in this course. It is included as a concrete example; the principles in the earlier lessons apply whatever tools you choose.
The problem it addresses
Many business decisions are repeated predictions from structured records: which customers are likely to buy, how much they will spend, which cases need attention first. Lesson 2 explained why a model trained on your own records is often worth evaluating for these, and Lesson 4 explained why its accuracy must be measured on data it has never seen and compared with simpler alternatives. Model Manager is built around those two points, for people who are not data scientists.
How the ideas in this course show up in the product
| Principle from this course | How Model Manager applies it |
|---|---|
| Train on the data that matches the decision (Lesson 2) | Models are trained only on the customer's own spreadsheet or CSV data, uploaded as a file or loaded from a link, such as a published Google Sheets export or an S3 presigned link. |
| Keep data under the owner's control (Lesson 3) | Each subscription runs on its own dedicated server, not shared infrastructure. |
| Test on unseen data and compare alternatives (Lesson 4) | An automatic "tournament" builds twelve candidate models of different types, evaluates each against data held back from training, and selects the best performer. |
| Use measures that fit the decision (Lesson 4) | Results are reported with standard measures: R², MAE and RMSE for numbers; confusion matrices and per-class accuracy for categories; confidence analysis and row-by-row results. |
| Keep evidence that can be checked (Lessons 4 and 5) | Each model receives a certificate made of specific, evidence-based claims about how it was built and tested, so that its performance can be reviewed rather than taken on trust. |
| Keep measuring as things change (Lesson 4) | Models can be retrained on new data, and predictions and training can be scheduled to run regularly. |
You can read how model certification works on the YourCloudGroup blog.
What this does not mean
Model Manager provides evidence about how models were built and evaluated, including reported performance measures. Such evidence supports review; it does not guarantee that a model will perform equally well on future data.
- No model is right every time. Measured performance tells you how often, and how badly, a model is wrong on data like the test data, not that it will never be wrong.
- Good results depend on good data: enough relevant, accurate historical records with a known outcome.
- A person should still own decisions that matter, using the model's prediction and its measured accuracy as evidence.
Where to go next
If you want to see these ideas in practice, the YourCloudGroup home page describes Model Manager and its free weekly webinar. Whatever tools you use, keep the three questions from Lesson 1 in mind: What did it learn from? How was it checked? What happens when it is wrong?