# Chapter 19 Guided Lab: Build a Feature Store and Model Serving Contract

This guided lab turns the Chapter 19 production feature-store and model-serving concepts into a small, reviewable contract. The objective is not to deploy Redis, Feast, Triton, or Kubernetes. The objective is to specify the **offline feature definition, online feature payload, serving release plan, and skew-test evidence** that allow a model endpoint to be promoted safely.

## Learning objectives

By the end of the lab, you should be able to explain how a production model connects point-in-time-correct offline features, online materialized values, feature freshness controls, inference endpoint versions, rollout strategy, and skew tests. You should also be able to run a lightweight validation script that checks whether the contract includes the minimum evidence expected by data engineers, ML engineers, and platform reviewers.

## Materials

| File | Purpose |
|---|---|
| `feature_store_contract.yml` | Defines entities, feature views, event timestamps, source delay, lookback, online materialization, owners, and freshness rules. |
| `online_feature_payloads.jsonl` | Provides sample online-store payloads that the model service would retrieve by entity key during inference. |
| `skew_test_cases.csv` | Compares expected offline feature values with online feature values for sampled entities and records pass/fail evidence. |
| `serving_release_plan.yml` | Specifies endpoint versions, runtime choice, batching policy, canary rollout, monitoring, and rollback controls. |
| `tests/validate_feature_serving_lab.py` | Runs local structural checks against the contract artifacts using only the Python standard library. |

## Workflow

First, inspect `feature_store_contract.yml` and confirm that every feature view declares an entity, event timestamp, source delay, temporal lookback, owner, offline source, online store, TTL, and freshness expectation. The `source_delay` and `temporal_join_lookback` fields should make it clear which feature values are safe to use at prediction time.

Second, review `online_feature_payloads.jsonl`. Confirm that each record includes an entity key, feature timestamps, a materialization timestamp, a model feature service name, and the numeric feature values required by the serving contract. Online values should be explicit rather than implied by application defaults.

Third, inspect `skew_test_cases.csv`. A useful skew test compares offline and online values for the same entity and feature, checks freshness, and records the result. The starter cases intentionally include multiple customers and multiple features so reviewers can see how the test would scale beyond a single happy path.

Fourth, review `serving_release_plan.yml`. Confirm that the endpoint has explicit model versions, a runtime choice, a latency SLO, a batching policy, a canary plan, monitoring rules, and rollback thresholds. The plan should show how a model can move from shadow testing to canary traffic and then to full production.

Finally, run the validator from the lab directory:

```bash
python3 tests/validate_feature_serving_lab.py
```

Expected output:

```text
PASS validate_feature_store_contract
PASS validate_online_payloads
PASS validate_skew_test_cases
PASS validate_serving_release_plan
All Chapter 19 feature store and model serving checks passed.
```

## Extension path

After the contract passes, extend the files to support a real platform implementation. For example, you can translate the feature contract into Feast definitions, load the online payloads into Redis or Tair, implement the skew checks as a scheduled CI job, or map the serving release plan to KServe, Triton, or Alibaba Cloud PAI-EAS.

## Cleanup

This lab does not create external cloud resources. If you extend the lab by deploying an endpoint, creating an online store, or provisioning GPUs, delete unused services, keys, queues, storage buckets, and model artifacts after completing the exercise.

## Troubleshooting

| Problem | Likely cause | Fix |
|---|---|---|
| `Missing required file` | The validator is not being run from the lab folder or a file was renamed. | Run the command from `shared/labs/ch19_feature_store_model_serving` and keep the starter filenames. |
| `Feature contract missing keys` | A required feature-store field was removed or renamed. | Restore the missing key or update the validator intentionally. |
| `Payload missing required field` | An online-store JSON record does not include entity, timestamp, or feature data. | Add the missing field to every JSONL record. |
| `Skew test should include both PASS and REVIEW evidence` | The skew cases are too shallow to illustrate operational review. | Keep at least one passing case and one review case, or add richer evidence. |
| `Serving plan missing term` | A release control such as canary, monitoring, or rollback was omitted. | Add the missing operational term to `serving_release_plan.yml`. |