# Chapter 19 Instructor Solution: Feature Store and Model Serving Contract

A strong submission should demonstrate that the student understands the operational boundary between feature engineering and inference serving. The contract should make it possible to answer four questions: which feature values were eligible at prediction time, which values were materialized online, which model endpoint consumed them, and which controls decide whether the candidate model can be promoted.

## Reference assessment

| Artifact | What a correct solution should show |
|---|---|
| `feature_store_contract.yml` | The contract identifies entities, join keys, event timestamps, created timestamps, offline sources, online stores, source delays, lookback windows, TTLs, owners, and quality gates. A correct answer explicitly prevents future data from entering training rows. |
| `online_feature_payloads.jsonl` | The sample payloads include entity keys, feature-view names, feature timestamps, materialization timestamps, a feature service name, and concrete numeric feature values. The payload format should resemble what a low-latency model endpoint would retrieve from Redis, Tair, DynamoDB, or another online store. |
| `skew_test_cases.csv` | The skew evidence compares offline and online values for the same entity and feature. A strong answer includes both passing examples and at least one review case so students learn that skew testing is an operational gate rather than a cosmetic checklist. |
| `serving_release_plan.yml` | The release plan connects endpoint versions, runtime choice, batching, latency SLOs, shadow testing, canary rollout, monitoring, rollback thresholds, and reviewer approval. It should be possible to determine who stops a bad rollout and which metric triggers the stop. |
| `tests/validate_feature_serving_lab.py` | The validator should remain dependency-free and structural. It is not a replacement for integration tests against a real feature store or serving platform. |

## Interpretation of starter choices

The starter contract uses `customer_id` and `merchant_id` as separate entity keys because fraud scoring often needs both customer behavior and merchant risk context. The `source_delay` fields are intentionally present because a point-in-time join should not assume that a feature is available immediately after the business event occurs. The `temporal_join_lookback` fields prevent arbitrarily old feature values from being selected when recent values are missing.

The online payload examples are deliberately simple JSONL records. This keeps the lab runnable in a clean environment while preserving the essential online-store contract: an entity key maps to a feature-view payload with event time, materialization time, and named feature values. In production, the same information might be encoded as Redis hashes, DynamoDB items, Tair records, or feature-server responses.

The serving plan selects a KServe-style platform with a Triton-style runtime to force students to distinguish between **deployment orchestration** and **inference execution**. KServe, Knative, and a cloud service such as Alibaba Cloud PAI-EAS handle endpoint lifecycle, scaling, traffic management, and monitoring. Triton optimizes inference execution, batching, scheduling, and multi-framework runtime support.

## Validation command

From `shared/labs/ch19_feature_store_model_serving`, run:

```bash
python3 tests/validate_feature_serving_lab.py
```

Expected output:

```text
PASS validate_feature_store_contract
PASS validate_online_payloads
PASS validate_skew_test_cases
PASS validate_serving_release_plan
All Chapter 19 feature store and model serving checks passed.
```

## Common grading issues

| Issue | Why it matters | Corrective feedback |
|---|---|---|
| Missing `source_delay` | The training set may include values that would not have been available during production scoring. | Add source-delay-aware point-in-time logic to the feature contract and training query. |
| Online payload lacks timestamps | The endpoint cannot enforce freshness or debug stale decisions. | Store both the feature event timestamp and the materialization timestamp. |
| Skew test only checks schema | Training-serving skew can occur even when names and types match. | Compare values, tolerances, freshness, and missing-key behavior for sampled entities. |
| Canary plan lacks rollback thresholds | A rollout can continue harming users after early warning signals appear. | Define automatic rollback triggers, owners, and observation windows. |
| Runtime and platform are conflated | Teams may choose tools for the wrong layer of the system. | Separate model runtime concerns such as batching from platform concerns such as scaling and traffic splitting. |