MLS-C01 (AWS Certified Machine Learning – Specialty) does not ask where the buttons are in SageMaker. It asks scenario questions: "To solve this business problem, what data do you use, how do you process it, which algorithm do you choose, which metrics do you evaluate with, and how do you deploy and monitor it?" So on day one we redraw the entire lifecycle at Specialty depth. Where the Associate level asks "what each stage is," the Specialty level asks about "the trade-offs between stages."
Today's goals are to nail down (1) how to translate a problem into an ML problem, (2) the cyclical structure of data → features → model → deployment → monitoring, and (3) the mindset of connecting offline model metrics to business metrics.
The most common failure happens not in modeling but in problem definition. "We want to reduce churn" is a business goal, not an ML problem. To translate it into an ML problem, you must fix three things.
If you get the problem type wrong, everything after it goes off the rails. The Specialty exam constantly tests the following mapping through scenarios.
| Business question | ML problem type | Typical output |
|---|---|---|
| Is this transaction fraudulent? | Binary classification | Probability between 0 and 1 |
| Which tier does this customer belong to? | Multiclass classification | Class label |
| What will next month's revenue be? | Regression | Continuous value |
| Which group is similar to this user? | Clustering | Cluster ID |
| What product will they buy next? | Recommendation | Ranked list |
| Is this sensor reading abnormal? | Anomaly detection | Anomaly score |
💡 Related theory: Supervised learning learns an input→output mapping from labeled data, while unsupervised learning discovers structure without labels. "Fraud detection" is usually solved as supervised binary classification, but if labels (historical fraud cases) are extremely scarce, you approach it with anomaly detection (an unsupervised technique like Random Cut Forest). The fact that the same business problem changes problem type depending on is a classic Specialty trap.
Click a choice to reveal the answer and explanation.
Question 1
A fintech company is building a fraud detection model on data where only 0.2% of transactions are fraudulent. The cost of a false negative (missing actual fraud) is far greater than that of a false positive. What is the most appropriate combination of evaluation metrics?
Question 2
When translating the request "we want to reduce customer churn" into an ML problem, what must be fixed first?
Question 3
A new payment service wants to start fraud detection but has almost no historical fraud labels. What is the most realistic approach?
Question 4
A recommendation model showed a large AUC improvement in offline evaluation and was deployed to all traffic immediately — and revenue dropped. What is the most appropriate way to prevent this in advance?
Question 5
What is the most essential reason to view the ML lifecycle as a "cyclical loop" rather than a "linear pipeline"?
An ML system is not built once and done. When data changes during operation (drift), you go back to the beginning.
1. Data : Collect → Clean → Label → Store (data lake)
2. Features : Feature engineering → Transform → Feature Store
3. Model : Algorithm selection → Training → HPO tuning → Evaluation
4. Deploy : Real-time endpoint / Batch transform / Serverless
5. Monitor : Data & model quality drift → Retraining trigger
└──────────────(loop back to 1)──────────────┘
This week (Week 1) focuses on stage 1 — data — and the stages just before it: ingestion, storage, and labeling. That is because the data engineering domain carries a large weight on the Specialty exam (about 20% of the total).
# The lifecycle as seen through the SageMaker SDK — separating responsibilities by stage
import sagemaker
from sagemaker.processing import ProcessingInput, ProcessingOutput
session = sagemaker.Session()
role = sagemaker.get_execution_role()
# Stages 1-2: data cleaning + feature engineering as a Processing Job
from sagemaker.sklearn.processing import SKLearnProcessor
processor = SKLearnProcessor(
framework_version="1.2-1",
role=role,
instance_type="ml.m5.xlarge",
instance_count=1,
)
processor.run(
code="preprocess.py",
inputs=[ProcessingInput(source="s3://my-lake/raw/", destination="/opt/ml/processing/input")],
outputs=[ProcessingOutput(source="/opt/ml/processing/train", destination="s3://my-lake/features/train")],
)Separating each stage into its own job makes reproducibility and re-runs easy. You can re-run just the cleaning step, or train a different algorithm on the same features.
💡 Related theory: Training-serving skew is the performance degradation caused when the feature transformation logic used at training time differs from the logic used at inference time. Fixing preprocessing in code (
preprocess.py) as above and reusing it identically for training and inference, or managing features centrally with SageMaker Feature Store, reduces this skew. Features improvised ad hoc in a notebook almost always create skew.
This is where the Specialty exam digs deepest. A single metric like accuracy can mislead the business. On data where fraudulent transactions are 0.1%, predicting "all legitimate" still yields 99.9% accuracy. That is why you must consider class imbalance and error costs together.
# Computing the core metrics for a classification problem (based on the confusion matrix)
from sklearn.metrics import precision_score, recall_score, f1_score, roc_auc_score
# precision = TP / (TP + FP) → "of everything flagged as fraud, the fraction that is actually fraud" (cost of false positives)
# recall = TP / (TP + FN) → "of all actual fraud, the fraction caught" (cost of false negatives)
precision = precision_score(y_true, y_pred)
recall = recall_score(y_true, y_pred)
f1 = f1_score(y_true, y_pred) # harmonic mean of precision and recall
auc = roc_auc_score(y_true, y_score) # threshold-independent, ranking qualityBusiness context determines the choice of metric.
💡 Related theory: ROC-AUC is the area under the TPR-FPR curve across thresholds, so it is relatively insensitive to class imbalance. However, under extreme imbalance (0.1% positives) ROC-AUC looks optimistic, so PR-AUC — the area under the precision-recall curve — is a more honest signal. The Specialty exam frequently asks "which metric for imbalanced data?", and the answer is usually PR-AUC, or whichever of recall/precision carries the higher cost.
Even if offline metrics look good, actual user behavior (revenue, session time) can decline. That is why deployment does not switch over all at once — you split traffic and validate. SageMaker puts multiple variants behind a single endpoint and distributes traffic by weight.
from sagemaker.session import production_variant
variant_a = production_variant(model_name="model-v1", instance_type="ml.m5.large",
initial_instance_count=1, variant_name="A", initial_weight=90)
variant_b = production_variant(model_name="model-v2", instance_type="ml.m5.large",
initial_instance_count=1, variant_name="B", initial_weight=10)
session.endpoint_from_production_variants(
name="fraud-endpoint", production_variants=[variant_a, variant_b]
)
# Send only 10% to the new model (B), compare business metrics in CloudWatch, then adjust weightsUse offline metrics as a gate (no deployment if they fail) and online metrics as the final verdict. This separation is the operational sense the Specialty exam demands.