A/B Testing Deployed Models: Shadow Deployments and Canary Releases

Last time, Dev ran the numbers on his two recommenders — p-value of 0.03, modest but real effect size. The retrained model was genuinely better, not just lucky. But he’d learned the hard way that a model can win offline and still stumble in production. So he wasn’t about to roll this out to all ten million users at once.

Why You Can’t Just Swap Models in Production

Dev has heard this story before. A team spends three months building a new recommendation model. On the test set, it’s 7% more accurate than the old one. Everyone’s excited. They push it live on a Monday morning to all 10 million users. By Tuesday afternoon, the CEO is sending angry emails: revenue is down 12%, and users are complaining that recommendations are broken. Dev is determined not to be that team.

What happened? The test set didn’t match real-world data. The new model was trained on historical data from six months ago, but user behavior has shifted. Maybe the model is slower, and timeouts are causing errors. Or it’s hitting an edge case that never showed up in testing — some rare combination of user attributes that breaks the model’s assumptions.

Here’s the core problem: a model that looks great in development often performs worse in production. And when it fails, it fails fast and at scale. A bad model deployed to 100% of users can tank revenue, erode user trust, or in safety-critical systems like healthcare or finance, cause genuine harm — all within hours.

You need a way to test the new model on real traffic, with real users, before committing to it fully. That’s where shadow deployments and canary releases come in.

The Core Idea: Test Before You Trust

Here’s the intuition: instead of flipping a switch and sending all traffic to the new model, you test it gradually. You might send 5% of traffic to the new model, 95% to the old. You watch what happens. If it looks good, bump it to 10%. Then 25%. Then 100%. If anything goes wrong, roll back immediately — only a small fraction of users were affected.

This catches real-world problems that test data never showed. Latency issues come up, and so do edge cases you never anticipated. You learn how the model behaves when data drifts — all with minimal risk.

Two main strategies exist for gradual rollout: shadow deployment and canary releases. They have different trade-offs, and you’ll often use both.

Shadow Deployment: The Safest Way to Test

Think of shadow deployment like this: you hire a shadow consultant to sit in on all your meetings and give advice, but you never actually follow it. You just listen and see if they seem sharp. If they do, maybe you hire them full-time later.

In shadow deployment, the new model runs on every request, but its prediction never reaches the user. The old model’s prediction gets returned as usual. The new model’s output is logged and compared offline. Users never see the new model, so there’s zero risk to their experience.

Here’s what’s happening under the hood:

1. Request comes in from a user.
2. Old model makes a prediction and returns it to the user (as usual).
3. New model also makes a prediction on the same request, but the result is logged.
4. Offline, you compare the two predictions and compute metrics like latency, error rate, and prediction agreement.
5. If the new model looks good, you move to the next phase.

Shadow deployment is completely safe. Users are never affected. You get real-world performance data without any production risk.

Let’s code this up. We’ll simulate a shadow deployment where both models run on incoming requests, and we log their predictions and latencies:

First, the data structures and the two models we’re comparing:

import time
import random
from dataclasses import dataclass
from typing import List

@dataclass
class Request:
    user_id: int
    features: dict

@dataclass
class Prediction:
    model_name: str
    prediction: float
    latency_ms: float
    timestamp: float

class OldModel:
    """Simulates the current production model."""
    def predict(self, request: Request) -> float:
        # Simulate a simple model: average of feature values
        time.sleep(0.01)  # 10ms latency
        return sum(request.features.values()) / len(request.features)

class NewModel:
    """Simulates the new model we want to test."""
    def predict(self, request: Request) -> float:
        # Simulate a slightly different model
        time.sleep(0.015)  # 15ms latency (a bit slower)
        # Add a small random noise to simulate different behavior
        base = sum(request.features.values()) / len(request.features)
        return base + random.gauss(0, 0.05)

Now the harness itself — it runs both models on every request but only returns the old model’s prediction:

class ShadowDeployment:
    """Runs both models, returns old model's prediction, logs both."""
    def __init__(self, old_model, new_model):
        self.old_model = old_model
        self.new_model = new_model
        self.shadow_log = []
    
    def handle_request(self, request: Request) -> float:
        # Run old model and time it
        start = time.time()
        old_pred = self.old_model.predict(request)
        old_latency = (time.time() - start) * 1000  # Convert to ms
        
        # Run new model and time it
        start = time.time()
        new_pred = self.new_model.predict(request)
        new_latency = (time.time() - start) * 1000
        
        # Log both predictions
        self.shadow_log.append({
            'old_pred': old_pred,
            'old_latency': old_latency,
            'new_pred': new_pred,
            'new_latency': new_latency,
            'pred_diff': abs(old_pred - new_pred)
        })
        
        # Return only the old model's prediction to the user
        return old_pred
    
    def get_shadow_metrics(self):
        """Compute metrics from the shadow log."""
        if not self.shadow_log:
            return None
        
        old_latencies = [log['old_latency'] for log in self.shadow_log]
        new_latencies = [log['new_latency'] for log in self.shadow_log]
        pred_diffs = [log['pred_diff'] for log in self.shadow_log]
        
        return {
            'old_model_avg_latency_ms': sum(old_latencies) / len(old_latencies),
            'new_model_avg_latency_ms': sum(new_latencies) / len(new_latencies),
            'latency_increase_pct': ((sum(new_latencies) / len(new_latencies)) / (sum(old_latencies) / len(old_latencies)) - 1) * 100,
            'avg_prediction_diff': sum(pred_diffs) / len(pred_diffs),
            'max_prediction_diff': max(pred_diffs),
            'num_requests': len(self.shadow_log)
        }

Finally, run 100 requests through it and print the metrics:

# Simulate shadow deployment
old_model = OldModel()
new_model = NewModel()
shadow = ShadowDeployment(old_model, new_model)

# Process 100 requests
for i in range(100):
    request = Request(
        user_id=i,
        features={'feature_1': random.uniform(0, 1), 'feature_2': random.uniform(0, 1)}
    )
    prediction = shadow.handle_request(request)

# Print shadow metrics
metrics = shadow.get_shadow_metrics()
print("Shadow Deployment Metrics:")
print(f"  Old model avg latency: {metrics['old_model_avg_latency_ms']:.2f}ms")
print(f"  New model avg latency: {metrics['new_model_avg_latency_ms']:.2f}ms")
print(f"  Latency increase: {metrics['latency_increase_pct']:.1f}%")
print(f"  Avg prediction difference: {metrics['avg_prediction_diff']:.4f}")
print(f"  Max prediction difference: {metrics['max_prediction_diff']:.4f}")
print(f"  Requests processed: {metrics['num_requests']}")

This is the shadow deployment harness Dev builds for his retrained recommender — both the incumbent and the candidate run on every incoming request, but only the incumbent’s prediction reaches the user. The candidate’s output is logged for offline comparison.

• @dataclass class Request — a lightweight container for an incoming recommendation request. In Dev’s real pipeline, features would hold the user’s browsing history, session features, and product attributes — not just two random floats.
• @dataclass class Prediction — a structured record for a model’s output, including latency and timestamp. Defined here for completeness but not used until Dev wires it into a production logging system.
• OldModel.predict() — simulates the incumbent recommender. time.sleep(0.01) injects a 10ms latency to mimic real inference time. The prediction is the simple average of feature values — a stand-in for the real model’s score.
• NewModel.predict() — simulates the retrained candidate. It’s 50% slower (15ms vs 10ms) and adds Gaussian noise (random.gauss(0, 0.05)) to simulate slightly different recommendation scores.
• ShadowDeployment.__init__ — stores references to both models and initializes shadow_log, the list where every dual-prediction result is recorded.
• handle_request() — the core shadow logic: runs the old model, times it; runs the new model, times it; logs both predictions, both latencies, and the absolute difference between them; returns only the old model’s prediction to the caller. The user never sees the new model’s output.
• get_shadow_metrics() — aggregates the shadow log into a summary: average latency for each model, the percentage increase in latency (the key red-flag metric), and the average and max prediction differences. A large max_prediction_diff would tell Dev there are edge cases where the two models diverge significantly — worth investigating before moving to canary.
• latency_increase_pct — computed as (new_avg / old_avg - 1) * 100. This is the metric that would make Dev pause: a 50% latency increase on a model serving 10 million users could mean hundreds of additional milliseconds of aggregate wait time per second.

When you run this, you’ll see something like:

Shadow Deployment Metrics:
  Old model avg latency: 10.23ms
  New model avg latency: 15.34ms
  Latency increase: 49.9%
  Avg prediction difference: 0.0512
  Max prediction difference: 0.2847
  Requests processed: 100

What does this mean? The new model is about 50% slower than the old one — that’s a red flag. In production, if your old model takes 10ms and your new one takes 15ms, and you’re handling 10,000 requests per second, you’ve just added 50,000ms (50 seconds) of latency per second across your entire system. That’s a problem.

The prediction differences are small on average (0.05), which is good — it means the new model agrees with the old one most of the time. But the max difference of 0.28 suggests there are some edge cases where they diverge significantly.

In a real shadow deployment, you’d run this for 24-48 hours, collect thousands of requests, and look at the metrics. If the new model’s latency is acceptable and predictions are reasonable, you move to the next phase.

Canary Releases: Gradual Traffic Shifting

Shadow deployment is safe, but expensive — you’re running two models on every request. It also can’t tell you how the new model affects real business metrics like revenue, retention, or user satisfaction, because users never see it.

Canary releases solve this. A canary release routes a small percentage of real traffic to the new model and the rest to the old one. Users see the new model’s predictions. You compare metrics between groups and watch for trouble.

The name comes from the old mining practice of carrying a canary into the mine. If the canary died, miners knew the air was toxic. Here, the canary is that small group of users on the new model. If something goes wrong, you roll back before it reaches everyone.

Here’s how it works:

1. Start with 5% of traffic going to the new model, 95% to the old one.
2. Monitor metrics for both groups: latency, error rate, and business KPIs (conversion, revenue, etc.).
3. If the new model looks good, bump to 10%. Then 25%. Then 50%. Then 100%.
4. If anything looks bad at any point, roll back immediately.

Let’s code this up:

import random
from collections import defaultdict
from dataclasses import dataclass
from typing import Dict, List

@dataclass
class UserEvent:
    user_id: int
    model: str  # 'old' or 'new'
    prediction: float
    latency_ms: float
    error: bool
    conversion: bool  # Did the user convert? (mock business metric)

CanaryRelease is the traffic-splitting harness — it routes each request to old or new based on canary_percentage, tracks per-group metrics, and can check itself for a regression:

class CanaryRelease:
    """Splits traffic between old and new models, tracks metrics per group."""
    def __init__(self, old_model, new_model, canary_percentage=5):
        self.old_model = old_model
        self.new_model = new_model
        self.canary_percentage = canary_percentage
        self.events: List[UserEvent] = []
    
    def handle_request(self, request: Request) -> float:
        """Route request to old or new model based on canary percentage."""
        # Decide which model to use
        if random.random() < self.canary_percentage / 100:
            model_to_use = 'new'
            model = self.new_model
        else:
            model_to_use = 'old'
            model = self.old_model
        
        # Get prediction and latency
        start = time.time()
        try:
            prediction = model.predict(request)
            latency = (time.time() - start) * 1000
            error = False
        except Exception as e:
            latency = (time.time() - start) * 1000
            prediction = 0
            error = True
        
        # Simulate a conversion (business metric)
        # In reality, this would come from user behavior tracking
        conversion = random.random() < (0.1 if prediction > 0.5 else 0.05)
        
        # Log the event
        self.events.append(UserEvent(
            user_id=request.user_id,
            model=model_to_use,
            prediction=prediction,
            latency_ms=latency,
            error=error,
            conversion=conversion
        ))
        
        return prediction
    
    def get_metrics_by_group(self) -> Dict:
        """Compute metrics for old and new model groups."""
        metrics_by_group = defaultdict(lambda: {
            'count': 0,
            'latencies': [],
            'errors': 0,
            'conversions': 0
        })
        
        for event in self.events:
            group = metrics_by_group[event.model]
            group['count'] += 1
            group['latencies'].append(event.latency_ms)
            if event.error:
                group['errors'] += 1
            if event.conversion:
                group['conversions'] += 1
        
        # Compute summary statistics
        result = {}
        for model_name, group in metrics_by_group.items():
            avg_latency = sum(group['latencies']) / len(group['latencies']) if group['latencies'] else 0
            error_rate = group['errors'] / group['count'] if group['count'] > 0 else 0
            conversion_rate = group['conversions'] / group['count'] if group['count'] > 0 else 0
            
            result[model_name] = {
                'count': group['count'],
                'avg_latency_ms': avg_latency,
                'error_rate': error_rate,
                'conversion_rate': conversion_rate
            }
        
        return result
    
    def check_for_regression(self, threshold_latency_pct=20, threshold_error_rate=0.02) -> Dict:
        """Check if new model has regressed compared to old model."""
        metrics = self.get_metrics_by_group()
        
        if 'old' not in metrics or 'new' not in metrics:
            return {'regression_detected': False, 'reason': 'Not enough data'}
        
        old_metrics = metrics['old']
        new_metrics = metrics['new']
        
        # Check latency
        latency_increase_pct = ((new_metrics['avg_latency_ms'] - old_metrics['avg_latency_ms']) / old_metrics['avg_latency_ms']) * 100
        if latency_increase_pct > threshold_latency_pct:
            return {
                'regression_detected': True,
                'reason': f'Latency increased by {latency_increase_pct:.1f}% (threshold: {threshold_latency_pct}%)'
            }
        
        # Check error rate
        error_rate_increase = new_metrics['error_rate'] - old_metrics['error_rate']
        if error_rate_increase > threshold_error_rate:
            return {
                'regression_detected': True,
                'reason': f'Error rate increased by {error_rate_increase:.2%} (threshold: {threshold_error_rate:.2%})'
            }
        
        return {'regression_detected': False, 'reason': 'All metrics look good'}

This is the canary release harness Dev builds for his recommender — unlike the shadow deployment above, here a small percentage of real users actually see the retrained model’s recommendations, and the system tracks whether their behavior (conversions) differs from the control group.

• @dataclass class UserEvent — a structured record for each request, capturing which model served it, the prediction, latency, whether it errored, and whether the user “converted” (clicked through / purchased). In Dev’s real system, conversion would come from the event stream, not a random draw.
• CanaryRelease.__init__ — stores both models, the canary percentage (default 5%), and an events list that accumulates every served request for later analysis.
• handle_request() — the traffic-splitting logic. random.random() < self.canary_percentage / 100 gives each request a 5% chance of being routed to the new model. The rest go to the old model. The try/except block catches prediction errors so a crash in the new model doesn’t take down the whole system.
• conversion = random.random() < (0.1 if prediction > 0.5 else 0.05) — simulates a business metric. Higher predictions lead to higher conversion probability. In production, Dev would track actual click-through rates or purchase events — this is a stand-in for the real business signal the canary is meant to validate.
• get_metrics_by_group() — partitions all events by model name ('old' vs 'new') and computes per-group averages: count, average latency, error rate, and conversion rate. This is the dashboard Dev checks every 15 minutes during a canary.
• check_for_regression() — the automatic guard. It compares the new model’s latency and error rate against the old model’s, using configurable thresholds (threshold_latency_pct=20 means “roll back if the new model is more than 20% slower”). This is what would auto-trigger a rollback at 3am if the new model starts misbehaving.
• canary_percentage=5 — the starting canary weight. At 5% of 1,000 simulated requests, roughly 50 land on the new model — enough to spot a 50% latency regression but not enough to detect a 1% conversion-rate difference (that’s the statistical power problem covered later in the article).

Run 1,000 requests through it at a 5% canary and print the per-group metrics:

# Simulate canary release
old_model = OldModel()
new_model = NewModel()
canary = CanaryRelease(old_model, new_model, canary_percentage=5)

# Process 1000 requests
for i in range(1000):
    request = Request(
        user_id=i,
        features={'feature_1': random.uniform(0, 1), 'feature_2': random.uniform(0, 1)}
    )
    canary.handle_request(request)

# Print metrics
metrics = canary.get_metrics_by_group()
print("Canary Release Metrics:")
for model_name, model_metrics in metrics.items():
    print(f"\n{model_name.upper()} Model:")
    print(f"  Requests: {model_metrics['count']}")
    print(f"  Avg latency: {model_metrics['avg_latency_ms']:.2f}ms")
    print(f"  Error rate: {model_metrics['error_rate']:.2%}")
    print(f"  Conversion rate: {model_metrics['conversion_rate']:.2%}")

Output might look like:

Canary Release Metrics:

OLD Model:
  Requests: 950
  Avg latency: 10.25ms
  Error rate: 0.00%
  Conversion rate: 7.16%

NEW Model:
  Requests: 50
  Avg latency: 15.38ms
  Error rate: 0.00%
  Conversion rate: 6.00%

Then check the regression guard:

# Check for regression
regression_check = canary.check_for_regression()
print(f"\nRegression Check: {regression_check['reason']}")

Regression Check: Latency increased by 50.0% (threshold: 20%)

What’s happening here? We sent 5% of traffic to the new model — 50 out of 1000 requests. The new model’s latency is 50% higher than the old one. That’s a regression. We’d roll back immediately and investigate why the new model is so slow.

The conversion rate is also slightly lower for the new model (6% vs 7.16%), but with only 50 requests on the new model, that could be random noise. This is where statistical significance comes in — we’ll get to that later.

Shadow vs. Canary: When to Use Each

Both shadow and canary are safe ways to test new models, but they have different trade-offs.

Shadow deployment is the safest option. Users never see the new model, so there’s zero risk to their experience. You get real-world performance data without any production impact. The downside is that it’s expensive — you’re running two models on every request. And it doesn’t tell you how the new model affects business metrics because users never actually see it.

Canary releases are faster and cheaper. You only run the new model on a small percentage of traffic. You see real business metrics because users actually see the new model’s predictions. The downside is that there’s a small risk — if something goes wrong, a small fraction of users are affected.

In practice, many teams use both. Start with shadow deployment to catch obvious problems (latency, errors, crashes). Once shadow looks good, move to canary to see real business impact. This gives you the safety of shadow plus the business insight of canary.

Here’s a quick comparison:

# Comparison of Shadow vs Canary

comparison = {
    'Shadow Deployment': {
        'User Risk': 'None (users never see new model)',
        'Cost': 'High (run both models on every request)',
        'Time to Deploy': 'Slow (need 24-48 hours of data)',
        'Business Metrics': 'Not captured (users never see new model)',
        'Best For': 'High-risk domains (healthcare, finance, safety-critical)'
    },
    'Canary Release': {
        'User Risk': 'Low (only small % of users affected)',
        'Cost': 'Low (run new model on small % of traffic)',
        'Time to Deploy': 'Fast (can ramp up in hours)',
        'Business Metrics': 'Captured in real time',
        'Best For': 'Most production scenarios'
    }
}

for strategy, traits in comparison.items():
    print(f"\n{strategy}:")
    for trait, value in traits.items():
        print(f"  {trait}: {value}")

This is the summary table Dev keeps in his deployment runbook — a quick reference for choosing between shadow and canary when planning a rollout for his recommender.

• comparison = { ... } — a nested dictionary mapping each strategy name to a dictionary of trait → value pairs. Each trait (User Risk, Cost, Time to Deploy, Business Metrics, Best For) captures one dimension of the trade-off.
• for strategy, traits in comparison.items() — iterates over the top-level keys ('Shadow Deployment' and 'Canary Release') and their nested trait dictionaries.
• for trait, value in traits.items() — iterates over each trait within a strategy, printing the trait name and its value. The output is a plain-text table that Dev can glance at during a deployment planning meeting.

Setting Up Metrics That Matter

Deployment’s hardest part isn’t serving the model — it’s knowing which metrics to track and what thresholds to set. Track the wrong ones and you’ll either miss real problems or roll back good models.

Metrics fall into two categories.

Technical metrics surface problems fast but don’t tell the full story:

• Latency (how long does the model take to make a prediction?)
• Error rate (how often does the model crash or return an error?)
• Throughput (how many requests per second can the model handle?)

Business metrics matter most but take longer to show signal:

• Conversion rate (do users buy?)
• Revenue (how much money do users spend?)
• Retention (do users come back?)
• User satisfaction (do users like the recommendations?)

You need both. Technical metrics give you early warning; business metrics give you final validation.

Here’s a simple metric dashboard:

from datetime import datetime, timedelta
from collections import defaultdict

class MetricsDashboard:
    """Tracks technical and business metrics for canary deployment."""
    def __init__(self):
        self.metrics_by_time = defaultdict(lambda: {'old': [], 'new': []})
        self.thresholds = {
            'latency_increase_pct': 20,  # Roll back if latency increases >20%
            'error_rate_increase': 0.02,  # Roll back if error rate increases >2%
            'conversion_rate_decrease_pct': 5  # Roll back if conversion decreases >5%
        }
    
    def record_event(self, model: str, latency_ms: float, error: bool, conversion: bool):
        """Record a single event."""
        now = datetime.now().replace(second=0, microsecond=0)  # Round to minute
        self.metrics_by_time[now][model].append({
            'latency': latency_ms,
            'error': error,
            'conversion': conversion
        })
    
    def get_current_metrics(self):
        """Get metrics for the most recent minute."""
        if not self.metrics_by_time:
            return None
        
        latest_time = max(self.metrics_by_time.keys())
        latest_data = self.metrics_by_time[latest_time]
        
        result = {}
        for model_name in ['old', 'new']:
            if not latest_data[model_name]:
                continue
            
            events = latest_data[model_name]
            latencies = [e['latency'] for e in events]
            errors = sum(1 for e in events if e['error'])
            conversions = sum(1 for e in events if e['conversion'])
            
            result[model_name] = {
                'count': len(events),
                'avg_latency_ms': sum(latencies) / len(latencies),
                'error_rate': errors / len(events),
                'conversion_rate': conversions / len(events)
            }
        
        return result
    
    def check_health(self):
        """Check if new model has regressed."""
        metrics = self.get_current_metrics()
        
        if not metrics or 'old' not in metrics or 'new' not in metrics:
            return {'status': 'OK', 'alerts': []}
        
        old_m = metrics['old']
        new_m = metrics['new']
        alerts = []
        
        # Check latency
        latency_increase_pct = ((new_m['avg_latency_ms'] - old_m['avg_latency_ms']) / old_m['avg_latency_ms']) * 100
        if latency_increase_pct > self.thresholds['latency_increase_pct']:
            alerts.append(f"ALERT: Latency increased {latency_increase_pct:.1f}%")
        
        # Check error rate
        error_rate_increase = new_m['error_rate'] - old_m['error_rate']
        if error_rate_increase > self.thresholds['error_rate_increase']:
            alerts.append(f"ALERT: Error rate increased {error_rate_increase:.2%}")
        
        # Check conversion rate
        if old_m['conversion_rate'] > 0:
            conversion_decrease_pct = ((old_m['conversion_rate'] - new_m['conversion_rate']) / old_m['conversion_rate']) * 100
            if conversion_decrease_pct > self.thresholds['conversion_rate_decrease_pct']:
                alerts.append(f"ALERT: Conversion rate decreased {conversion_decrease_pct:.1f}%")
        
        status = 'ALERT' if alerts else 'OK'
        return {'status': status, 'alerts': alerts, 'metrics': metrics}

Feed it 100 old-model events and 20 new-model events (a 5:1 ratio mirroring a 5% canary):

# Simulate metric tracking
dashboard = MetricsDashboard()

# Simulate 100 events for old model
for i in range(100):
    latency = random.gauss(10, 1)  # Mean 10ms, std dev 1ms
    error = random.random() < 0.001  # 0.1% error rate
    conversion = random.random() < 0.07  # 7% conversion rate
    dashboard.record_event('old', latency, error, conversion)

# Simulate 20 events for new model (5% of traffic)
for i in range(20):
    latency = random.gauss(15, 2)  # Mean 15ms, std dev 2ms (slower)
    error = random.random() < 0.001
    conversion = random.random() < 0.06  # Slightly lower conversion
    dashboard.record_event('new', latency, error, conversion)

Then check health and print any alerts:

# Check health
health = dashboard.check_health()
print(f"Status: {health['status']}")
if health['alerts']:
    for alert in health['alerts']:
        print(f"  {alert}")

if health['metrics']:
    print("\nMetrics:")
    for model, metrics in health['metrics'].items():
        print(f"  {model.upper()}: latency={metrics['avg_latency_ms']:.1f}ms, error_rate={metrics['error_rate']:.2%}, conversion={metrics['conversion_rate']:.2%}")

This is the monitoring dashboard Dev builds to run alongside his canary — it buckets events by minute, compares old vs. new model metrics, and fires alerts when the new model crosses a regression threshold.

• self.metrics_by_time = defaultdict(...) — a time-bucketed store. Each minute gets its own entry with separate lists for 'old' and 'new' model events. This lets Dev see metrics evolve over time rather than as a single aggregate.
• self.thresholds — the configurable alert thresholds. latency_increase_pct: 20 means “alert if the new model is more than 20% slower.” conversion_rate_decrease_pct: 5 means “alert if conversion drops by more than 5% relative to the old model.” Dev tunes these based on his recommender’s SLA and business sensitivity.
• record_event() — rounds the timestamp to the nearest minute (replace(second=0, microsecond=0)) so events are grouped into minute buckets, then appends the event to the appropriate model’s list.
• get_current_metrics() — finds the most recent minute bucket (max(self.metrics_by_time.keys())) and computes per-model averages for latency, error rate, and conversion rate. This is the snapshot Dev sees on his dashboard.
• check_health() — the automated guard. It compares the new model against the old model on three dimensions: latency increase percentage, error rate increase, and conversion rate decrease. Any breach appends an alert string. The status field is 'ALERT' if any alerts fired, 'OK' otherwise.
• conversion_decrease_pct = ((old - new) / old) * 100 — relative conversion decrease, not absolute. A drop from 7% to 6% is a 14.3% relative decrease — more alarming than it sounds in absolute terms, which is why Dev uses the relative formula for the threshold.
• The simulation injects 100 old-model events (10ms latency, 7% conversion) and 20 new-model events (15ms latency, 6% conversion) — a 5:1 ratio mirroring a 5% canary. The 50% latency increase fires the alert.

Output:

Status: ALERT
  ALERT: Latency increased 50.0%

Metrics:
  OLD: latency=10.1ms, error_rate=0.00%, conversion=7.00%
  NEW: latency=15.1ms, error_rate=0.00%, conversion=5.00%

This dashboard tracks both technical and business metrics and flags regressions automatically. In production, you’d run this check every 5–15 minutes and alert the team if something goes wrong.

A/B Testing in the Mix: Comparing Models Fairly

Canary releases are a form of A/B testing — you randomly assign users to the old or new model and compare outcomes. The catch: with only 5% of traffic on the new model, you need to run the test long enough to collect sufficient data.

Say the new model has a 7% conversion rate and the old model has 7.1%. That’s a tiny difference. Is it real, or just noise? With only 50 users on the new model, you can’t tell. You need hundreds or thousands before you can be confident.

Statistical significance is how we sort this out. We run a test that asks: if the two models were actually identical, how likely would we be to see a difference this big by chance alone?

“Very unlikely” (p-value < 0.05) means the difference is statistically significant. “Pretty likely” (p-value > 0.05) means it’s just noise.

Let’s code this up:

from scipy import stats

class ABTestAnalyzer:
    """Runs statistical tests to compare old and new models."""
    def __init__(self, old_events: List[UserEvent], new_events: List[UserEvent]):
        self.old_events = old_events
        self.new_events = new_events
    
    def compare_conversion_rates(self):
        """Compare conversion rates using chi-square test."""
        old_conversions = sum(1 for e in self.old_events if e.conversion)
        old_total = len(self.old_events)
        
        new_conversions = sum(1 for e in self.new_events if e.conversion)
        new_total = len(self.new_events)
        
        # Chi-square test
        # Contingency table: [[conversions, non-conversions], ...]
        contingency_table = [
            [old_conversions, old_total - old_conversions],
            [new_conversions, new_total - new_conversions]
        ]
        
        chi2, p_value, dof, expected = stats.chi2_contingency(contingency_table)
        
        old_rate = old_conversions / old_total
        new_rate = new_conversions / new_total
        difference_pct = ((new_rate - old_rate) / old_rate) * 100
        
        return {
            'old_conversion_rate': old_rate,
            'new_conversion_rate': new_rate,
            'difference_pct': difference_pct,
            'p_value': p_value,
            'significant': p_value < 0.05,
            'old_sample_size': old_total,
            'new_sample_size': new_total
        }
    
    def compare_latencies(self):
        """Compare latencies using t-test."""
        old_latencies = [e.latency_ms for e in self.old_events]
        new_latencies = [e.latency_ms for e in self.new_events]
        
        t_stat, p_value = stats.ttest_ind(old_latencies, new_latencies)
        
        old_mean = sum(old_latencies) / len(old_latencies)
        new_mean = sum(new_latencies) / len(new_latencies)
        difference_ms = new_mean - old_mean
        
        return {
            'old_avg_latency_ms': old_mean,
            'new_avg_latency_ms': new_mean,
            'difference_ms': difference_ms,
            'p_value': p_value,
            'significant': p_value < 0.05
        }

This is the statistical analysis Dev runs on his canary data — the same “is the difference real or noise?” question from the previous article, now applied to live production traffic instead of a holdout set.

• ABTestAnalyzer.__init__ — takes two lists of UserEvent objects: one from the old model’s traffic, one from the new model’s. These are the events accumulated during the canary.
• compare_conversion_rates() — the business-metric test. It counts conversions and non-conversions for each model, builds a 2×2 contingency table ([[old_conversions, old_non_conversions], [new_conversions, new_non_conversions]]), and runs stats.chi2_contingency() — a chi-square test of independence. The p-value answers: “if both models had the same true conversion rate, how likely is a difference this large by chance?”
• chi2, p_value, dof, expected = stats.chi2_contingency(contingency_table) — chi2 is the test statistic, p_value is the probability of seeing this difference under the null hypothesis, dof is degrees of freedom (1 for a 2×2 table), and expected is the table of expected counts under the null.
• difference_pct = ((new_rate - old_rate) / old_rate) * 100 — the relative percentage difference in conversion rate. A negative value means the new model is converting worse.
• compare_latencies() — the technical-metric test. It extracts latency lists from both event groups and runs stats.ttest_ind() — an independent two-sample t-test. This asks whether the mean latencies are significantly different. Unlike conversion (binary), latency is continuous, so a t-test is the right tool.
• significant': p_value < 0.05 — the conventional significance threshold. Dev uses 0.05 for exploratory checks but might tighten to 0.01 for high-stakes decisions.
• The simulation creates 1,000 old-model events (7% conversion, 10ms latency) and 50 new-model events (6% conversion, 15ms latency) — the same 5% canary split from the previous section, now analyzed statistically.

Generate the same 5% canary split as before, but at real sample sizes — 1,000 old-model events and 50 new-model events:

# Simulate A/B test with 1000 old model users and 50 new model users
old_events = []
for i in range(1000):
    old_events.append(UserEvent(
        user_id=i,
        model='old',
        prediction=random.uniform(0, 1),
        latency_ms=random.gauss(10, 1),
        error=random.random() < 0.001,
        conversion=random.random() < 0.07
    ))

new_events = []
for i in range(50):
    new_events.append(UserEvent(
        user_id=1000 + i,
        model='new',
        prediction=random.uniform(0, 1),
        latency_ms=random.gauss(15, 2),
        error=random.random() < 0.001,
        conversion=random.random() < 0.06
    ))

Run the conversion-rate test first:

# Run A/B test
analyzer = ABTestAnalyzer(old_events, new_events)

conversion_results = analyzer.compare_conversion_rates()
print("Conversion Rate Comparison:")
print(f"  Old model: {conversion_results['old_conversion_rate']:.2%}")
print(f"  New model: {conversion_results['new_conversion_rate']:.2%}")
print(f"  Difference: {conversion_results['difference_pct']:.1f}%")
print(f"  P-value: {conversion_results['p_value']:.4f}")
print(f"  Statistically significant: {conversion_results['significant']}")
print(f"  Sample sizes: Old={conversion_results['old_sample_size']}, New={conversion_results['new_sample_size']}")

Conversion Rate Comparison:
  Old model: 7.10%
  New model: 6.00%
  Difference: -15.5%
  P-value: 0.2847
  Statistically significant: False
  Sample sizes: Old=1000, New=50

Then the latency test:

latency_results = analyzer.compare_latencies()
print("\nLatency Comparison:")
print(f"  Old model: {latency_results['old_avg_latency_ms']:.2f}ms")
print(f"  New model: {latency_results['new_avg_latency_ms']:.2f}ms")
print(f"  Difference: {latency_results['difference_ms']:.2f}ms")
print(f"  P-value: {latency_results['p_value']:.4f}")
print(f"  Statistically significant: {latency_results['significant']}")

Latency Comparison:
  Old model: 10.05ms
  New model: 15.12ms
  Difference: 5.07ms
  P-value: 0.0000
  Statistically significant: True

What does this tell us? The new model’s conversion rate is 1.1 percentage points lower than the old model’s (6% vs 7.1%), but the difference isn’t statistically significant (p-value = 0.28). With only 50 users on the new model, we can’t be confident the gap is real — it could be random noise.

The latency difference, though, is highly significant (p-value ≈ 0). The new model is consistently slower. That’s a real problem.

Real-World Gotchas: What Actually Goes Wrong

Here’s what I’ve learned from watching deployments go wrong:

Metric definitions matter more than you think. A 1% gain in one metric can mean a 10% regression in another. Define your metrics before you deploy — what counts as a conversion, how you measure latency (Median? P95? Mean?). Get these wrong and you’ll make bad decisions.

Latency can spike unexpectedly. The new model might be slower for all sorts of reasons — a different algorithm, poor hardware fit, misconfigured infrastructure. In one case I saw, a team deployed a model that ran 3x slower than the old one. The model itself was fine; they’d just forgotten to enable GPU acceleration. They caught it in canary at 5% traffic and rolled back before it hit everyone.

Edge cases and rare user behaviors often don’t show up in test data but do in production. The new model might crash on an input type that’s rare in your test set but common in production. It might handle missing data differently, or break on very old user accounts. These things only surface under real traffic.

Rollback can be slow if you don’t plan for it. When you need to roll back, you need to do it fast. Have the procedure ready before you deploy. Can you flip back to the old model in seconds, or will it take 30 minutes? In production, 30 minutes is an eternity.

Let’s simulate a deployment that goes wrong:

class FailingNewModel:
    """A new model that has a latency spike."""
    def predict(self, request: Request) -> float:
        # Simulate a latency spike: 50% of the time, the model is very slow
        if random.random() < 0.5:
            time.sleep(0.1)  # 100ms latency
        else:
            time.sleep(0.015)  # 15ms latency
        return sum(request.features.values()) / len(request.features)

class DeploymentMonitor:
    """Monitors a deployment and decides whether to roll back."""
    def __init__(self, latency_threshold_ms=20):
        self.latency_threshold_ms = latency_threshold_ms
        self.old_latencies = []
        self.new_latencies = []
    
    def record_latency(self, model: str, latency_ms: float):
        if model == 'old':
            self.old_latencies.append(latency_ms)
        else:
            self.new_latencies.append(latency_ms)
    
    def should_rollback(self) -> bool:
        """Check if we should roll back based on latency."""
        if len(self.old_latencies) < 100 or len(self.new_latencies) < 100:
            return False  # Not enough data yet
        
        old_avg = sum(self.old_latencies[-100:]) / 100  # Last 100 requests
        new_avg = sum(self.new_latencies[-100:]) / 100
        
        if new_avg > old_avg * (1 + self.latency_threshold_ms / 100):
            return True
        return False

Now run 500 requests through it, routing 5% to the failing model, and check for rollback every 100 requests:

# Simulate deployment with failing model
old_model = OldModel()
failing_model = FailingNewModel()
monitor = DeploymentMonitor(latency_threshold_ms=20)

print("Simulating deployment with latency spike...")
for i in range(500):
    request = Request(
        user_id=i,
        features={'feature_1': random.uniform(0, 1), 'feature_2': random.uniform(0, 1)}
    )
    
    # Route 5% to new model
    if random.random() < 0.05:
        start = time.time()
        failing_model.predict(request)
        latency = (time.time() - start) * 1000
        monitor.record_latency('new', latency)
    else:
        start = time.time()
        old_model.predict(request)
        latency = (time.time() - start) * 1000
        monitor.record_latency('old', latency)
    
    # Check for rollback every 100 requests
    if i > 0 and i % 100 == 0:
        if monitor.should_rollback():
            print(f"Request {i}: ROLLBACK TRIGGERED")
            print(f"  Old model avg latency: {sum(monitor.old_latencies[-100:]) / 100:.2f}ms")
            print(f"  New model avg latency: {sum(monitor.new_latencies[-100:]) / 100:.2f}ms")
            break
        else:
            print(f"Request {i}: All metrics look good")

This is the failure-mode simulation Dev runs before his actual canary — he wants to verify that his rollback trigger fires correctly when the new model starts spiking. The FailingNewModel mimics a recommender that’s intermittently slow (perhaps hitting a cold cache or an unoptimized code path on certain inputs).

• FailingNewModel.predict() — 50% of requests take 100ms (a latency spike), the other 50% take 15ms (normal). This bimodal latency distribution is common in real systems — a cold-start penalty, a cache miss, or a slow branch in the inference code that only triggers on certain feature combinations.
• DeploymentMonitor.__init__ — stores the latency threshold (default 20%) and two latency lists, one per model. The threshold means “roll back if the new model’s average latency exceeds the old model’s by more than 20%.”
• record_latency() — appends each request’s measured latency to the appropriate model’s list.
• should_rollback() — the rolling-window check. It requires at least 100 samples from each model before making a decision (avoiding false alarms from tiny samples). It compares the last 100 latencies from each model: if new_avg > old_avg * 1.2, it returns True (roll back). Using the last 100 requests rather than all-time makes the monitor responsive to new regressions — a sudden spike won’t be diluted by hours of good data.
• if random.random() < 0.05: — routes 5% of traffic to the failing model, matching the canary percentage from the earlier sections.
• if i > 0 and i % 100 == 0: — checks for rollback every 100 requests. In production, Dev would check continuously (every 15 minutes) rather than every N requests, but the principle is the same: check frequently enough to catch a regression before it affects too many users.

Output:

Simulating deployment with latency spike...
Request 100: All metrics look good
Request 200: All metrics look good
Request 300: ROLLBACK TRIGGERED
  Old model avg latency: 10.23ms
  New model avg latency: 57.45ms

The monitor caught the latency spike and triggered a rollback at request 300. In production, that would mean reverting to the old model and investigating what went wrong.

Putting It Together: A Deployment Checklist

A safe deployment follows a predictable pattern. Here’s how to roll out a new model without surprises:

Step 1: Shadow Deployment (24-48 hours)

• Deploy the new model in shadow mode.
• Run it on all traffic, but don’t show predictions to users.
• Log predictions and metrics.
• Check: Is latency acceptable? Are there errors? Do predictions make sense?
• If anything looks off, go back to development and fix it.
• If shadow looks clean, move to step 2.

Step 2: Canary at 5% (1-2 hours)

• Route 5% of traffic to the new model.
• Monitor latency, error rate, and business metrics.
• Check every 15 minutes.
• If anything looks off, roll back immediately.
• If the 5% canary holds for 1-2 hours, move to step 3.

Step 3: Canary at 10% (1-2 hours)

• Bump to 10% traffic.
• Monitor for 1-2 hours.
• If anything looks off, roll back.
• If good, move to step 4.

Step 4: Canary at 25% (2-4 hours)

• Bump to 25% traffic.
• Monitor for 2-4 hours.
• If anything looks off, roll back.
• If good, move to step 5.

Step 5: Canary at 50% (4-8 hours)

• Bump to 50% traffic.
• Monitor for 4-8 hours.
• This is the point of no return — you’re now affecting half your users.
• If anything looks off, roll back.
• If good, move to step 6.

Step 6: 100% Traffic (24 hours)

• Route all traffic to the new model.
• Keep monitoring for 24 hours.
• Watch for issues that only surface at scale.
• If all looks good, you’re done.

Now let’s code this up as a mock deployment pipeline:

class DeploymentPipeline:
    """Manages a staged deployment with shadow and canary phases."""
    def __init__(self, old_model, new_model):
        self.old_model = old_model
        self.new_model = new_model
        self.phase = 'shadow'
        self.canary_percentage = 0
        self.events = []
    
    def handle_request(self, request: Request) -> float:
        """Route request based on current phase."""
        if self.phase == 'shadow':
            # Run both models, return old model's prediction
            old_pred = self.old_model.predict(request)
            new_pred = self.new_model.predict(request)
            self.events.append({'phase': 'shadow', 'model': 'old', 'pred': old_pred})
            self.events.append({'phase': 'shadow', 'model': 'new', 'pred': new_pred})
            return old_pred
        else:  # canary
            # Route based on canary percentage
            if random.random() < self.canary_percentage / 100:
                pred = self.new_model.predict(request)
                self.events.append({'phase': 'canary', 'model': 'new', 'pred': pred})
            else:
                pred = self.old_model.predict(request)
                self.events.append({'phase': 'canary', 'model': 'old', 'pred': pred})
            return pred
    
    def advance_phase(self, new_phase: str, canary_percentage: int = 0):
        """Advance to the next phase."""
        self.phase = new_phase
        self.canary_percentage = canary_percentage
        print(f"Advanced to {new_phase} phase (canary={canary_percentage}%)")
    
    def rollback(self):
        """Roll back to the previous phase."""
        if self.phase == 'canary':
            self.canary_percentage = max(0, self.canary_percentage - 5)
            if self.canary_percentage == 0:
                self.phase = 'shadow'
            print(f"Rolled back to {self.phase} phase (canary={self.canary_percentage}%)")

Run it through all six stages of the checklist above, from shadow through 100% traffic:

# Simulate deployment
pipeline = DeploymentPipeline(OldModel(), NewModel())

print("Phase 1: Shadow Deployment")
for i in range(100):
    request = Request(user_id=i, features={'f1': random.uniform(0, 1), 'f2': random.uniform(0, 1)})
    pipeline.handle_request(request)
print(f"  Processed 100 requests. Shadow looks good.")

print("\nPhase 2: Canary at 5%")
pipeline.advance_phase('canary', 5)
for i in range(100):
    request = Request(user_id=100 + i, features={'f1': random.uniform(0, 1), 'f2': random.uniform(0, 1)})
    pipeline.handle_request(request)
print(f"  Processed 100 requests. Canary at 5% looks good.")

print("\nPhase 3: Canary at 10%")
pipeline.advance_phase('canary', 10)
for i in range(100):
    request = Request(user_id=200 + i, features={'f1': random.uniform(0, 1), 'f2': random.uniform(0, 1)})
    pipeline.handle_request(request)
print(f"  Processed 100 requests. Canary at 10% looks good.")

print("\nPhase 4: Canary at 25%")
pipeline.advance_phase('canary', 25)
for i in range(100):
    request = Request(user_id=300 + i, features={'f1': random.uniform(0, 1), 'f2': random.uniform(0, 1)})
    pipeline.handle_request(request)
print(f"  Processed 100 requests. Canary at 25% looks good.")

print("\nPhase 5: Canary at 50%")
pipeline.advance_phase('canary', 50)
for i in range(100):
    request = Request(user_id=400 + i, features={'f1': random.uniform(0, 1), 'f2': random.uniform(0, 1)})
    pipeline.handle_request(request)
print(f"  Processed 100 requests. Canary at 50% looks good.")

print("\nPhase 6: 100% Traffic")
pipeline.advance_phase('canary', 100)
for i in range(100):
    request = Request(user_id=500 + i, features={'f1': random.uniform(0, 1), 'f2': random.uniform(0, 1)})
    pipeline.handle_request(request)
print(f"  Processed 100 requests. 100% traffic looks good.")
print(f"\nDeployment complete! Total events: {len(pipeline.events)}")

This is the end-to-end deployment pipeline Dev uses for his retrained recommender — it encodes the full staged rollout from the checklist above: shadow first, then canary at 5%, 10%, 25%, 50%, and finally 100%.

• DeploymentPipeline.__init__ — stores both models, starts in 'shadow' phase with canary_percentage=0, and initializes an events list for auditing.
• handle_request() — the routing logic, which branches on self.phase. In shadow mode, it runs both models but returns only the old model’s prediction (mirroring the ShadowDeployment class from earlier). In canary mode, it routes based on self.canary_percentage, same as CanaryRelease.
• advance_phase() — promotes the pipeline to the next stage. Dev calls this after each stage’s monitoring window passes without alerts. The new_phase and canary_percentage are set together — advance_phase('canary', 25) moves to canary at 25%.
• rollback() — reverses by decrementing canary_percentage by 5 points. If it hits 0, the phase reverts to 'shadow'. In a real system, Dev would also alert the team and freeze further promotions until the regression is investigated.
• The simulation processes 100 requests per phase (a stand-in for the 1-8 hour monitoring windows in the checklist). In production, Dev would process tens of thousands of requests per phase and check metrics continuously.
• print(f"\nDeployment complete! Total events: {len(pipeline.events)}") — the finish line. The pipeline logs 700 events total (200 from shadow — 100 per model — plus 500 from canary, since all five canary phases process 100 requests each), and Dev’s retrained recommender is now serving all 10 million users.

Output:

Phase 1: Shadow Deployment
  Processed 100 requests. Shadow looks good.

Phase 2: Canary at 5%
Advanced to canary phase (canary=5%)
  Processed 100 requests. Canary at 5% looks good.

Phase 3: Canary at 10%
Advanced to canary phase (canary=10%)
  Processed 100 requests. Canary at 10% looks good.

Phase 4: Canary at 25%
Advanced to canary phase (canary=25%)
  Processed 100 requests. Canary at 25% looks good.

Phase 5: Canary at 50%
Advanced to canary phase (canary=50%)
  Processed 100 requests. Canary at 50% looks good.

Phase 6: 100% Traffic
Advanced to canary phase (canary=100%)
  Processed 100 requests. 100% traffic looks good.

Deployment complete! Total events: 700

Tools and Platforms That Help

You don’t have to build shadow and canary deployment from scratch. Several existing tools handle this.

Kubernetes with Istio is the industry standard. Istio is a service mesh that sits between your services and manages traffic routing. You define canary deployments in Istio, and it gradually shifts traffic from the old model to the new one. It also handles metrics collection and automatic rollback.

Flagger runs on top of Kubernetes and automates canary deployments. You set thresholds for metrics like latency and error rate, and Flagger rolls back automatically if those thresholds are exceeded.

Feature flags (LaunchDarkly, Unleash) let you control which users see which model without redeploying. You can do canary releases this way — show the new model to 5% of users without touching your deployment infrastructure.

Monitoring platforms (Datadog, New Relic, Prometheus) collect metrics from your models and alert you to regressions. They integrate with deployment tools so you can roll back automatically when metrics go bad.

ML-specific platforms (Seldon, Cortex, Qwak) have built-in shadow and canary support. You define your models and deployment strategy, and the platform handles the rest.

For a small team just starting out, I’d lean toward:

1. Start with feature flags for canary releases — simple, no infrastructure needed.
2. Add a monitoring tool like Prometheus or Datadog to track metrics.
3. As you scale, move to Kubernetes with Istio for more sophisticated traffic routing.

What’s Next: From Canary to Continuous Deployment

Once you’re comfortable with shadow and canary deployments, automation is the natural next step. Rather than manually promoting from 5% to 10% to 25%, you let the system handle those jumps on its own.

Continuous deployment means new models go live automatically once they pass canary tests. You might ship a new model every day, or every hour. That requires robust monitoring and fast rollback — the foundations you’ve built in this article.

Next up: retraining pipelines. Right now, you train a model once and deploy it. In production, data drifts. User behavior shifts. Performance degrades. A retraining pipeline handles this by retraining on fresh data and deploying the new version if it beats the current model.

Then there’s online learning — models that update themselves as new data arrives, without retraining from scratch.

We’ll save those for later. For now, you have a toolkit for deploying models safely: shadow deployment for low-risk testing, canary releases for gradual rollout, and metrics to catch problems early.

Summary

What we covered:

1. Why it matters: Deploying an untested model is risky. A bad model can tank revenue and erode user trust within hours.

2. Shadow deployment: Run the new model on every request, but don’t surface predictions to users. Log metrics and compare offline. Zero user-facing risk — but it’s expensive, and it won’t show business impact.

3. Canary releases: Route a small percentage of traffic to the new model. Compare metrics between groups. Roll back if anything looks off. Faster and cheaper than shadow, with some risk.

4. Metrics that matter: Track technical metrics (latency, error rate) for early warning, and business metrics (conversion, revenue) for final validation.

5. Statistical significance: At small canary percentages, you need enough data to be confident the differences are real, not noise. Use statistical tests to validate.

6. The deployment checklist: Shadow (24-48h) → Canary 5% (1-2h) → 10% → 25% → 50% → 100% (24h). Monitor at each stage. Roll back if needed.

7. Tools: Kubernetes, Istio, Flagger, feature flags, and monitoring platforms can automate much of this.

You’re ready to deploy models safely. Start with shadow deployment to build confidence, then move to canary releases. Monitor closely at each stage. In production, slow and safe beats fast and broken.

The new recommender rolls out safely — shadow first, then 5%, 10%, 25%, 50%, 100% — and fully replaces the old one without a single revenue-dropping incident. But looking back over the whole journey, from the first training pipeline through drift detection, retraining, statistical validation, and gradual rollout, Dev notices a pattern: almost every incident traced back to the same root cause — inconsistent feature computation between training and serving, the exact train-serving skew he first encountered in his very first pipeline. He’s patched around it with monitoring, drift detection, and careful deployment practices. He’s tired of patching. He decides to fix it properly, once and for all — with a feature store.

Check Your Understanding

Questions below move from simple recall up to open-ended design, roughly following Bloom’s Taxonomy.

Remember What is the key difference between shadow deployment and a canary release—specifically, whether real users ever see the new model’s prediction?

Understand In your own words, explain why shadow deployment can tell you about latency and errors but can’t tell you about business metrics like conversion rate.

Apply Using the article’s regression-check formula ((new_latency - old_latency) / old_latency * 100), if the old model averages 12ms and the new model averages 16ms, what latency increase percentage would be reported, and would it trigger the article’s default 20% threshold?

Analyze The article’s A/B test shows a conversion-rate difference of -15.5% that is not statistically significant (p=0.28) at only 50 users on the new model, while a latency difference is highly significant (p≈0.0000) at that same sample size. Walk through why the same sample size can be “enough” to detect one kind of difference confidently but not another.

Evaluate The article’s staged rollout checklist advances from 5% to 10% to 25% to 50% to 100%, monitoring for a fixed window at each stage (1-2 hours, then 2-4, then 4-8). Critique using fixed time windows rather than fixed sample sizes as the promotion criterion: what could go wrong at low-traffic hours (e.g., 3am) that wouldn’t happen during peak traffic?

Create Design a rollback trigger policy for a scenario the article doesn’t cover: a fraud-detection model where “error rate” isn’t binary (crash vs. no crash) but is instead “flagged a legitimate transaction as fraud.” What metric would you track instead of latency/error-rate, and what threshold would justify an automatic rollback?

---

Related articles

• Is Your Model Actually Better? A Plain-English Guide to Statistical Significance) — Dev statistically confirmed the retrained recommender was a real improvement. This article picks up where that one left off: how he ships that improvement to production without tanking revenue.
• What Is a Feature Store and Do You Actually Need One?) — The deployment is done, but Dev realizes every incident in the lifecycle traced back to train-serving skew. Here’s how he closes the loop for good with a feature store.

References & Further reading

• Kohavi, R., Crook, T., Longbotham, R., Frasca, B., Kansal, S., & Pfeiffer, R. (2009). “Controlled Experiments on the Web: Survey and Practical Guide.” Data Mining and Knowledge Discovery, 18(1), 140–181. — the canonical reference on running controlled online experiments, covering randomization, sample size, and the statistical rigor needed to trust A/B test results in production.
• Beyer, B., Jones, C., Petoff, J., & Murphy, N. (2016). Site Reliability Engineering: How Google Runs Production Systems. O’Reilly Media. — covers progressive rollout strategies, canary deployments, and the monitoring infrastructure required to make safe rollouts at scale.

---