Feature Stores In Practice Feast And The Problem T

The Nightmare Before Deployment

Picture this: it’s 2 AM. Your phone buzzes with a PagerDuty alert. The fraud detection model you deployed last week is suddenly letting through transactions it should be blocking. The precision has cratered — down 20%. You roll back to the old model, and the problem disappears. But here’s the thing: the new model was better. In the notebook, on the holdout set, on every backtest you ran, it outperformed the old one by a mile. So what happened?

You spend the next three days debugging. You check the model code. It’s fine. You check the serving infrastructure. It’s fine. Then, deep in a Jupyter notebook from six weeks ago, you find it. The feature you called user_transactions_last_1hr was computed one way in your training pipeline — counting transactions from a batch ETL job that ran hourly. In production, the serving pipeline computes it in real time from a Kafka stream, and there’s a subtle difference in how it handles transactions that are “in flight” at the hour boundary. The feature has the same name, but it’s not the same value. Your model was trained on apples; it’s being served oranges.

This is training-serving skew. And if you’ve been doing machine learning long enough, you’ve probably been bitten by it. The root cause isn’t the model — it’s the silent divergence between the offline world where we train and the online world where we serve. How many times have you debugged a model only to find the feature pipeline was the real culprit?

This is the exact problem feature stores were invented to solve. And here’s what’s easy to miss: a feature store isn’t really about storing features. It’s about guaranteeing that the features you train on are the exact same features you serve with, everywhere, every time.

What a Feature Store Actually Is (in Plain English)

Let’s build an intuition before we touch any code or architecture diagrams. Think of a feature store as a librarian for your machine learning features. You don’t walk into a library, find the shelf for “user behavior,” and hope the book you pull off is the right edition. You ask the librarian for a specific book by name, and they bring you the authoritative copy — the same one everyone else gets.

A feature store does the same thing. You ask for a feature by name — say, user_transactions_last_1hr — and it hands you the exact same value whether you’re training a model on last year’s data or serving a prediction for a user right now. It doesn’t matter if the training pipeline runs in BigQuery and the serving pipeline hits Redis. The librarian makes sure the answer is consistent.

This consistency rests on three core promises. First, consistency across environments — the feature value is identical in training and serving. Second, no data leakage — the store handles point-in-time correct joins so you never accidentally use future information to predict the past. Third, decoupling — data scientists don’t need to know where the feature lives or how it’s computed; they just ask for it by name.

Now, the librarian has two “shelves.” The offline store is the big, slow, cheap shelf — it holds months or years of historical feature values for training and batch scoring. Think of it as the warehouse. The online store is the small, fast, expensive shelf — it holds the latest feature values for low-latency serving, like a Redis cache. The feature store keeps them in sync.

So here’s the plain-English definition: a feature store is a system that makes sure the features you train on are the features you serve with. That’s it. Everything else — the registries, the APIs, the point-in-time joins — is in service of that one promise.

The Feature Store Hierarchy of Needs

Now here’s the interesting part: not everyone who says “feature store” means the same thing. Some people mean a shared spreadsheet of feature definitions. Others mean a full platform that computes, stores, and serves features automatically. To make sense of this, we can borrow a framework from the excellent post on applyingml.com: the Feature Store Hierarchy of Needs.

Think of it like Maslow’s hierarchy, but for features. There are five tiers, and each one solves a deeper problem.

Tier 1: Access. This is the most basic need — a shared catalog so data scientists stop duplicating each other’s work. Instead of five people writing five different queries to compute “user average purchase value,” they all use the same feature definition. Even a wiki page counts here.

Tier 2: Serving. Now you need to get those features into production. The offline store serves features for training; the online store serves them for real-time predictions. Consistency between the two starts to matter.

Tier 3: Integrity. This is where the real pain hits. Training-serving skew and point-in-time correctness live here. If your model trains on features that don’t match what it sees in production, your metrics are lying to you. This is the tier where a feature store stops being a nice-to-have and becomes essential.

Tier 4: Convenience. At this tier, the feature store doesn’t just serve features — it computes them too. You define a feature once, and the platform handles the pipelines to generate it for both offline and online use. Uber’s Michelangelo and Airbnb’s Chronon live here.

Tier 5: Autopilot. The holy grail. The system automatically generates features, detects drift, and retrains models. Almost no one is here yet.

This is the hardest part of understanding the feature store landscape: “feature store” means different things to different people. Feast, the open-source tool we’re going to explore, primarily operates at Tiers 1 through 3. It stores and serves features with integrity guarantees. It does not compute features or orchestrate pipelines. That’s not a flaw — it’s a deliberate design choice, as we’ll see.

The Giants Before Feast: Michelangelo and Zipline

The feature store concept wasn’t invented by Feast. It was born from the extreme scale of companies like Uber and Airbnb, where hundreds of data scientists were building thousands of models and stepping on each other’s toes constantly.

Uber’s Michelangelo platform, described in a 2017 blog post, managed around 10,000 features shared across teams. The key innovation wasn’t the storage technology — it was the canonical feature name. Every feature had a single, authoritative name and an owner, with a service-level agreement (SLA) attached. If you needed “driver_rating_7day_avg,” you didn’t write a query. You called it by its canonical name, and Michelangelo served it from the appropriate store — HDFS for training, Cassandra for online serving. The metadata was the product.

The operational pain behind this was real. In 2021, Uber shared a post-mortem about an incident caused by inadequate schema validation in their feature metadata system. A schema change propagated incorrectly, and downstream models broke silently. The fix — a new metadata store called Palette — cut metadata update time from hours to minutes and added rigorous validation. This is the kind of problem you only encounter when thousands of models depend on shared features.

Around the same time, Airbnb built Zipline, which later evolved into the open-source project Chronon. The core insight was “define once, power both offline training and online inference.” The motivation, as Varant Zanoyan put it, was that engineers were spending the majority of their time managing data, not modeling. Zipline let you define a feature once, and it would automatically generate the pipelines to compute it for both batch training and real-time serving.

But here’s the catch: these systems were deeply integrated with their companies’ infrastructure. Michelangelo assumed you had HDFS and Cassandra. Zipline assumed you had Hive and Kafka. They were not open-source, and they were not portable. If you were a 20-person startup, you couldn’t just install Michelangelo.

This sets the stage for Feast: an attempt to bring these ideas to everyone else, without requiring Uber-scale infrastructure.

Enter Feast: The Open-Source, Infrastructure-Agnostic Feature Store

Feast was born in 2018 from a collaboration between GO-JEK, the Indonesian ride-hailing giant, and Google Cloud. The problem at GO-JEK was rampant duplication of feature engineering — multiple teams were writing the same feature pipelines over and over, with subtle inconsistencies. They needed a shared feature catalog and serving layer, but they didn’t want to be locked into a specific infrastructure stack.

The name “Feast” is a pun on “Feature Store” — it’s meant to be a feast of features, not a famine. And from day one, its architecture was deliberately decoupled. Feast doesn’t replace your data warehouse or your cache. It plugs into them. Your offline store might be BigQuery or Redshift. Your online store might be Redis or Datastore. Feast sits on top, providing a consistent API and metadata layer.

There are three core components. The Offline Store holds historical feature data for training and batch scoring. The Online Store holds the latest feature values for low-latency serving. And the Registry is the brain — it’s a metadata catalog that stores the definitions of all your features, entities, and feature views. The Registry is what makes consistency possible. When you ask Feast for a feature, it consults the Registry to know exactly where to find it and how to join it.

The killer feature — the one that makes Feast worth the setup — is point-in-time correct joins. This is how Feast prevents data leakage automatically. And it’s so important that it deserves its own section.

Point-in-Time Correctness: The Hardest Problem You’ve Never Heard Of

Let’s set up a concrete scenario. Imagine you’re building a model to predict customer churn. You have a dataset of users, and for each user, you have a timestamp of when you made the prediction — say, 9:00 AM on a Tuesday. You also have a table of purchases, with timestamps. You want to use “number of purchases in the last 24 hours” as a feature.

A naive approach: you join the user table with the purchases table on user ID, filter to purchases in the last 24 hours, and count. But wait — what does “last 24 hours” mean? If you compute it relative to now, you’re in trouble. A user who made a purchase at 10:00 AM (an hour after your 9:00 AM prediction) would be included in the count. Your model would be trained on data from the future. That’s data leakage.

This is the hardest part of feature engineering for temporal problems: you must join features as they existed at the time of the prediction, not as they exist now. Without this, your model’s validation metrics are lying to you. They look better than they are because the model has seen the future.

Feast’s point-in-time correct join does this automatically. You provide an entity (like a user ID) and a timestamp, and it retrieves the features as they were at that exact moment. It ensures that no event that happened after the timestamp can leak into the feature value.

Let’s see what happens with a concrete code example. We’ll simulate a naive pandas merge that leaks data, and then show how Feast prevents it.

import pandas as pd

# Simulate a user prediction event at 9:00 AM
prediction_events = pd.DataFrame({
    "user_id": [1],
    "prediction_timestamp": [pd.Timestamp("2024-01-15 09:00:00")]
})

# Simulate a purchase history. Notice the purchase at 10:00 AM — AFTER the prediction.
purchases = pd.DataFrame({
    "user_id": [1, 1],
    "purchase_timestamp": [
        pd.Timestamp("2024-01-15 08:30:00"),  # Before prediction
        pd.Timestamp("2024-01-15 10:00:00")   # AFTER prediction — this is the future!
    ]
})

# Naive merge: join on user_id, ignoring timestamps.
# This will include the 10:00 AM purchase.
naive_merge = prediction_events.merge(purchases, on="user_id")
print("Naive merge — all purchases, including future ones:")
print(naive_merge)

# Now let's do a point-in-time correct join manually.
# We only keep purchases that happened BEFORE the prediction timestamp.
correct_merge = naive_merge[
    naive_merge["purchase_timestamp"] < naive_merge["prediction_timestamp"]
]
print("\nPoint-in-time correct — only purchases before the prediction:")
print(correct_merge)

# Count purchases for the feature.
naive_count = naive_merge.groupby("user_id").size().reset_index(name="purchase_count_24h")
correct_count = correct_merge.groupby("user_id").size().reset_index(name="purchase_count_24h")

print("\nNaive feature value (leaked future data):")
print(naive_count)
print("\nCorrect feature value (no leakage):")
print(correct_count)

Look at the output. The naive merge gives a purchase count of 2 — it includes the 10:00 AM purchase that hadn’t happened yet at 9:00 AM. The correct join gives a count of 1. If you train a model on the naive feature, it learns patterns that don’t exist in reality. When you deploy it, the feature in production won’t have that future information, and the model’s performance will drop. Your validation metrics were inflated.

This is the single biggest reason to adopt a feature store, even if you’re not at Uber scale. Point-in-time correctness isn’t a nice-to-have. For any temporal prediction problem — fraud, churn, recommendations — it’s the difference between a model that works and a model that silently fails.

Feast in Practice: A Minimal Working Example

Now let’s walk through a complete, runnable Feast setup. We’ll define a feature, register it, retrieve it for training with point-in-time correctness, materialize it to an online store, and retrieve it for serving — all from the same feature definition. This is the “define once, use everywhere” pattern in action.

First, we need to set up a minimal Feast project. We’ll use a local file-based offline store and a SQLite online store so you can run this without any external infrastructure.

# This block sets up the Feast environment and creates the configuration files.
# In a real project, you'd run 'feast init' from the CLI, but we'll do it programmatically here.

import os
import yaml
from pathlib import Path

# Create a temporary directory for our Feast project
project_dir = Path("./feast_example")
project_dir.mkdir(exist_ok=True)

# Write the feature_store.yaml configuration
feast_config = {
    "project": "churn_prediction",
    "provider": "local",
    "registry": str(project_dir / "registry.db"),
    "offline_store": {
        "type": "file"
    },
    "online_store": {
        "type": "sqlite",
        "path": str(project_dir / "online_store.db")
    },
    "entity_key_serialization_version": 2
}

with open(project_dir / "feature_store.yaml", "w") as f:
    yaml.dump(feast_config, f)

print("Feast project directory created at:", project_dir)

Now let’s create some sample data. We’ll simulate a table of user transaction counts, with timestamps, that we can use as a feature source.

import pandas as pd
from datetime import datetime, timedelta

# Create sample transaction data for two users over a few hours
base_time = datetime(2024, 1, 15, 8, 0, 0)
data = []
for user_id in [1, 2]:
    for hour_offset in range(5):
        ts = base_time + timedelta(hours=hour_offset)
        # User 1 has more transactions than User 2
        tx_count = 5 + hour_offset if user_id == 1 else 2 + hour_offset
        data.append({
            "user_id": user_id,
            "event_timestamp": ts,
            "transactions_last_hour": tx_count
        })

transaction_features = pd.DataFrame(data)

# Save as parquet — Feast's FileSource expects parquet files
data_dir = Path("./feast_example/data")
data_dir.mkdir(exist_ok=True)
transaction_features.to_parquet(data_dir / "transaction_features.parquet")

print("Sample transaction data:")
print(transaction_features.head(10))

Now we define our Feast objects: an Entity (the thing we’re making predictions about), a FeatureView (a logical group of features), and a FileSource (where the data lives).

from feast import Entity, FeatureView, Field, FileSource
from feast.types import Int64, UnixTimestamp
from datetime import timedelta

# Define the entity: a user. This is what we'll join features to.
user = Entity(
    name="user",
    join_keys=["user_id"],
    description="A user in our churn prediction system"
)

# Define the data source: our parquet file.
transaction_source = FileSource(
    path=str(data_dir / "transaction_features.parquet"),
    timestamp_field="event_timestamp",
    created_timestamp_column="event_timestamp"
)

# Define the FeatureView: a group of features that share an entity and a source.
transaction_fv = FeatureView(
    name="user_transactions",
    entities=[user],
    ttl=timedelta(days=7),  # How long features live in the online store
    schema=[
        Field(name="transactions_last_hour", dtype=Int64),
    ],
    source=transaction_source,
    description="User transaction features for churn prediction"
)

print("Entity defined:", user.name)
print("FeatureView defined:", transaction_fv.name)

With our definitions ready, we apply them to the Feast registry. This is the “define once” step — the registry now knows about our feature and where to find it.

from feast import FeatureStore

# Initialize the feature store from our configuration
store = FeatureStore(repo_path=str(project_dir))

# Apply the definitions to the registry
store.apply([user, transaction_fv])

print("Registry updated. Features registered:")
for fv in store.list_feature_views():
    print(f"  - {fv.name}: {[f.name for f in fv.schema]}")

Now let’s retrieve historical features for training. We’ll create a DataFrame of “prediction events” — user IDs and timestamps — and ask Feast to join the features as they existed at those exact moments.

# Create prediction events: we want features for User 1 at 9:00 AM and User 2 at 11:00 AM
prediction_events = pd.DataFrame({
    "user_id": [1, 2],
    "event_timestamp": [
        pd.Timestamp("2024-01-15 09:00:00"),
        pd.Timestamp("2024-01-15 11:00:00")
    ]
})

print("Prediction events (what we want features for):")
print(prediction_events)

# Retrieve historical features with point-in-time correctness
historical_features = store.get_historical_features(
    entity_df=prediction_events,
    features=store.get_feature_service("user_transactions") if False else [
        "user_transactions:transactions_last_hour"
    ]
).to_df()

print("\nHistorical features retrieved (point-in-time correct):")
print(historical_features)

# Interpret the output:
# At 9:00 AM, User 1 had a transactions_last_hour value from the 8:00 AM data point (5).
# At 11:00 AM, User 2 had a transactions_last_hour value from the 10:00 AM data point (4).
# Feast automatically selected the most recent value BEFORE each prediction timestamp.

Look at the output. For User 1 at 9:00 AM, Feast retrieved the feature value from the 8:00 AM data point — the most recent value before the prediction timestamp. It did not use the 9:00 AM or 10:00 AM data, because those hadn’t happened yet at 9:00 AM. For User 2 at 11:00 AM, it used the 10:00 AM value. This is point-in-time correctness in action.

Now for the online side. We need to materialize the latest feature values into the online store so we can retrieve them with low latency during serving.

from datetime import datetime

# Materialize the latest feature values into the online store.
# In production, you'd run this on a schedule to keep the online store fresh.
materialization_start = datetime(2024, 1, 15, 0, 0, 0)
materialization_end = datetime(2024, 1, 15, 23, 59, 59)

store.materialize(
    start_date=materialization_start,
    end_date=materialization_end
)

print("Features materialized to online store.")

Finally, let’s retrieve online features for a real-time prediction. This simulates what happens when your model serving endpoint asks for features for a specific user right now.

# Retrieve online features for User 1 at the current moment
online_features = store.get_online_features(
    features=[
        "user_transactions:transactions_last_hour"
    ],
    entity_rows=[
        {"user_id": 1}
    ]
).to_dict()

print("Online features retrieved for serving:")
print(online_features)

# Interpret the output:
# The feature value for User 1 is the latest materialized value.
# This is the same value that would have been used in training for a prediction
# at the same timestamp. Consistency guaranteed.

The feature value you see — say, 9 — is the number of transactions in the last hour for User 1, as of the latest materialized timestamp. This is the exact same value that would have been used in training if you had a prediction event at that same moment. The online store and the offline store are in sync because they both serve from the same FeatureView definition. That’s the promise of a feature store, delivered.

What Feast Doesn’t Do (and Why That’s Okay)

Feast is powerful, but it’s not a magic wand. Being honest about its limitations is how you decide if it’s the right tool for your team.

First, Feast does not compute features. It stores and serves them. You still need a pipeline — Spark, dbt, a SQL script, whatever — to generate the features in the first place and write them to the offline store. Feast will serve them consistently, but it won’t build them for you.

Second, Feast does not orchestrate. It doesn’t schedule feature generation or manage dependencies between pipelines. That’s Airflow’s job, or Dagster’s, or Prefect’s. Feast expects you to have an orchestrator that runs your feature pipelines and calls materialize() on a schedule.

Third, Feast does not provide a UI for discovery. The registry is a Python API (and a protobuf-based file or database). If you want a web dashboard where data scientists can browse features, see statistics, and understand lineage, you’ll need to build it yourself — or use a platform like Tecton that builds one on top of Feast.

This is by design. Feast is a component, not a platform. It’s the feature storage and serving layer, and it expects to be plugged into a larger MLOps stack. As Chip Huyen put it, Feast is not a “feature platform” — it doesn’t handle computation, only storage and retrieval.

When you need more, there are options. Tecton is a managed service built on Feast that adds feature computation, orchestration, and a UI. Hopsworks is a full-featured platform with its own feature store, model registry, and UI. Chronon (from Airbnb) handles both computation and serving, but it’s deeply tied to Spark and Kafka — if your infrastructure looks like Airbnb’s, it’s a great fit. If it doesn’t, Feast’s agnosticism is a feature, not a bug.

When Do You Actually Need a Feature Store?

Not every team needs a feature store. Adopting one too early adds complexity without benefit. So how do you know?

You need a feature store when three things are true. First, multiple models reuse the same features. If you have one model, a well-governed feature table in your data warehouse is probably enough. The pain starts when five teams are writing five different queries for “user_average_purchase_value” and getting five slightly different answers.

Second, you’ve been burned by training-serving skew. If you’ve ever debugged a production model only to find the feature pipeline was the culprit, you know the pain. A feature store’s consistency guarantee is the cure.

Third, you need point-in-time correct training data for temporal problems. If you’re predicting something that happens at a specific moment — fraud, churn, click-through — and you’re joining features from multiple time-based sources, point-in-time correctness isn’t optional. It’s the difference between a model that works and one that silently fails.

You don’t need a feature store if you have a single model, your features are simple and stateless, or your team is small and the overhead of a new system outweighs the benefit. A common sentiment on r/mlops is: start with a well-governed feature table in your data warehouse. When the pain of duplication and inconsistency becomes real, then adopt Feast.

The Xebia post “You Still Don’t Need A Feature Store” makes a healthy counterpoint: feature stores solve organizational problems at scale, not technical problems for individuals. If you’re the only data scientist, a feature store is probably overkill. If you’re on a team of 20, with five models in production and three more in development, the equation changes.

---

Let’s recap what we’ve learned. Training-serving skew is the silent killer of production ML models — your model trains on one version of a feature and serves on another. A feature store solves this by acting as a librarian: you define a feature once, and it guarantees the same value everywhere. Feast, the open-source feature store, brings this capability to any infrastructure stack, with point-in-time correct joins as its killer feature. It doesn’t compute features or orchestrate pipelines — it’s a component, not a platform — and that’s exactly what makes it fit into existing MLOps stacks. You need one when duplication and inconsistency are costing you more than the overhead of a new system.

Check Your Understanding

Remember: What are the two “shelves” in a feature store, and what is each used for?

Understand: In your own words, explain why a naive join between a predictions table and a features table can cause data leakage in temporal problems.

Apply: You have a fraud detection model that uses user_transactions_last_1hr as a feature. Write a Python snippet using Feast’s get_historical_features to retrieve this feature for a set of prediction events, ensuring point-in-time correctness.

Analyze: A teammate argues, “We don’t need a feature store — we just write features to a shared BigQuery table and everyone queries it.” What problems might this approach fail to solve as the team and model count grows?

Evaluate: Compare Feast and a full feature platform like Tecton or Chronon. Under what circumstances would you recommend Feast over a managed platform, and when would you recommend the opposite?

Create: Design a minimal feature store architecture for a 10-person data science team building three models that all use a common set of user behavior features. What components would you include? What would you deliberately leave out?