Skip to content

Tabular Feature Engineering

What This Is

Feature engineering for tabular data turns raw columns into representations that help models find signal. On tabular benchmarks, it is often the single most impactful step — more so than model selection. The decision this page forces is narrow:

  • which features are worth creating, and how do you create them without leaking the target

Most tabular feature-engineering bugs are not complexity bugs; they are leakage bugs.

When You Use It

  • raw features are not expressive enough for the model to learn patterns
  • domain knowledge suggests interactions, ratios, or aggregations
  • the baseline plateaus and you suspect representation is the bottleneck
  • you have high-cardinality categoricals (merchant IDs, zip codes, user IDs)
  • a boosted-tree baseline is strong but an extra few percentage points would matter

Do Not Use It When

  • the model is a modern deep tabular architecture that handles categoricals natively (TabNet, FT-Transformer) — let it learn
  • the dataset is tiny and new features would add variance without enough rows to support them
  • a leakage-prone encoding (target encoding) is needed but the CV infrastructure is not in place
  • a feature matches a column that will not be present at inference time (see the leakage section below)

Tooling

Tool What it does
PolynomialFeatures interaction and polynomial terms
KBinsDiscretizer bins continuous features into categories
FunctionTransformer wraps custom transformations in a pipeline
ColumnTransformer applies different transforms to different columns
pandas.cut / pandas.qcut manual binning by value or quantile
groupby().transform() group-level aggregation as new features
category_encoders.TargetEncoder target / mean encoding with smoothing
category_encoders.CountEncoder count / frequency encoding

Feature Engineering Ladder

  1. Clean — handle missing values, fix types (see Data Cleaning and Preprocessing)
  2. Encode — one-hot, ordinal, target, or count encoding for categoricals
  3. Scale — standardize or normalize numeric features for linear models / kNN
  4. Combine — interactions, ratios, differences
  5. Aggregate — group-level statistics (mean, count, std per category)
  6. Bin — convert continuous to categorical where the nonlinearity is useful
  7. Select — drop low-importance or redundant features

Categorical Encoding — The Big Decision

Encoding Works on Wins when Breaks when
one-hot low cardinality (≤ ~50) linear models, small categorical sets cardinality > 100 — explodes dimensions
ordinal / label ordered categories actual ordering exists (XS/S/M/L/XL) nominal categories — imposes false order
count / frequency any cardinality tree models, rare-category indicator two rare categories collide to the same count
target / mean any cardinality tree models, high-cardinality keys without honest CV, it leaks directly
hashing very high cardinality memory-bound, streaming collisions are uninterpretable
embeddings any cardinality deep models over-parameterized for small data

Rule of thumb: one-hot is the safe default for low-cardinality nominal features; target/count encoding shine at high cardinality but must be computed inside the CV loop.

Target Encoding With Honest CV

Target encoding replaces a category with the mean target value among rows in that category. It is powerful for high-cardinality IDs and leakage-prone if done naively.

Wrong:

# LEAKS — computes encoding on the full training set before any CV split.
df["city_enc"] = df.groupby("city")["target"].transform("mean")

Every row sees its own target in the mean. The model memorizes the encoding, cross-validation scores look wonderful, and the deployed model collapses.

Right — out-of-fold target encoding:

from sklearn.model_selection import KFold
import numpy as np
import pandas as pd

def oof_target_encode(df, col, target, n_splits=5, smoothing=10):
    global_mean = df[target].mean()
    enc = pd.Series(index=df.index, dtype="float64")
    kf = KFold(n_splits=n_splits, shuffle=True, random_state=0)
    for tr_idx, va_idx in kf.split(df):
        tr = df.iloc[tr_idx]
        counts = tr.groupby(col).size()
        means  = tr.groupby(col)[target].mean()
        # smoothed mean: shrinks small-sample categories toward global mean
        smoothed = (counts * means + smoothing * global_mean) / (counts + smoothing)
        enc.iloc[va_idx] = df.iloc[va_idx][col].map(smoothed).fillna(global_mean)
    return enc

df["city_enc"] = oof_target_encode(df, col="city", target="y")

For test-time inference, fit the encoding once on the full training set (now allowed, because the test set is disjoint) and apply the same map.

Smoothing matters: a category with 3 rows should not overpower the global mean, so the smoothing term alpha pulls low-count categories toward the overall average.

Count / Frequency Encoding

A cheaper, leakage-free alternative for high-cardinality IDs:

counts = X_train["merchant_id"].value_counts()
X_train["merchant_count"] = X_train["merchant_id"].map(counts)
X_valid["merchant_count"] = X_valid["merchant_id"].map(counts).fillna(0)

It encodes "how common is this category" — a surprisingly strong feature for fraud, retail, and anomaly tasks. Use alongside (not instead of) target encoding when both fit.

Interaction Features

df["rooms_per_person"]   = df["total_rooms"] / df["population"].clip(lower=1)
df["income_x_rooms"]     = df["median_income"] * df["total_rooms"]
df["income_minus_mean"]  = df["median_income"] - df.groupby("region")["median_income"].transform("mean")

For tree models, many explicit interactions are redundant — trees find splits on the raw features. For linear models, interactions are often the only way to recover a nonlinear signal. PolynomialFeatures(degree=2, interaction_only=True) produces all pairwise products in one line; use it on tens of features, not hundreds.

Date Decomposition

df["day_of_week"] = df["date"].dt.dayofweek
df["month"]       = df["date"].dt.month
df["is_weekend"]  = df["day_of_week"].isin([5, 6]).astype(int)
df["days_since_signup"] = (df["event_date"] - df["signup_date"]).dt.days

For cyclical features (hour of day, day of week), sin/cos encoding preserves the wrap-around:

df["hour_sin"] = np.sin(2 * np.pi * df["hour"] / 24)
df["hour_cos"] = np.cos(2 * np.pi * df["hour"] / 24)

Group-Level Aggregations

# computed on training data only; applied to validation/test via a map
train_means  = X_train.groupby("category")["score"].mean()
train_counts = X_train.groupby("category")["score"].count()

X_train["group_mean_score"] = X_train["category"].map(train_means)
X_train["group_count"]      = X_train["category"].map(train_counts)
X_train["score_vs_group"]   = X_train["score"] - X_train["group_mean_score"]

for X in [X_valid, X_test]:
    X["group_mean_score"] = X["category"].map(train_means).fillna(X_train["score"].mean())
    X["group_count"]      = X["category"].map(train_counts).fillna(0)
    X["score_vs_group"]   = X["score"] - X["group_mean_score"]

The key line is .groupby(...) on the training set only and then .map onto validation/test. If you run df.groupby(...) on the full dataframe, the validation rows' statistics leak.

Binning

# equal-frequency bins — each bucket has ~20% of rows
df["income_bin"] = pd.qcut(df["income"], q=5, labels=False, duplicates="drop")

Binning throws away resolution but trades it for monotonicity and robustness to outliers. Useful when:

  • the feature-target relationship is piecewise (tax brackets, age groups)
  • the linear model cannot express the nonlinearity
  • outliers are destabilizing the model

Leakage Warnings — The Real Topic

Feature engineering is where leakage lives. A checklist:

  1. Target statistics in features. Target encoding, mean-by-group, "was this flagged later" — all leak unless computed strictly out-of-fold.
  2. Future-looking columns. days_until_churn, status_at_close, final_tier — if the column is populated after the event you are predicting, it cannot be an input.
  3. Global normalization. StandardScaler().fit_transform(X) on the full dataset before splitting — the scaler's mean/std see the test rows. Fix: fit on train, transform on both.
  4. Group statistics across splits. Group-by aggregations computed on the concatenated data, or on validation rows themselves.
  5. De-duplication or near-duplicate rows across splits. The same user (or near-duplicate) in train and test is implicit leakage. Use Honest Splits and Baselines — group-aware splits.
  6. Time-ordered data with random split. Features that carry temporal signal (rolling means, lag) must use time-ordered CV.

See Leakage Patterns for the systematic treatment.

What To Inspect

  • feature importance before and after — did the new feature land in the top?
  • permutation importance on validation — the honest check; tree-based importance can mislead
  • target encoding CV stability — scores per fold should not wobble more than the model gap
  • NaN and inf counts — ratio features with zero denominators, log of zero
  • category coverage — how many validation categories are unseen? If many, count encoding will default to zero and target encoding to the global mean
  • per-feature correlation with target on the train fold only — a correlation that spikes implausibly is a leakage signal
  • train vs. validation feature distribution — big shifts in a new feature suggest the engineering process introduced a train-only artifact

Failure Pattern

Creating hundreds of features without checking which ones actually help. Feature engineering should increase signal, not just dimensionality.

The leakage failure: computing group aggregations or target encodings on the full dataset before splitting. CV scores look wonderful; deployed model collapses. The only reliable fix is computing all target-aware features inside the CV loop.

The "future-looking column" failure: a column populated at event-close time gets used as an input to predict the event. This is the silent accuracy inflator in many business datasets. Audit the source of every feature; if the value is set after the target is known, drop it.

Common Mistakes

  • creating ratios that produce infinity or NaN when the denominator is zero
  • target encoding without out-of-fold CV — leaks the target directly
  • one-hot encoding high-cardinality features and drowning the model in sparse noise
  • forgetting to apply the same transformation pipeline to validation and test data
  • computing StandardScaler / target stats / group means on the concatenated dataset
  • generating hundreds of polynomial interactions without pruning, then wondering why the model overfits
  • using date components without checking whether the decomposition (day-of-week, hour) actually corresponds to a domain-meaningful cycle
  • letting new-to-validation categories default silently — always handle unseen keys explicitly

Decision: Which Encoding For A High-Cardinality Column

Situation Start with Escalate to
fewer than ~50 unique values one-hot target/mean encoding if the model is tree-based
50–10k unique values count encoding target encoding inside CV, with smoothing
10k+ unique values, many rare target encoding with strong smoothing learned embeddings in a deep tabular model
streaming or memory-bound hashing encoder hashing + count as a pair
ordered categories ordinal encoding ordinal + a small monotonic model constraint

Practice

  1. Create 3 interaction features from a dataset. Compare the model's CV score before and after; is the gain larger than the CV band?
  2. Decompose a date column into weekday, month, is_weekend, and days_since_signup. Identify which components carry signal via permutation importance.
  3. Implement out-of-fold target encoding. Compare it against in-sample target encoding — show the training-vs-validation gap inflate when leakage is present.
  4. On a high-cardinality column, compare one-hot, count, and target encoding on the same model. Report CV score, training time, and memory footprint.
  5. Add group-level mean and count features. Show that computing them on the full dataset leaks, and computing them on the training fold does not.
  6. Find one "future-looking" column in a dataset you care about. Explain in two sentences how it would fail at inference.
  7. Use PolynomialFeatures(degree=2, interaction_only=True) on ten features. Measure how many of the 45 generated interactions have nonzero importance.

Runnable Example

Longer Connection

Tabular feature engineering sits next to:

A feature is worth creating only if you can explain (a) which signal it encodes, (b) whether it will exist at inference time, and (c) whether it was computed without peeking at the target. Three sentences. If the student cannot write them, the feature is not yet earned.