Before Building a Prediction Model, Write Down When Each Feature Exists

A beginner data science project can accidentally answer an easier question than the one it claims to solve. One way this happens is using information that would not exist when a prediction is needed.

Consider a fictional delivery project. You want to predict, at order checkout, whether a parcel will arrive late. Before choosing an algorithm, write down exactly what you know at checkout.

Start with the prediction moment

“At checkout” is part of the problem definition. An estimate made after a courier has attempted delivery is a different task. Both might be useful, but their inputs and evaluation should not be mixed.

For each proposed feature, record its meaning, source, availability time and whether it can change later. A short feature inventory can catch a problem that a high accuracy score hides.

Classify four candidate features

Destination region: usable if the customer has provided it by checkout.

Basket item count: usable if it describes the basket at checkout, rather than the final amended order.

Actual delivery timestamp: unavailable at checkout. It belongs to the later outcome and must not be an input for this prediction.

Final delivery status: also unavailable at checkout. If it says “delivered late,” it effectively reveals the answer.

Historical aggregates need a cutoff too

A feature such as “average delay for this region” sounds reasonable. But if it uses deliveries that happened after the order being predicted, it brings future information into the past.

For a historical simulation, calculate that value from records available before the prediction time. Some older orders may not yet have completed, so their eventual outcomes would also be unavailable. Document how you handle that delay.

Separate the target from the features

The target can be determined after delivery because it is the outcome you want to predict. Define “late” consistently, for example as arrival after the promised deadline recorded at checkout. Do not quietly replace that deadline with a later revised promise without explaining the change.

Use synthetic records for this exercise. Real delivery data can contain addresses and other personal information that do not belong in a public portfolio.

Build an evaluation that matches the use

If the intended use is predicting future orders, reserve a later period for evaluation and develop the model using earlier data. Keep the final evaluation period out of model selection. Fit learned preprocessing on the training data, then apply it to the evaluation data.

Also consider repeated entities. A split that places very similar records from the same entity in both groups can give an overly easy test if the intended use is predicting for previously unseen entities. The correct split depends on the deployment question.

A time split alone is not a guarantee against leakage. A future-derived feature remains invalid even when rows are separated by date.

What to include in a portfolio

Show the prediction question, feature availability inventory, target definition and training and evaluation periods. Include one rejected feature and explain why you excluded it. Compare the model with a simple baseline and describe limitations instead of presenting a single score as proof of usefulness.

AI can suggest suspicious features, but it cannot establish when your source system recorded them. Check that timing against the data documentation and your own assumptions.

Published by ScholarLogic, an IT training provider in Bengaluru. Related Data Science learning path: https://scholarlogic.net/data-science-course-in-bangalore/

Prepared with AI assistance. This is a proposed exercise using fictional delivery records, not a claim about a deployed model.