A feature can be accurate and still arrive too late.

Start with the moment of the decision

A neighborhood repair desk wants to prioritize appointments likely to be cancelled. The prediction runs at 09:00 on the appointment day. The outcome is whether the appointment is cancelled before the desk closes. The first contract is temporal: every input must be available to the service at 09:00. A column being present in a historical table does not make it a legitimate feature.

Here is a fictional audit sample. It illustrates information timing; it is not a model benchmark or customer data.

appointment	reminder sent by 09:00	refund recorded at	cancelled by close
R1	yes	17:10	yes
R2	no	none	no
R3	yes	16:40	yes
R4	no	none	no

A rule that predicts cancellation whenever a refund exists scores four correct rows out of four in this tiny table. It is unusable at 09:00 because the informative refunds have not happened. The problem is not excessive model complexity. The rule is learning from the future.

Build an availability ledger

For every candidate feature, record its definition, event time, ingestion time and earliest time the scoring service can retrieve it. A reminder sent at 08:50 but ingested at 09:15 is also unavailable for the 09:00 decision. Reconstruct features using the same availability rule during training and evaluation. If historical ingestion times are missing, state that limitation rather than silently treating event time as proof of availability.

A useful audit asks two different questions: could the feature exist by the decision, and could this particular service access it then? Permissions, synchronization delays and refresh schedules affect the second answer. Offline access to a warehouse can be broader than access available to the deployed scorer.

Split before learning a transformation

Suppose two training measurements are 2 and 4, and the held-out measurement is 8. Using the population convention, training mean is 3 and standard deviation is 1. The held-out standardized value is therefore 5. Computing the mean across all three observations changes the transformation using evaluation information.

Fit imputation, scaling and feature selection on the training partition. Apply their fitted parameters to validation and test partitions. Save those parameters with the model artifact. A pipeline can enforce this boundary, but it cannot determine whether a column was knowable at prediction time. Both audits are needed.

Choose the split that answers the deployment question

A random row split can place appointments from the same customer in both partitions. If deployment serves entirely new customers, that split may answer the wrong question. A later-time evaluation can instead test future appointments for known customers. A customer-held-out evaluation tests generalization to unseen customers. Neither is universally correct: define the intended population and document the chosen boundary.

Inspect duplicates and repeated entities after splitting. A duplicated record with a new row identifier is still a duplicated observation. Keep selection decisions on validation data; repeatedly choosing features after inspecting test results makes the test set part of development.

Create a release checklist with observable evidence

Keep a versioned feature ledger, an example reconstructed at the scoring cutoff, the split rule, fitted preprocessing parameters and a baseline result. Add a deliberately late feature to a test fixture and confirm that the feature builder excludes it. Test an eligible but missing input and define whether scoring falls back or declines to predict.

After removing leakage, a lower evaluation score can be a more honest result. Do not restore a future-derived feature to recover an attractive number. Investigate what information can legitimately improve the decision and measure it on the intended population. This four-row example proves the timing contradiction, not the quality or fairness of a real cancellation model.