You have heard models need good inputs, and now you are asking what is feature engineering. It is the work of creating and transforming the variables a model learns from, so it predicts better.
For the wider picture, start with the machine learning basics overview, then return here for the data-preparation step.
| Item | Value |
|---|---|
| Guide publication date | May 16, 2024 |
| Standardization output | Mean of 0 and standard deviation of 1 |
| Min-max scaling output | Fixed range, such as [0, 1] |
| Common imputation methods | Mean/median, mode and interpolation |
| Common categorical encodings | One-hot, label and target encoding |
These figures come from the KDnuggets guide Feature Engineering for Beginners, published May 16, 2024. Methods and recommendations can change, so check the source before you rely on them.
What features are and why raw data is not enough
A feature is a measurable characteristic of whatever you are observing. Typical examples are age, income, a timestamp or longitude, according to the KDnuggets guide. A model never sees your business, only these columns.
The same guide names four main feature types:
- Numerical: continuous or discrete numbers, such as age or salary.
- Categorical: qualitative values that represent categories.
- Text: words or strings of words.
- Time series: data ordered by time, such as stock prices.
Raw data is rarely ready to use. The Train in Data blog says data in its original format can almost never be used straightaway to train classification or regression models. It defines feature engineering as transforming variables, and extracting and creating new variables from the original data points, to train machine learning models.
The same post adds that the work is very time-consuming, which is why it pays to know the standard moves.
Core feature engineering techniques
Most projects begin with a handful of core techniques. The KDnuggets guide groups the basics into handling missing values, encoding categories and scaling numbers.
Handling missing values
Gaps in a dataset can hurt a model, so you need a plan for them. The guide lists these common imputation methods:
- Mean or median imputation fills gaps with the mean or median of the column.
- Mode imputation fills gaps with the most common entry in the same column.
- Interpolation fills gaps using the values of the data points around them.
Pick the method based on the nature of the data and its likely effect on the final model. The Train in Data post adds that the share of missing data and the model you plan to train also shape the choice.
Encoding categorical variables
Most algorithms work best, or only, with numbers, so categories must be mapped to numeric values. The KDnuggets guide lists three common encoding schemes:
- One-hot encoding produces a separate column for each category.
- Label encoding assigns an integer to each category.
- Target encoding encodes categories by their outcome-variable averages.
Some libraries cannot take these raw inputs at all. Train in Data notes that some machine learning libraries do not support missing values or categorical data as inputs, naming scikit-learn as an example.
Scaling and normalizing numbers
Scaling puts features on comparable ranges so one column does not swamp the others. The KDnuggets guide describes three scaling options:
- Standardization transforms data to a mean of 0 and a standard deviation of 1.
- Min-max scaling squeezes data into a fixed range, such as [0, 1].
- Robust scaling uses the median and interquartile range.
Not every model needs this step. A university statistical computing lesson says linear models such as logistic regression generally benefit from scaling, while tree-based models do not need such preprocessing but will not suffer from it. Train in Data lists linear regression, support vector machines, neural networks and nearest-neighbor methods as sensitive to variable scale.
Picture a model that compares customers by height in meters and income in dollars. Course notes on feature engineering principles point out that without scaling, income dominates the distance.
Transforming and binning
Skewed variables can be reshaped. Train in Data names the log, reciprocal and square root as the most commonly used variable transformations. It also describes discretization, sometimes called binning, as sorting the values of a variable into bins or intervals.
Feature engineering versus feature selection
The two terms are often confused. The KDnuggets guide defines feature selection as culling the important features from the full set of available features, which reduces dimensionality and promotes model performance.
Feature engineering, by contrast, creates new features and changes existing ones so the model performs better. In the guide’s words, selection leaves behind the signal in the data, while engineering creates features that help model the outcome better.
In practice you use both. You build candidate features first, then keep the ones that carry real signal.
What makes a feature trustworthy
A clever feature is useless if it cannot exist when the model runs. Course notes on feature engineering principles set three tests for the inputs you design:
- They are informative for the task.
- They respect the data-generating process.
- They are available at prediction time.
The third test is where data leakage creeps in. The same notes give an example: predicting 30-day hospital readmission at the time of discharge, using a patient-level readmission rate computed with a groupby over all rows in the dataset. That calculation includes the stay being predicted and future stays, which the notes call target leakage.
Leakage is easier to spot once you understand training data vs test data, since a transform fitted on the whole dataset has already seen the rows you meant to hold back. The notes say pipelines help prevent leakage because transforms are fit inside cross-validation folds, not on the whole dataset.
Tools and pipelines in practice
You rarely hand-code every step. Train in Data lists several Python libraries for data preparation:
- Pandas
- Scikit-learn
- Feature-engine
- Category Encoders
- tsfresh
- Featuretools
Some of them even automate the creation of features, according to the same post.
Scale matters too. Snowflake’s developer documentation, as one neutral example, describes three approaches: open-source preprocessors for small to medium datasets and quick prototyping, native preprocessing APIs for larger datasets, and Ray map_batches for highly customizable large-scale processing.
The course notes also explain why wiring steps into a pipeline helps:
- It prevents leakage.
- It keeps code honest, with no forgotten manual scaling in another cell.
- It makes experiments reproducible.
Teams that need these steps to run identically in production can read more in the guide on what is mlops.
Frequently asked questions
Is feature engineering the same as feature selection?
Show answer
No. According to the KDnuggets guide, selection culls the most useful features from those available, while engineering creates new features and changes existing ones.
Do all models need scaled features?
Show answer
No. A university lesson says linear models generally benefit from scaling, while tree-based models such as decision trees and random forests do not need it.
Why can’t I feed raw data straight into a model?
Show answer
Raw data usually has gaps, text categories and mixed scales. Train in Data says features in the raw data are almost never suitable inputs, so data scientists fill in missing data and transform categorical features into numbers.
What is target leakage?
Show answer
It happens when a feature carries information the model would not have at prediction time, such as outcomes from future records. The course notes show it with a readmission-rate feature computed over the whole dataset.
