Data preparation for ML

Read on realworldml.net

This is Part 2 of this mini-series on how to solve real-world business problems using Machine Learning.

Last week we covered Problem framing. Today, we will take the next step, and learn how to prepare the data.

Remember 🙋

The 4 steps to building a real-world ML product are

1. Problem framing (last week)

2. Data preparation (today) 📊

3. Model training (next week)

4. MLOps

Let’s get started!

Example

Imagine you work at a ride-sharing app company in NYC as an ML engineer. And you want to help the operations team allocate the fleet of drivers optimally each hour of the day. The end goal is to maximize revenue.

Last week you learned how to frame this business problem as an ML problem.

Problem framing 🖼️

We will build a predictive model for taxi demand. The model will predict how many rides will be requested

• on each area of NYC

• in the following 60 minutes

Before we can start building any ML model, we need to prepare the data.

Step 2. Data preparation

In real-world ML projects there is no Kaggle-like dataset with N columns for the features and 1 with the target. Instead, you have to create this dataset yourself, starting with raw data.

In this case, you have the list of taxi rides that have happened in NYC in the last 24 months, including

• the date and time of the ride, and

• the pickup location

This data is collected by the application backend and sent to data storage (aka data warehouse), where you can read it and use it for your ML service.

However, before doing so, you need to pre-process, by following these 3 steps:

1. Data validation.
   Remove wrong or buggy ride events. For example, test events generated by your development team that accidentally ended up in the production data.

2. Aggregation of events into time-series data.
   Your model will use historical one-hour data intervals

3. Transformation of time-series data into pairs (features, target)

How to transform time-series data into Supervised ML data?

Most Supervised ML models (e.g. XGBoost) do not work directly with time-series data. Instead, you need to pre-process time-series data, into pairs (features, target), where

• features are the model inputs

• target is the model output

To transform time-series data into (features, target) pairs you define

• a window length (e.g. last 12 hours) for the size of the input feature vector.

• a step size (e.g. 1 hour), to control the total number of samples.

and you apply a slice-and-dice operation.

Slice and dice (Image by the author)

Next steps

So far we have

✅ defined the ML problem to solve and

✅ generated our training data ready.

Next week, we will move on to step 3, aka model training.

Have a great weekend

And keep on learning!

Pau