The data project playbook

Before writing a single line of code, you need to understand what you're working with. These terms appear in every job description, paper, and codebase.

Types of Data

Before any technical work, define the business goal. What problem are you solving? How will the model's output be used? This step determines your target variable, evaluation metric, and whether ML is even the right tool.

Getting data from its source - files, databases, APIs, or cloud storage.

# Python (Pandas)
import pandas as pd

df = pd.read_csv('data.csv')                          # CSV file
df = pd.read_excel('data.xlsx')                       # Excel file
df = pd.read_sql('SELECT * FROM orders', conn)        # SQL database
df = pd.read_json('data.json')                        # JSON file
df = pd.read_parquet('data.parquet')                  # Parquet (columnar)

# PySpark
from pyspark.sql import SparkSession
spark = SparkSession.builder.appName("DataProject").getOrCreate()

df = spark.read.csv('s3://bucket/data.csv', header=True, inferSchema=True)
df = spark.read.parquet('s3://bucket/data.parquet')   # Preferred format at scale
df = spark.read.format("jdbc") \
    .option("url", "jdbc:postgresql://host/db") \
    .option("dbtable", "orders") \
    .option("user", "user").option("password", "pwd") \
    .load()                                           # JDBC database

Get a first look at the data's structure, basic properties, and obvious quality issues. A max age of 150 or a min salary of 0 should trigger immediate investigation.

# Python (Pandas)
df.info()             # Schema, dtypes, non-null counts
df.shape              # (rows, columns)
df.head(10)           # First 10 rows
df.sample(5)          # Random sample  - better for bias detection

# Numerical columns
df.describe()         # count, mean, std, min, 25%, 50%, 75%, max

# Categorical columns
df['column'].value_counts()          # Frequency distribution
df['column'].value_counts(normalize=True)  # As proportions

# Check for mixed types
df.dtypes                            # A numeric column read as 'object' = mixed types

# PySpark
df.printSchema()                            # Schema and types
print((df.count(), len(df.columns)))        # Shape equivalent
df.show(10)
df.sample(False, 0.01).show(5)             # ~1% random sample

df.describe().show()                        # Summary stats
df.groupBy('column').count().orderBy('count', ascending=False).show()

The most time-consuming phase - transforming raw data into a clean format suitable for analysis and modelling. Some cleaning must happen before EDA to avoid being misled.

Handling Missing Values

# Python (Pandas)  - Detection
df.isnull().sum()                          # Count nulls per column
df.isnull().mean() * 100                   # Null % per column

# Drop: only if nulls are few and missing at random
df.dropna(subset=['critical_column'])

# Impute numerical: prefer median (robust to outliers)
df['age'].fillna(df['age'].median(), inplace=True)

# Impute categorical: use mode or explicit "Unknown"
df['city'].fillna(df['city'].mode()[0], inplace=True)
df['city'].fillna('Unknown', inplace=True)

# PySpark
from pyspark.sql.functions import col, count, when, isnan

# Count nulls per column
df.select([count(when(col(c).isNull() | isnan(c), c)).alias(c) for c in df.columns]).show()

# Drop rows where specific column is null
df = df.dropna(subset=['critical_column'])

# Fill with a constant
df = df.fillna({'age': df.approxQuantile('age', [0.5], 0.01)[0]})

# Fill categorical with 'Unknown'
df = df.fillna({'city': 'Unknown'})

Outlier Detection & Treatment

# IQR method (Python)
Q1 = df['salary'].quantile(0.25)
Q3 = df['salary'].quantile(0.75)
IQR = Q3 - Q1
lower = Q1 - 1.5 * IQR
upper = Q3 + 1.5 * IQR

# Identify outliers
outliers = df[(df['salary'] < lower) | (df['salary'] > upper)]

# Option 1: Remove (only if data entry error)
df = df[(df['salary'] >= lower) & (df['salary'] <= upper)]

# Option 2: Winsorize / cap at percentiles (preferred)
df['salary'] = df['salary'].clip(lower=lower, upper=upper)

Feature Scaling - Critical Timing Rule

# Python  - CORRECT order (after splitting)
from sklearn.preprocessing import StandardScaler, MinMaxScaler

scaler = StandardScaler()                         # Z-score: (x - mean) / std
X_train_scaled = scaler.fit_transform(X_train)    # Fit AND transform training
X_test_scaled  = scaler.transform(X_test)         # Transform ONLY  - no fit

# MinMaxScaler for bounded range [0, 1]
mm_scaler = MinMaxScaler()
X_train_mm = mm_scaler.fit_transform(X_train)
X_test_mm  = mm_scaler.transform(X_test)

# Save the fitted scaler  - you'll need it at inference time
import joblib
joblib.dump(scaler, 'scaler.pkl')

Understand the patterns, relationships, and story within the data. This is an iterative process - visualise everything, then go back and clean more if needed.

# Python  - Univariate Analysis
import matplotlib.pyplot as plt
import seaborn as sns

# Numerical: distribution shape, skewness, outliers
df['salary'].hist(bins=50)
sns.boxplot(x=df['salary'])

# Categorical: frequency balance
df['department'].value_counts().plot(kind='bar')

# Python  - Bivariate Analysis
# Numerical vs Numerical: correlation and scatter
sns.scatterplot(data=df, x='experience_years', y='salary')
corr_matrix = df.select_dtypes(include='number').corr()
sns.heatmap(corr_matrix, annot=True, cmap='coolwarm')

# Categorical vs Numerical
sns.boxplot(data=df, x='education_level', y='salary')

# Categorical vs Categorical
pd.crosstab(df['purchased'], df['marketing_channel'], normalize='index')

# PySpark  - Summary and distributions at scale
from pyspark.sql.functions import corr, col, skewness, kurtosis

# Distribution stats
df.select(
    skewness('salary').alias('salary_skew'),
    kurtosis('salary').alias('salary_kurt')
).show()

# Correlation between two numerical columns
df.select(corr('experience_years', 'salary')).show()

# Categorical frequency
df.groupBy('department').count().orderBy('count', ascending=False).show()

# Cross-tabulation
df.crosstab('purchased', 'marketing_channel').show()

Creating new features or transforming existing ones to help your model learn better. All engineering steps must be learned from training data only and then applied to the test set.

# Python  - Feature Creation
# Date parts
df['signup_date'] = pd.to_datetime(df['signup_date'])
df['signup_day_of_week'] = df['signup_date'].dt.dayofweek
df['signup_month']       = df['signup_date'].dt.month

# Combining features
df['area'] = df['height'] * df['width']

# Binning continuous to categorical
df['age_group'] = pd.cut(df['age'], bins=[0, 25, 45, 65, 100],
                         labels=['Young', 'Middle', 'Senior', 'Elder'])

# Log transform to reduce skewness
import numpy as np
df['log_salary'] = np.log1p(df['salary'])   # log1p handles zeros

# Encoding categorical variables
# One-Hot Encoding (for nominal categories)
df = pd.get_dummies(df, columns=['city'], drop_first=True)

# Ordinal Encoding (for ordered categories)
from sklearn.preprocessing import OrdinalEncoder
oe = OrdinalEncoder(categories=[['Low', 'Medium', 'High']])
df['priority_encoded'] = oe.fit_transform(df[['priority']])

# PySpark  - Feature Engineering
from pyspark.sql.functions import dayofweek, month, log1p, col
from pyspark.ml.feature import StringIndexer, OneHotEncoder, VectorAssembler

# Date parts
df = df.withColumn('signup_day_of_week', dayofweek('signup_date'))
df = df.withColumn('signup_month', month('signup_date'))

# Log transform
df = df.withColumn('log_salary', log1p(col('salary')))

# String to numeric index, then one-hot
indexer = StringIndexer(inputCol='city', outputCol='city_idx')
encoder = OneHotEncoder(inputCol='city_idx', outputCol='city_vec')

# Assemble all features into a single vector (required for Spark ML)
assembler = VectorAssembler(
    inputCols=['age', 'log_salary', 'city_vec', 'signup_day_of_week'],
    outputCol='features'
)

# Python  - The correct split-first workflow
from sklearn.model_selection import train_test_split, StratifiedKFold, cross_val_score
from sklearn.ensemble import RandomForestClassifier, GradientBoostingClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import classification_report, roc_auc_score

X = df.drop('target', axis=1)
y = df['target']

# Stratify ensures class balance is preserved in both splits
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y
)

# NOW apply your scaler (fit on train only)
scaler = StandardScaler()
X_train_sc = scaler.fit_transform(X_train)
X_test_sc  = scaler.transform(X_test)

# Cross-validation on training set (no test data involved)
model = RandomForestClassifier(n_estimators=100, random_state=42)
cv_scores = cross_val_score(model, X_train_sc, y_train, cv=5, scoring='roc_auc')
print(f"CV AUC: {cv_scores.mean():.3f} ± {cv_scores.std():.3f}")

# Train on full training set
model.fit(X_train_sc, y_train)

# Final evaluation  - done ONCE on the held-out test set
y_pred = model.predict(X_test_sc)
print(classification_report(y_test, y_pred))
print(f"Test AUC: {roc_auc_score(y_test, model.predict_proba(X_test_sc)[:, 1]):.3f}")

# PySpark ML Pipeline  - encapsulates transformers + model in correct order
from pyspark.ml import Pipeline
from pyspark.ml.classification import RandomForestClassifier as SparkRF
from pyspark.ml.evaluation import BinaryClassificationEvaluator

# Split first
train_df, test_df = df.randomSplit([0.8, 0.2], seed=42)

# Define pipeline stages (fit happens only on train_df)
pipeline = Pipeline(stages=[indexer, encoder, assembler, SparkRF(
    featuresCol='features',
    labelCol='target',
    numTrees=100,
    seed=42
)])

# Train
model = pipeline.fit(train_df)

# Evaluate on test
predictions = model.transform(test_df)
evaluator = BinaryClassificationEvaluator(labelCol='target', metricName='areaUnderROC')
print(f"Test AUC: {evaluator.evaluate(predictions):.3f}")

The workflow above is standard. But it is incomplete. Real-world projects require additional layers that tutorials ignore.

# Using Great Expectations for data contracts
import great_expectations as ge

df_ge = ge.from_pandas(df)

# Define your data contract
df_ge.expect_column_values_to_be_between('age', min_value=0, max_value=120)
df_ge.expect_column_values_to_not_be_null('customer_id')
df_ge.expect_column_values_to_be_in_set('country', {'US', 'UK', 'CA', 'AU'})

results = df_ge.validate()
if not results['success']:
    raise ValueError("Data contract violated. Pipeline halted.")

# Option 1: Class weights (built into most sklearn estimators)
from sklearn.ensemble import RandomForestClassifier
model = RandomForestClassifier(class_weight='balanced', n_estimators=100, random_state=42)

# Option 2: SMOTE oversampling
from imblearn.over_sampling import SMOTE
smote = SMOTE(random_state=42)
X_resampled, y_resampled = smote.fit_resample(X_train, y_train)

# Always evaluate with the right metrics for imbalanced problems
from sklearn.metrics import classification_report
print(classification_report(y_test, y_pred))  # precision, recall, F1

"Data scientist" is not one job. It is many. And the differences matter. In a startup, you do all of them. In a large enterprise, you specialize. Know which you want - and which you are walking into.

Technical skill is table stakes. The professionals who advance are the ones who master what no course covers.

The framework that ties everything together. Most beginners stay at "See." The professionals move through all four levels.