Zero-Dollar Data Engineer

Kaggle is a trap. It does not teach you how data moves from an application database to a warehouse, how to handle PII and compliance requirements, how to schedule pipelines that run reliably at 3 AM, or how to debug a broken data flow at 2 PM on a Friday. Real-world data work is 90% plumbing, 10% modeling. This guide is the antidote.

A zero-dollar modern data stack that mirrors what enterprises use - just on free tiers. Every component maps directly to a paid enterprise equivalent.

ZERO-DOLLAR MODERN DATA STACK

  ┌─────────────────────────────────────────────────────────────────┐
  │  DATA GENERATION                                                │
  │  Python Faker + Cloudflare D1  →  GitHub Actions Cron  (FREE)  │
  └──────────────────────────┬──────────────────────────────────────┘
                             │
                             ▼
  ┌─────────────────────────────────────────────────────────────────┐
  │  PII COMPLIANCE GATEWAY                                         │
  │  Cloudflare Worker + Microsoft Presidio  (FREE)                 │
  └──────────────────────────┬──────────────────────────────────────┘
                             │
                             ▼
  ┌─────────────────────────────────────────────────────────────────┐
  │  DATA LAKE                                                      │
  │  AWS S3 Free Tier (5 GB) or Cloudflare R2 (10 GB/month)        │
  └───────────────┬─────────────────────────────────────────────────┘
                 │
        ┌────────┴────────┐
        ▼                 ▼
  ┌──────────────┐  ┌─────────────────────────────────────────┐
  │  ETL ENGINE  │  │  ORCHESTRATION                          │
  │  dbt Core    │  │  GitHub Actions (2000 min/month FREE)   │
  └──────┬───────┘  └────────────────────┬────────────────────┘
         └───────────────┬───────────────┘
                         ▼
  ┌─────────────────────────────────────────────────────────────────┐
  │  DATA WAREHOUSE                                                 │
  │  BigQuery Sandbox (1 TB queries/month FREE)                     │
  └────────────────┬──────────────────────┬───────────────────────┬─┘
                   │                      │                       │
                   ▼                      ▼                       ▼
  ┌──────────────────────┐  ┌─────────────────────┐  ┌──────────────────────┐
  │  MODEL TRAINING      │  │  VISUALIZATION      │  │  DATA VERSIONING     │
  │  Google Colab (FREE) │  │  Tableau Public     │  │  DVC + GitHub        │
  │  Databricks CE       │  │  Looker Studio      │  │  (FREE)              │
  └──────────────────────┘  └─────────────────────┘  └──────────────────────┘

Stack Components

You do not need 4 months. The entire foundation can be working in 7 days. Here is the daily plan.

Production-ready code snippets for every layer of the stack. Copy, adapt, and commit these to your repo.

1. Synthetic Data Generation with Faker

# generate_customers.py
from faker import Faker
import pandas as pd
import random
from datetime import datetime

fake = Faker()
Faker.seed(42)
random.seed(42)

def generate_customers(n: int) -> pd.DataFrame:
    records = []
    for _ in range(n):
        records.append({
            'customer_id':   fake.uuid4(),
            'name':          fake.name(),
            'email':         fake.email(),
            'phone':         fake.phone_number(),
            'ssn':           fake.ssn(),               # PII - will be masked
            'address':       fake.address().replace('\n', ', '),
            'signup_date':   fake.date_between(start_date='-2y', end_date='today').isoformat(),
            'plan':          random.choice(['free', 'pro', 'enterprise']),
            'monthly_spend': round(random.uniform(0, 500), 2),
            'churned':       random.choice([0, 0, 0, 1]),  # ~25% churn rate
        })
    return pd.DataFrame(records)

if __name__ == '__main__':
    df = generate_customers(1000)
    df.to_parquet('data/raw/customers.parquet', index=False)
    print(f"Generated {len(df)} customers → data/raw/customers.parquet")

2. GitHub Actions Schedule

# .github/workflows/daily_pipeline.yml
name: Daily Data Pipeline

on:
  schedule:
    - cron: '0 3 * * *'   # 3 AM UTC every day
  workflow_dispatch:       # allow manual trigger

jobs:
  generate-and-load:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Set up Python
        uses: actions/setup-python@v5
        with:
          python-version: '3.11'

      - name: Install dependencies
        run: pip install faker pandas pyarrow boto3 google-cloud-bigquery

      - name: Generate synthetic data
        run: python generate_customers.py

      - name: Mask PII via Cloudflare Worker
        env:
          WORKER_URL: ${{ secrets.WORKER_URL }}
        run: python mask_pii.py

      - name: Upload to S3
        env:
          AWS_ACCESS_KEY_ID:     ${{ secrets.AWS_ACCESS_KEY_ID }}
          AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
        run: python upload_to_s3.py

      - name: Load to BigQuery
        env:
          GOOGLE_CREDENTIALS: ${{ secrets.GOOGLE_CREDENTIALS }}
        run: python load_to_bigquery.py

      - name: Run dbt models
        run: |
          pip install dbt-bigquery
          dbt run --profiles-dir .
          dbt test --profiles-dir .

3. PII Masking Cloudflare Worker

// worker.js - deployed to Cloudflare Workers
const PII_PATTERNS = {
  email:  /[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}/g,
  phone:  /(\+?1[-.\s]?)?\(?\d{3}\)?[-.\s]?\d{3}[-.\s]?\d{4}/g,
  ssn:    /\b\d{3}-\d{2}-\d{4}\b/g,
};

function maskPII(text) {
  let masked = text;
  masked = masked.replace(PII_PATTERNS.email,  '[EMAIL_REDACTED]');
  masked = masked.replace(PII_PATTERNS.phone,  '[PHONE_REDACTED]');
  masked = masked.replace(PII_PATTERNS.ssn,    '[SSN_REDACTED]');
  return masked;
}

export default {
  async fetch(request, env) {
    if (request.method !== 'POST') {
      return new Response('Method Not Allowed', { status: 405 });
    }

    const body  = await request.json();
    const clean = {};

    for (const [key, value] of Object.entries(body)) {
      clean[key] = typeof value === 'string' ? maskPII(value) : value;
    }

    // Log PII detection event to D1 audit table
    await env.DB.prepare(
      'INSERT INTO pii_audit (ts, record_id, fields_masked) VALUES (?, ?, ?)'
    ).bind(Date.now(), body.customer_id, JSON.stringify(Object.keys(clean))).run();

    return Response.json(clean);
  },
};

4. ETL to BigQuery

# load_to_bigquery.py
import pandas as pd
from google.cloud import bigquery
from google.oauth2 import service_account
import json, os

credentials = service_account.Credentials.from_service_account_info(
    json.loads(os.environ['GOOGLE_CREDENTIALS'])
)
client = bigquery.Client(credentials=credentials, project='your-project-id')

def load_parquet_to_bq(parquet_path: str, table_id: str) -> None:
    df = pd.read_parquet(parquet_path)

    job_config = bigquery.LoadJobConfig(
        write_disposition=bigquery.WriteDisposition.WRITE_TRUNCATE,
        autodetect=True,
        source_format=bigquery.SourceFormat.PARQUET,
    )

    job = client.load_table_from_dataframe(df, table_id, job_config=job_config)
    job.result()   # wait for completion

    table = client.get_table(table_id)
    print(f"Loaded {table.num_rows} rows into {table_id}")

if __name__ == '__main__':
    load_parquet_to_bq(
        parquet_path='data/masked/customers.parquet',
        table_id='your-project-id.raw.customers',
    )

5. BigQuery Query from Colab

# Google Colab - authenticate and query BigQuery
from google.colab import auth
from google.cloud import bigquery
import pandas as pd

auth.authenticate_user()   # opens browser OAuth flow
client = bigquery.Client(project='your-project-id')

query = """
    SELECT
        plan,
        COUNT(*)                          AS total_customers,
        AVG(monthly_spend)                AS avg_spend,
        SUM(churned) / COUNT(*)           AS churn_rate
    FROM `your-project-id.marts.customers`
    GROUP BY plan
    ORDER BY avg_spend DESC
"""

df = client.query(query).to_dataframe()
print(df.to_string(index=False))

# Quick visualisation
import matplotlib.pyplot as plt
df.plot(x='plan', y='churn_rate', kind='bar', figsize=(8, 4))
plt.title('Churn Rate by Plan')
plt.tight_layout()
plt.show()

6. Idempotent Load Pattern

# idempotent_load.py - safe to re-run any number of times
from google.cloud import bigquery

def idempotent_merge(client, staging_table: str, target_table: str, key: str) -> None:
    """
    MERGE is idempotent: running it twice produces the same result.
    Never use APPEND without a deduplication step.
    """
    merge_sql = f"""
    MERGE `{target_table}` AS target
    USING `{staging_table}` AS source
    ON target.{key} = source.{key}
    WHEN MATCHED THEN
        UPDATE SET
            target.email         = source.email,
            target.monthly_spend = source.monthly_spend,
            target.churned       = source.churned,
            target.updated_at    = CURRENT_TIMESTAMP()
    WHEN NOT MATCHED THEN
        INSERT ROW
    """
    job = client.query(merge_sql)
    job.result()
    print(f"Merge complete: {staging_table} -> {target_table} on {key}")

7. DVC Data Versioning

# bash - set up DVC with S3 remote
pip install dvc[s3]

# Initialize DVC in your repo
dvc init
git commit -m "Initialize DVC"

# Configure S3 as the remote storage
dvc remote add -d myremote s3://your-bucket/dvc-store
git add .dvc/config
git commit -m "Configure DVC remote"

# Track your data files
dvc add data/raw/customers.parquet
dvc add data/masked/customers.parquet
git add data/raw/customers.parquet.dvc data/masked/customers.parquet.dvc .gitignore
git commit -m "Add DVC tracked datasets v1"

# Push data to S3
dvc push

# After next pipeline run - track new version
dvc add data/raw/customers.parquet
git add data/raw/customers.parquet.dvc
git commit -m "Update dataset v2 - 5000 records"
dvc push

# Reproduce a previous version
git checkout v1
dvc pull

Lessons from production pipelines that beginner guides skip. These are the details that separate engineers who understand data systems from those who just learned the syntax.

Every component in this stack maps directly to its enterprise equivalent. When you join a company using Snowflake and Airflow, you already understand the patterns.

Reading is not building. Start this week. The first milestone is straightforward and takes less than two hours.

The zero-dollar stack covers the visible parts of data engineering. But experienced practitioners know that roughly 80% of what determines whether a project succeeds or fails is invisible in any tutorial. Here are the layers nobody teaches.

This is not about adding more tools. It is about building the mindset that separates someone who can run a pipeline from someone who owns one.

Official documentation and community resources for every tool in the stack.