Kubeflow MLOps: Orchestrating End-to-End Machine Learning Pipelines on Kubernetes

Kubeflow is the de facto standard for running machine learning workflows on Kubernetes. It provides pipeline orchestration, experiment tracking, hyperparameter tuning, and model serving — all running on your existing K8s infrastructure.

1. Kubeflow Pipeline Definition

from kfp import dsl, compiler

@dsl.component(base_image="python:3.11")
def preprocess_data(input_path: str, output_path: dsl.OutputPath("Dataset")):
    import pandas as pd
    df = pd.read_csv(input_path)
    df_clean = df.dropna().reset_index(drop=True)
    df_clean.to_parquet(output_path)

@dsl.component(base_image="pytorch/pytorch:2.3.0-cuda12.1")
def train_model(dataset: dsl.InputPath("Dataset"), model_output: dsl.OutputPath("Model")):
    import torch
    # Training logic here
    torch.save(model.state_dict(), model_output)

@dsl.pipeline(name="training-pipeline")
def ml_pipeline(data_path: str):
    preprocess_task = preprocess_data(input_path=data_path)
    train_task = train_model(dataset=preprocess_task.outputs["output_path"])

compiler.Compiler().compile(ml_pipeline, "pipeline.yaml")

Kubeflow brings software engineering discipline — version control, CI/CD, monitoring — to machine learning workflows that traditionally lived in Jupyter notebooks.