标签：#mlops

Google Vertex AI is Google Cloud's managed ML platform with experiment tracking, training jobs, pipelines, a model registry, and endpoints, but it locks you into GCP-specific APIs and per-use billing. ClearML is an open-source MLOps platform that covers the same ground on any Docker host or Kubernetes cluster, capturing training metrics automatically with no code changes and keeping every byte of data on infrastructure you control. This guide deploys ClearML Server with Docker Compose and Traefik, registers an agent, runs an experiment, builds a pipeline, runs a hyperparameter sweep, and deploys a Triton-served model. By the end, you'll have a self-hosted Vertex AI replacement covering the full ML lifecycle. Vertex AI → ClearML Mapping GCP Vertex AI ClearML Equivalent Purpose Vertex AI Workbench ClearML Web UI Browser-based monitoring and configuration Vertex AI Experiments ClearML Experiment Manager Automatic hyperparameter/metric/artifact tracking Vertex AI Training Job ClearML Agent + Tasks Any machine becomes a remote worker via queues Vertex AI Pipelines ClearML Pipelines Python-native DAGs, no separate compile step Vertex AI Model Registry ClearML Model Repository Versioned models with full lineage Vertex AI Endpoints ClearML Serving (Triton) Self-hosted inference with canary/A-B support Cloud Monitoring/Logging ClearML Scalars/Plots Built-in metrics and hardware dashboards Prerequisite: Ubuntu host with Docker + Compose, DNS A records for app.clearml.example.com , api.clearml.example.com , files.clearml.example.com . NVIDIA Container Toolkit if you'll run GPU agents. Deploy the ClearML Server 1. Raise Elasticsearch's virtual memory limit and restart Docker: $ echo "vm.max_map_count=524288" | sudo tee /etc/sysctl.d/99-clearml.conf $ sudo sysctl --system $ sudo systemctl restart docker 2. Create persistent data directories with the correct ownership: $ sudo mkdir -p /opt/clearml/ { data/elastic_7,data/mongo_4/db,data/mongo_4/configdb,data/redis,data/fileserver,

What We Built in This Guide In the previous guide, we went from RAG to cloud deployment. In this guide, we systematically implemented everything needed to take that system to production . evals/ dataset.py # Evaluation dataset eval_rag.py # Context Recall · Relevancy · Faithfulness observability/ traced_rag.py # RAG pipeline tracing with @observe() (Langfuse v4) traced_agent.py # Trace each Agent step security/ input_validator.py # Prompt injection detection output_validator.py # PII masking and leakage detection guardrails.py # Rate limiting, security log integration secure_rag.py # RAG with guardrails llmops/ prompt_registry.py # Prompt version management (v1.0–v1.2) ci_eval.py # Quality gate (Overall ≥ 75% to deploy) cost_tracker.py # API cost tracking finetuning/ prepare_dataset.py # Convert to Alpaca format train_lora.py # LoRA fine-tuning (r=8, 2 min on CPU) inference.py # Compare with base model multiagent/ search_worker.py # Search specialist worker quality_worker.py # Quality check specialist worker orchestrator.py # Task decomposition and result integration 14_multiagent.py # Execution script governance/ ai_registry.py # AI system inventory risk_assessor.py # Risk assessment (score 0.18 → LOW) audit_logger.py # Audit log (Article 12 compliant) compliant_rag.py # RAG with AI disclosure (Article 50 compliant) Key Design Decisions from Each Chapter Chapter 2: Evals Combining rule-based (Context Recall, Answer Relevancy) with LLM-as-a-Judge (Faithfulness) strikes the right balance between speed, cost, and coverage. Chapter 3: Observability (Langfuse v4) Adding @observe() decorators is all it takes to start recording traces. The critical v4 change: you must call get_client() after load_dotenv() . Chapter 4: Security Defense in Depth is the principle: Input validation → System prompt → Output validation → Rate limiting — four layers of protection. Chapter 5: MLOps / LLMOps On every push to GitHub, Evals run automatically. Only when the quality threshold (Overall

Introduction Through Chapter 7 (Multi-Agent) , we have a complete, functioning AI system. The final step is building organizational infrastructure to operate AI safely over time. [Before] Technical safety Security → Block malicious input Evals → Measure quality [Now] Organizational / regulatory safety Governance → Know what AI systems are in use Risk mgmt → Classify and assess risks Audit logs → Record who did what, when EU AI Act → Regulatory compliance EU AI Act Status (as of June 2026) The EU AI Act came into force on August 1, 2024, with full enforcement on August 2, 2026 . Transparency rules (disclosing when users are interacting with AI, labeling AI-generated content) also take effect on that date. AI systems are classified into three risk tiers: Risk Level Description Examples Prohibited Not permitted Social scoring, manipulative AI High Risk Strict regulation Hiring, credit scoring, law enforcement Limited Risk Transparency obligations Chatbots, AI-generated content Minimal Risk No regulation Spam filters, game AI Our RAG system's classification: Limited Risk (chatbot). We are required to disclose to users that they are interacting with AI. Directory Structure pgvector-tutorial/ ├── existing files └── governance/ ├── ai_registry.py # ★ AI system inventory ├── risk_assessor.py # ★ Risk assessment ├── audit_logger.py # ★ Audit logging └── compliant_rag.py # ★ Governance-compliant RAG 1. AI System Inventory — governance/ai_registry.py Most organizations lack a systematic inventory of their AI systems, making risk classification and compliance planning difficult. Knowing what you have is the essential first step. # governance/ai_registry.py """ AI system inventory Centrally manage all AI systems in use across the organization. Forms the foundation for technical documentation required by EU AI Act Annex IV. """ from datetime import datetime from dataclasses import dataclass , asdict from enum import Enum class RiskLevel ( Enum ): UNACCEPTABLE = " prohibited " HIG

TL;DR— Most production AI teams build safety layers using the content-moderation mental model: classify input, classify output, block or pass. But the incidents that actually take down AI systems in production look like distributed-systems failures— retries amplifying bad state, cascading errors across agent steps, silent drift with no rollback path. Guardrails need to borrow from SRE, not from trust-and-safety. Ask a team how they handle AI safety in production and you'll get the same answer almost every time: an input classifier, an output classifier, maybe a moderation API bolted on the side. This is the content-moderation mental model— filter bad stuff in, filter bad stuff out. It's borrowed wholesale from trust-and-safety teams who spent a decade building spam filters and abuse detectors. It's also the wrong model for most of what actually breaks AI systems in production. The incidents that page you at 2am rarely look like a jailbreak slipping past a classifier. They look like distributed-systems failures: a retry loop that amplifies a bad tool call, a hallucinated intermediate result that poisons every downstream step, a silent shift in output distribution that nobody notices until a customer complains three weeks later. These are not content problems. They're systems problems, and they need systems solutions. The Cascade, Not the Jailbreak Consider a typical agent pipeline: retrieve context, call a model to plan, call tools, call a model again to synthesize, maybe loop if a tool fails. Each step has some non-zero error rate. In a single-call chatbot, that error rate is the whole risk surface. In a five-step agent chain, errors compound, and worse, they compound non-linearly because failed steps often trigger retries, and retries on a stateful action are not free. A model that hallucinates a tool argument doesn't just produce one bad output— it produces a bad state that the next step reasons over as if it were true. If that next step is another LLM call, it wi

AI agents are only as smart as the context you give them. OKF is a new open specification that packages your organizational knowledge as plain markdown files so any agent can read it without custom integrations or proprietary SDKs. Every team building AI agents hits the same wall. The model is capable. The agent framework is set up. But the agent doesn't know anything about your organization. It doesn't know what your orders table means, what the churn_score metric formula is, or what the on-call runbook says to do when the pipeline breaks. That knowledge exists. It's scattered across Confluence pages, Notion wikis, data catalog entries, Slack threads, and the heads of senior engineers. Getting it into an agent means building a custom integration for every source. Every team solves this from scratch. Published on June 12, 2026, the Open Knowledge Format (OKF) is a vendor-neutral specification that solves this with the simplest possible approach: a directory of markdown files. 🎯 🏗️ What OKF Actually Is An OKF bundle is a directory of markdown files representing concepts: anything you want to capture, including tables, datasets, metrics, playbooks, runbooks, and APIs. Each concept is one file. That's the entire model. A directory of .md files with YAML frontmatter. The format is deliberately minimal: one required field ( type ), optional metadata ( title , description , resource , tags , timestamp ), and a free-form markdown body. A concept document looks like this: --- type : table title : " orders" description : " One row per customer order. Source of truth for revenue reporting." resource : " postgresql://prod-db/ecommerce/orders" tags : [ revenue , core , sla ] timestamp : 2026-06-15T10:00:00Z --- # orders The `orders` table records every purchase event. It is the join root for all revenue queries. Do not filter on `status = 'complete'` unless you specifically want to exclude in-flight orders from the count. ## Key columns - `order_id` - UUID primary key - `custom

TL;DR An AI agent registry is a centralized catalog of every agent in your organization — what each agent does, what tools it can access, what version is running, who owns it, and how to call it It's to agents what a container registry is to Docker images or what a service mesh is to microservices — the layer that makes distributed components governable We hit the "which agents do we have?" wall at 14 agents across 3 teams. That's when the registry stopped being a nice-to-have About four months into our agentic AI buildout, our head of security asked a question I couldn't answer: "Can you give me a list of every AI agent running in production, what systems they have access to, and what version of each is currently deployed?" I had a rough mental model. I knew about the agents my team had built. I had a vague idea of what the data engineering team had shipped. The product team had recently added two agents I'd heard about secondhand. I spent the better part of a day pulling together a spreadsheet. By the time I finished, one of the agents I'd listed had already been replaced by a newer version. Two of them had been granted access to an internal API I hadn't known about. The spreadsheet was outdated before I sent it. That was our forcing function for building a proper agent registry. This post is what I wish I'd read before that conversation happened. What an agent registry is An agent registry is a centralized catalog of AI agents — a single source of truth that tracks every agent deployed in your organization, its capabilities, its integrations, its ownership, and its current state. The analogy that landed for me: it's to agents what a container registry (Docker Hub, ECR, GCR) is to container images. When you have three containers running, you don't need a registry — you know what you have. When you have 40 containers across six teams, you need a registry to know what's running, who owns it, what version is deployed, and what depends on what. Agents are the same. At

TL;DR: An API gateway manages HTTP traffic between services — auth, routing, rate limiting, load balancing for REST and gRPC. An AI gateway manages LLM workloads — token-based rate limiting, model routing, cost attribution, semantic caching, guardrails. Use an API gateway for your microservices. Use an AI gateway for your LLM traffic. Most production teams eventually need both, sitting at different layers. This post walks through exactly where each one fits. When we started adding LLM features to our platform, we already had Kong running for our microservices. The instinct was natural: route the LLM traffic through Kong too. Same auth, same rate limiting, same observability stack. One gateway to rule them all. It worked — for about six months, and only in the sense that requests got through. What it didn't give us was anything useful for actually managing AI workloads. We had no idea what each team was spending on tokens. We had no way to set a budget cap that would fire before the bill arrived. Our rate limits were based on requests per minute, which meant a single request with a 50k token prompt counted the same as one with a 200 token prompt. And when OpenAI had a partial outage, Kong had no concept of "try Anthropic instead" — we just served errors. None of that is a criticism of Kong. It's doing exactly what it was designed to do. The problem was us expecting an API gateway to handle a fundamentally different category of infrastructure problem. Here's the precise distinction, and why it matters architecturally. What an API gateway actually does An API gateway is a reverse proxy that sits between client applications and backend services. It handles the cross-cutting concerns of service-to-service HTTP communication: authentication, authorization, rate limiting, load balancing, SSL termination, request transformation, and routing based on URL paths or headers. A typical request flow through an API gateway: Client sends a request to the gateway endpoint Gateway ve

MLflow is an open-source platform for managing the machine learning lifecycle — experiment tracking, model registry, and reproducible runs. This guide deploys MLflow using Docker Compose with a PostgreSQL backend, S3-compatible artifact storage, basic-auth, and Traefik handling automatic HTTPS, then logs a sample scikit-learn run. By the end, you'll have MLflow recording experiments at your domain over HTTPS. Prerequisite: An S3-compatible bucket (e.g. Vultr Object Storage) with access key, secret key, region, and endpoint URL. Set Up the Directory Structure 1. Create the project directory: $ mkdir -p ~/mlflow $ cd ~/mlflow 2. Create the environment file: $ nano .env DOMAIN = mlflow.example.com LETSENCRYPT_EMAIL = admin@example.com POSTGRES_USER = mlflow POSTGRES_PASSWORD = StrongDatabasePassword123 MLFLOW_AUTH_CONFIG_PATH = /app/basic_auth.ini MLFLOW_FLASK_SERVER_SECRET_KEY = GENERATED_SECRET_KEY S3_BUCKET = mlflow-artifacts S3_ACCESS_KEY = YOUR_ACCESS_KEY S3_SECRET_KEY = YOUR_SECRET_KEY S3_REGION = YOUR_REGION S3_ENDPOINT = https://YOUR_OBJECT_STORAGE_ENDPOINT 3. Create the basic-auth configuration: $ nano basic_auth.ini [mlflow] default_permission = READ database_uri = sqlite:///basic_auth.db admin_username = admin admin_password = ADMIN_PASSWORD authorization_function = mlflow.server.auth:authenticate_request_basic_auth 4. Create a Dockerfile that adds the auth-server extras and Postgres/S3 clients to the official image: $ nano Dockerfile FROM ghcr.io/mlflow/mlflow:v3.10.1 RUN pip install --no-cache-dir psycopg2-binary boto3 'mlflow[auth]' Deploy with Docker Compose 1. Create the Compose manifest: $ nano docker-compose.yml services : traefik : image : traefik:v3.6 container_name : traefik command : - " --providers.docker=true" - " --providers.docker.exposedbydefault=false" - " --entrypoints.web.address=:80" - " --entrypoints.websecure.address=:443" - " --entrypoints.web.http.redirections.entrypoint.to=websecure" - " --entrypoints.web.http.redirections.entrypoint

Over three posts I built three fine-tuned models for the same banking-intent task — full fine-tuning a 270M model , LoRA on 1.5B , QLoRA on 7B . They all landed around the same accuracy. Which raises an honest, slightly uncomfortable question: if a 270M model on my laptop already worked, why reach for a 7B model at all? The answer most "bigger is better" content skips For this task — you wouldn't. A good engineer picks the smallest model that clears the bar , not the biggest one available. The small model is cheaper to serve, runs in milliseconds, and you fully own it. Choosing the 7B here would be over-engineering. Reaching for a bigger model isn't a flex. It's a response to a requirement the small one can't meet. Here are the four cases where small stops being enough: 1. The task is genuinely hard Banking77 is easy — 77 fixed labels, short clean queries. Small models saturate it. But ask for reasoning ("which of these three issues is the primary one?"), open-ended generation (write the reply, don't just classify), or real nuance, and there's a capability floor that more parameters buy. No amount of fine-tuning gives a 270M model abilities it doesn't have. 2. You have little data I had ~10,000 labeled examples — plenty for a small model. With 50, a small model can't learn the task, but a 7B model already "knows" banking concepts from pretraining and only needs a nudge. Bigger models need less task data because they bring more prior knowledge. 3. You need one model for many tasks This is the quiet superpower of LoRA/QLoRA. A single frozen 7B base can host dozens of swappable adapters — intent classifier, reply writer, summarizer, sentiment — all from one ~5GB footprint in memory. The 270M is single-purpose. This is why companies serve hundreds of fine-tunes from one base model. 4. Accuracy compounds at scale 93% means 7 in 100 queries misrouted. At 10M queries/month, that's 700,000 mistakes. If each costs a support escalation, the 2–3 points a bigger model buys can

How GPU scheduling complexity and MLOps integration are forcing platform teams to rearchitect Kubernetes clusters before operational debt becomes insurmountable. As AI workloads consume roughly 40% of enterprise Kubernetes clusters by 2026, the platform's default scheduler is proving fundamentally mismatched with the topology-aware, gang-scheduled demands of GPU-intensive training and inference. Platform engineering teams that invest now in purpose-built GPU scheduling layers, multi-tenant partitioning, and FinOps-driven autoscaling will separate themselves from organizations drowning in 30-45% GPU utilization rates and mounting infrastructure costs. Why the Default Kubernetes Scheduler Fails GPU Workloads Kubernetes was designed for stateless, CPU-bound services, and its pod-by-pod bin-packing scheduler has no native awareness of GPU topology, NUMA boundaries, or NVLink interconnect bandwidth. This becomes a critical failure point with NVIDIA H100 SXM5 nodes, where achieving full-bandwidth tensor parallelism requires all 8 GPUs on a node to be scheduled as a single atomic unit. The default scheduler cannot guarantee this co-placement, meaning distributed PyTorch FSDP or MPI training jobs frequently land on suboptimal node configurations, wasting expensive NVLink bandwidth and forcing teams to over-provision GPU capacity. Idle GPU memory stranded across partially-utilized nodes is the primary driver behind the 30-45% utilization rates reported in 2025 surveys by Gradient Dissent and Weights and Biases, representing millions of dollars in annual wasted spend for mid-to-large enterprises running mixed AI workloads. Building the GPU Scheduling Stack: Volcano, KAI Scheduler, and MIG Platform teams are converging on a layered scheduling architecture that replaces or augments the default Kubernetes scheduler with GPU-aware primitives. Volcano has become the dominant choice for distributed training workloads, using its PodGroup abstraction to enforce gang scheduling across

I run a production multi-agent AI system on a single M1 Mac in Jamaica. 6 autonomous agents. 26 cron workflows. 5-layer persistent memory. All containerized, all running 24/7. I checked my OpenRouter dashboard last week and realized something: I'd processed 2.4 billion tokens across 52 different AI models and spent a total of $0.52 . That's not a typo. Here's exactly where that money went and what it means. The Numbers Metric Value Total Requests 26,600+ Tokens Processed 2.4 Billion Models Used 52 Total Cost $0.52 Cost per Token $0.00000021 Tokens per Dollar 4.6 Million For context: GPT-4 Turbo costs about $0.00001 per token at scale. I'm running at roughly 50x below that rate. Where the $0.52 Actually Went Here's the breakdown by model: Model Requests Tokens Cost openrouter/owl-alpha 1,334 251.2M $0.00 nvidia/nemotron-3-super-120b 32 1.8M $0.00 google/gemma-4-31b-it 47 1.8M $0.00 openai/gpt-5 1 2.8K $0.03 google/gemini-3.1-pro-preview 1 3.2K $0.04 anthropic/claude-opus-4 1 2.0K $0.13 qwen/qwen3.5-plus 1 6.3K $0.01 z-ai/glm-5-turbo 1 3.0K $0.01 moonshotai/kimi-k2.5 2 4.1K $0.01 google/gemini-2.5-flash 2 5.5K $0.01 +42 other models ~125 ~8.5M ~$0.28 99.6% of my requests cost exactly $0.00. They ran on free-tier models or local inference. The $0.52 comes from a handful of premium model calls: Claude Opus, GPT-5, Gemini Pro. These are reserved for specific high-quality tasks — not everyday inference. What This Would Cost on Cloud Approach Hardware Monthly Cost Annual Cost My setup (M1 Mac) M1 Mac 16GB, local + free tier ~$0.09 ~$1.04 OpenRouter Paid Tier API-only, no local $15-30 $180-360 AWS (g4dn.xlarge + API) 1x T4 GPU, on-demand $350-500 $4,200-6,000 AWS (g5.xlarge + API) 1x A10G GPU, on-demand $700-1,000 $8,400-12,000 A $1,200 laptop replaces $500-1,000/month in cloud bills. The break-even point is about 2 weeks. How the Architecture Works The key insight: not every task needs a $20/month model . My system routes tasks intelligently: Local inference (free): Ollama