Things I learned building my first multi-agent AI system on Azure + NVIDIA

I recently built a multi-agent customer support system on Azure AI Foundry and NVIDIA NIM. First time doing anything like this. Made four predictions upfront about what would happen. Three of them were wrong. Here is what I actually learned. 1. "Tokens" is not a unit of cost It is a unit of work. The price per unit of work varies by 5-10x depending on which model did the work. I was tracking total token count across both the small 9B model and the large 49B model as if they cost the same. They do not. Total tokens went up in the optimized version. Cost in dollars probably went down. I was measuring the wrong thing the whole time. 2. A verbatim hash cache on natural language traffic deflects ~0% of queries I predicted 25-40% cache deflection. The actual number was 0%. Every query in my test set was a unique string, so the hash-based cache never had a single chance to fire. A verbatim cache is not a simpler version of a semantic cache. It is a different thing entirely. If your workload is natural language, build semantic similarity caching from day one, not as an upgrade later. 3. configure_azure_monitor() does not capture OpenAI SDK calls by default You need to install and initialize opentelemetry-instrumentation-httpx explicitly: pip install opentelemetry-instrumentation-httpx==0.61b0 from opentelemetry.instrumentation.httpx import HTTPXClientInstrumentor HTTPXClientInstrumentor().instrument() Without this, your App Insights Logs will show customMetric and performanceCounter entries (CPU, memory) but nothing about what your agent actually did. 4. Pin your OpenTelemetry versions or everything breaks Installing opentelemetry-instrumentation-httpx without version pinning pulled in opentelemetry-api 1.42.1. But azure-monitor-opentelemetry-exporter needs opentelemetry-api==1.40. The conflict is silent until things start misbehaving. Pin everything to the 0.61b0 / 1.40.0 line: pip install \ "opentelemetry-api==1.40.0" \ "opentelemetry-instrumentation==0.61b0" \ "opentelem