Trying Out AgentInspect

Imagine a travel agent answering: Tokyo is rainy today. Choose indoor sightseeing. Did it check the weather first? Or did it return that sentence without calling the weather tool? The final answer looks the same, but the execution paths are different. Checking that path matters. If a prompt should trigger a particular tool and the tool never runs, something has gone wrong before we even judge the answer. Seeing that failure helps identify where the application needs fixing. An agent's trajectory is its execution path: model calls, tool calls, their order, and their parent-child relationships. Checking it is another important part of keeping an agent's behavior reliable, alongside evaluating the final answer. Tools such as Promptfoo, LangSmith, and DeepEval can evaluate aspects of agent execution. This time, I tried AgentInspect . Its author sent me a cold email asking whether I'd give it a try. It was already a topic I was interested in, so I did. Apparently, cold emails still work in the AI era. Compared with better-known, more mature libraries, AgentInspect felt early-stage, but already useful for specific tasks such as CI checks. To keep the experiment simple, I built a small weather-tool example using Vercel AI SDK and the official adapter. What AgentInspect does AgentInspect is a TypeScript library for recording agent execution locally and checking the execution process. Traces are stored as JSONL files. You can define rules for whether a required tool was called, a prohibited tool was avoided, or the run completed. These are deterministic checks: applying the same rules to the same trace produces the same verdict. For this experiment, I separated two questions: Process checks: did the agent call the weather tool according to the defined requirements? Answer checks: did the returned text contain what we expected? I started by checking the process with AgentInspect and comparing the answer with a fixture string. Later, I added LLM-as-a-Judge to have a model grad