标签：#ai

OpenAI has issued another statement on the lawsuit, this time suggesting it lacks merit.

Last year my team had to pick a coding agent, and I volunteered to run the evaluation. I felt good about it. I pulled up the public benchmark scores, lined up the contenders, took the one at the top, and told everyone we had a winner. Then we actually pointed it at our repo. It did not blow up dramatically. It just kept being slightly wrong in ways that ate our time. It wrote diffs our reviewers would not approve. It renamed a function and broke three files it had never opened. The tests it ran passed, and the repo was still broken. I had confidently recommended a tool based on a number that turned out to say almost nothing about our situation. That was embarrassing enough that I went and figured out why. It took a few weeks of reading and a couple more bad calls before I landed on something that works. This is that, written plainly, and I hope it saves you the meeting where you have to walk your recommendation back. Why the benchmark score lied to me The score was not fake. It was just measuring somebody else's code. Once I looked properly, four gaps explained the whole thing: The agent might have already seen the answers. The problems in these public benchmarks are old. Models were very likely trained on the actual fixes used to grade them. So the score partly measures memory, not problem-solving. The setup is nothing like real work. A benchmark gives the agent a clean repo, one clear issue, and one command to run the tests. My engineers give it a half-open editor, a messy branch, a Slack thread, and a reviewer comment. Completely different job. Our codebase has its own habits. Our internal libraries, our wrappers, our test style, the imports we ban. No benchmark knows any of that, so an agent can write textbook-perfect code that our reviewers still reject on sight. The bar for passing is way lower. A benchmark passes a patch if the broken test now passes. My team passes a patch if it does that, and does not break unrelated tests, does not reformat the whole file,

One weekend I wrote an LLM eval framework in about two hundred lines of Python. It demoed beautifully. I felt clever. Six months later that same framework was a mess. Three different judge models with three different parsing hacks. A test dataset nobody had touched since November. A CI gate that kept failing because a vendor nudged their model, not because anyone broke a prompt. And the second engineer on rotation asking me, fairly, "how does this even work?" The framework did not fail. The eighty percent of the work the weekend tutorial skipped is what failed. That gap is the whole story, and this is what I would tell myself before starting again. The one line I wish someone had told me: build the rubric, buy the runner Here is the split that took me six months to see. Some parts of an eval setup are yours and only yours. The rubric that decides what "good" means for your product. The dataset built from your real failures. The rules for when a change is bad enough to block a release. Nobody else can write these, because they encode your domain. The rest is the same at every company. The thing that calls the judge model, parses its answer, retries, and caches. The machinery to run thousands of checks in parallel. The plumbing that scores live traffic. The system that groups failing calls together. Every team rebuilds these, hits the same bugs, and gains nothing by writing them twice. So build the first list. Do not hand-build the second. I rebuilt the second, and it cost me most of a year. Two questions I now ask about every single piece Before writing any part of this, I ask two things: Is it specific to me, or generic? A rubric for my domain is specific and worth owning. A retry-and-cache loop around a model call is generic. Everyone writes the same one. Does it compound, or does it rot? A dataset that grows from real production failures compounds. A year in, it is a regression suite no competitor can copy. A hand-built tracing layer rots. The moment a vendor chan

A number of social media posts claim that GPT-5.6 Sol deleted files and data without warning. OpenAI had basically disclosed the problem in June.

It seems some people take issue with one billionaire family overseeing a third of the US' entertainment media.

SPOILER: De 1 à 2000 Durante minha jornada como jogador , existiu uma coisa que me despertou muita curiosidade: os créditos. Quando eu jogava coisas como Final Fantasy, ficava completamente arrepiado quando via aquelas cenas antes do menu inicial, com CGIs bem bonitões e um monte de nomes em japonês — muitos deles que, honestamente, até hoje não sei quem são. Esse arrepio também acontecia quando eu chegava ao final de algum jogo e começavam a aparecer nomes e mais nomes. Quando eu entendi o que eram aqueles nomes, a primeira coisa que me veio à mente foi: peraí, precisa de tudo isso de gente pra fazer um jogo? Too long, didn't read : Sim e não. Skyrim levou aproximadamente 6 anos com uma equipe de 100 pessoas (e ainda é dito que é uma equipe enxuta), enquanto Stardew Valley levou 4,5 anos com uma "equipe" de apenas 1 pessoa. Eles têm quase o mesmo tamanho em horas jogadas na campanha principal. Not too long, I'll read it : É um pensamento lógico que trabalhos maiores no mundo da tecnologia envolvam uma quantidade maior de pessoas trabalhando E tempo. Dessa forma, seria correto assumir que um trabalho menor exige menos tempo e menos pessoas trabalhando. Mas o que os dados dizem? King of Fighters 94: 2 anos, começou com 6 pessoas, aumentou para 60. Clair Obscure - Expedition 33: aproximadamente 6 anos, 30 pessoas. TES V - Skyrim: 6 anos, 100 pessoas. Super Mario Bros: 2 anos, 7 pessoas na equipe principal. Stardew Valley: 4,5 anos, 1 pessoa. Kenshi: 12 anos, 1 pessoa por 6 anos (trabalhando meio período), aproximadamente 8 pessoas por mais 6 anos (tempo integral). Final Fantasy VII (PS1): aproximadamente 2 anos, de 100 a 150 pessoas (uma das maiores equipes da indústria na época). Final Fantasy VII Remake: Cerca de 5 anos, e não se tem números exatos, mas os créditos citam mais de 2000 pessoas, incluindo muitos -istas e -ores (vou tentar confirmar esse número depois). É claro que existem muitos fatores envolvidos aí, como época, demandas de mercado, prazos, tecnologia

Design + Product Thinking: NYC’s Path to Reliable AI AI delivers value when it’s useful, trusted, and operational. For city services that affect millions, those qualities don’t happen by accident — they come from applying design thinking (who the service is for, how it’s used) together with product thinking (what outcome we’re trying to achieve and how we operate over time). This article explains why hiring designers and product managers matters for NYC’s digital and AI initiatives, summarizes the city’s PIT Crew program, and outlines how Flamelit applies outcome-focused delivery in the public sector. Why design and product roles matter Designers and product managers have distinct but complementary responsibilities that reduce common AI delivery failures: Designers (Design Thinking): center human needs, prototype user flows, and validate that interfaces and decision workflows are understandable and accessible. They surface usability and trust issues early, preventing technically accurate models from becoming unusable in practice. Product managers (Product Thinking): define the measurable outcomes, prioritize use cases, align stakeholders, and manage the lifecycle from discovery to ongoing operations. They ensure work is evaluated against mission impact, not just technical metrics. Together they prevent common failures: building technically impressive models that nobody trusts, deploying brittle systems without human review, or shipping features with unclear ownership that decay in production. PIT Crew and NYC hiring context NYC’s PIT Crew program is a city initiative designed to attract and staff product, engineering, and design talent for public service projects. It’s a practical recognition that public-sector digital transformation needs people skilled in user research, product management, and delivery. Read more about the PIT Crew and how it works here: https://www.nyc.gov/content/pitcrew/pages/ (open in a new tab). Hiring programs like PIT Crew help create the c

Genkit Agents API, ORA, Python AI Explainer: New Tools for Workflow Automation Today's Highlights This week, Google's Genkit ships a powerful Agents API for TypeScript and Go, featuring human-in-the-loop capabilities for robust production deployments. Additionally, a new Go-based open-source task orchestrator, ORA, emerges for efficient model routing, alongside a practical Python tutorial for building an AI error explainer. Google's Genkit Ships Agents API with Detached Turns and Human-in-the-Loop for TypeScript and Go (InfoQ) Source: https://www.infoq.com/news/2026/07/genkit-agents-api-preview/?utm_campaign=infoq_content&utm_source=infoq&utm_medium=feed&utm_term=global Google has released a preview of its Genkit Agents API, significantly enhancing its open-source AI framework for building generative AI applications. This new API introduces features critical for deploying robust AI agents in production, specifically "Detached Turns" and "Human-in-the-Loop" functionalities. Detached Turns allow agents to operate asynchronously, handling long-running tasks or waiting for external events without blocking the main workflow, which is essential for complex, multi-step agentic processes. The Human-in-the-Loop feature provides crucial mechanisms for human oversight and intervention, ensuring reliability, safety, and compliance in critical applications where full automation is not yet feasible or desirable. The Genkit Agents API supports both TypeScript and Go, targeting a broad range of developers integrating AI into existing systems or building new agentic workflows. By offering structured patterns for agent interaction, state management, and human review, Genkit aims to streamline the development and deployment of intelligent agents, addressing key challenges in control and reliability for real-world AI applications. Comment: The introduction of Human-in-the-Loop into Genkit's Agents API is a game-changer for production-grade agent systems, offering the control and reliab

Browser automation used to be easier to describe. A test opened a page, filled in a form, clicked a button, and checked the result. The hardest parts were usually selectors, waits, and browser compatibility. Those problems still exist, but the surface area has expanded. Today, browser tests may need to handle streaming interfaces, MFA, AI-generated content, multiple operating systems, preview deployments, canary releases, and code changes proposed by AI assistants. The challenge is no longer just writing a script that passes. The challenge is building a testing system that remains understandable and affordable after hundreds of tests and thousands of CI runs. Start by measuring instability instead of normalizing it Flaky tests often become accepted background noise. A test fails, CI retries it, and the second run passes. The pipeline turns green, so the team moves on. Over time, the retry count grows and nobody is sure which failures matter. The problem is that a passing retry does not erase the cost of the first failure. The article on calculating the real cost of flaky test retries in CI provides a useful framework for evaluating compute costs, developer interruptions, delayed feedback, and investigation time. A simple reliability metric can help: first-attempt pass rate = tests passing without retry / total test executions This is often more revealing than the final pipeline pass rate. A suite with a 99% final pass rate may still be deeply unstable if many tests require multiple attempts. Reproduce the environment before changing the test When a browser test fails only in CI, teams often edit the test before reproducing the environment. That can lead to unnecessary waits and conditionals. One of the most common variations is a test that passes in visible Chrome but fails in headless mode. The explanation is not always “headless Chrome is flaky.” Differences in viewport, rendering, animation, fonts, and resource timing can all change application behavior. This det

For a long time, teams treated browser test reliability as a framework problem. When tests failed, the usual response was to change selectors, add waits, increase retries, or replace one automation library with another. That approach made sense when the main challenge was simply controlling a browser. Modern applications are different. A single user journey may now include an identity provider, multi-factor authentication, a streaming AI response, a background API request, a feature flag, a canary deployment, and a frontend rendered differently across several operating systems. The test framework is still important, but it is only one part of the reliability problem. The bigger question is whether the entire testing system gives the team enough evidence to make a release decision. Headless failures are usually a symptom, not the real problem A common example is a test that passes locally but fails only in headless Chrome. It is tempting to assume that headless mode is simply unreliable. In practice, the difference is often caused by viewport size, rendering behavior, animation timing, fonts, resource loading, or elements being positioned differently when no visible browser window exists. This breakdown of why browser tests fail only in Chrome headless is useful because it separates several failure categories that are often grouped together as “timing issues.” That distinction matters. A test that fails because an element is outside the viewport needs a different fix from a test that fails because a network request completes later in CI. Adding a longer timeout may hide both problems temporarily, but it does not make the test more trustworthy. Retries can make a weak test suite look healthy Retries are one of the easiest ways to reduce visible failures in CI. They are also one of the easiest ways to hide instability. A flaky test that passes on its third attempt still consumed runner time, delayed feedback, created extra logs, and made it harder to determine whether

While building with OpenAI, Anthropic, and other AI providers, I realized something surprising. I monitored my servers, databases, and application performance—but I had almost no visibility into my AI API spending until I checked the provider dashboard or received the monthly invoice. That led me to build AICostPass . It helps developers, indie hackers, startups, and agencies: ⚡ Track AI API costs in near real time 📊 Monitor spending by project or client 🚨 Get budget threshold email alerts 📧 Receive weekly spending summaries 💰 Export billable CSVs for client invoicing The goal is simple: help developers understand and control AI costs before the invoice arrives. 👉 https://aicostpass.com I'd love to hear how you're currently tracking AI API costs. Are you using provider dashboards, spreadsheets, or another tool?

OpenAI's first device is set to be a smart speaker that lets you talk with ChatGPT, according to a report from Bloomberg. The device apparently won't have a screen, but will use a camera and additional sensors to "understand" your environment. The report comes just days after Apple filed a lawsuit against OpenAI that accused […]

If Claude can already search the web, read a 10-K, and explain what a rate cut does to long-duration equities, the fair question is why you would connect anything to it at all. It is the right question, and the honest answer is that for a large class of questions you should not. Raw Claude is enough. The gap is narrower and sharper than "Claude does not know finance." Claude knows finance. What it does not know is you. What raw Claude already does well Be clear about this before the sales pitch, because pretending otherwise would insult anyone who has actually used it. Claude with web search will look up a current quote, summarize an earnings call, explain a valuation multiple, walk you through how a Monte Carlo simulation works, and reason about a macro scenario better than most of the commentary you would read instead. If your question is about the world, and not about your own balance sheet, a connector adds nothing. Ask Claude directly. The trouble starts the moment the answer depends on what you actually own. Four things that break when the question is about your money 1. It starts from zero every time A chat has no memory of your holdings. You can paste them in, and many people do, and it works for exactly one conversation. There is no cost basis, no purchase date, no daily snapshot series behind it. So "how concentrated am I really", "what is my realized gain this year", and "how correlated are my top five positions over the last 90 days" are not questions it can answer. It can only answer them about the numbers you re-typed, this once, from memory. 2. The same question gives a different answer twice LLM inference is not deterministic, and it is not deterministic even at temperature zero. Thinking Machines Lab traced the cause to batch-invariance in inference kernels: the batch your request lands in varies with server load, so the arithmetic varies with it. They fixed it in a research setting and got 1,000 bitwise-identical runs, which tells you how much engi

It's the latest in a barrage of efforts to win compensation from AI companies over training materials.

Meta denies using AI to terminate workers with disabilities and medical problems.

If you’ve been waiting to try Apple’s revamped Siri without installing a developer beta, you now can. The company on Tuesday released the iOS 27 public beta, giving iPhone owners early access to its AI-powered assistant and other new features before the software’s official launch this fall.

Anthropic has consistently attempted to depict itself as the ethical foil to other AI companies. This latest marketing stunt — which leans into criticism of AI as a way to make Anthropic seem aware of the responsibility it carries — would appear to be more of the same.

Overtone describes itself as "a voice- and audio-forward service, enabled by AI, that provides highly curated introductions."

SpaceXAI's Grok Build AI coding tool was spotted uploading users' entire codebases to Google Cloud before it was reported, and the company turned it off. The Register reports that Cereblab published findings on Monday showing how the Grok Build CLI was packaging and uploading entire code repositories, "including files it was told not to open […]

Your AI Agent Needs Tracing, Not Just Logs You've probably already called an LLM from a Node.js backend. That part's easy — every provider ships a solid SDK. The part that actually trips people up is what happens after : turning that one API call into an agent that reasons, uses tools, loops a few times, and still behaves once real users are hitting it. Here's a small, honest pattern for that — plus the one thing most tutorials skip: making the loop debuggable. Why Node.js is doing this job Node has quietly become the default home for the application layer around AI. It's become the preferred middle layer for deploying modern AI agents, wrapping heavier model inference behind fast Node APIs. Python still owns training and the heavy orchestration frameworks — Node owns the gateway, the auth, the streaming UI, and the business logic wrapped around all of it. On the SDK side, things consolidated fast: OpenAI's Node SDK holds roughly a third of weekly npm downloads across the major JS AI SDKs, and Anthropic's TypeScript SDK has grown nearly tenfold in a year. And despite all the framework noise, most production teams just use the Claude or OpenAI SDK directly — reaching for LangChain.js or Mastra only once multi-agent coordination actually earns its keep. The loop: reason, act, repeat Almost every "agent" in 2026 runs on the same loop: reason about the task, act through a tool call, look at what came back, reason again — repeat until done. That's it. The engineering is in the guardrails around it, not the loop itself. // agent.js import Anthropic from " @anthropic-ai/sdk " ; const anthropic = new Anthropic (); // reads ANTHROPIC_API_KEY from env const tools = [ { name : " get_order_status " , description : " Look up the status of a customer order by order ID. " , input_schema : { type : " object " , properties : { orderId : { type : " string " } }, required : [ " orderId " ], }, }, ]; async function getOrderStatus ({ orderId }) { // stand-in for a real DB/service call r