I Analyzed 1,000 AI-Generated Blog Posts for Quality. Here's the Data.

Last year, I was doing something that felt increasingly absurd: manually reading AI-generated content to decide if it was "good enough." PostAll — the content automation tool I've been building — was producing hundreds of blog posts per week for clients. And I had no systematic way to evaluate quality at scale. I was spot-checking. Vibes-checking, really. That doesn't work at volume. So I built a programmatic quality analysis pipeline, ran it over 1,000 AI-generated posts, and let the numbers tell me what my gut was missing. The findings surprised me. A few of them genuinely changed how I think about AI content quality. What I Actually Measured First, a definition of terms, because "quality" is almost meaninglessly vague in this space. I broke quality into five measurable dimensions: Readability — Flesch-Kincaid grade level and reading ease score Keyword density — Target keyword frequency and distribution across the post Grammar error rate — Errors per 1,000 words, caught via LanguageTool's API Factual accuracy — Claims that could be verified programmatically (dates, statistics, named entities cross-referenced against a knowledge base) Structural consistency — Presence of expected elements: intro hook, subheadings, conclusion, CTA I used 1,000 posts across three categories: SaaS product descriptions, long-form "how-to" articles (1,200–2,000 words), and listicles (500–900 words). All were generated by PostAll using GPT-4o, with various prompting strategies. The Setup The analysis pipeline isn't complicated, but the piece that makes it useful is the batch processing layer: import anthropic import language_tool_python import textstat from dataclasses import dataclass from typing import Optional import json @dataclass class QualityReport : post_id : str flesch_reading_ease : float flesch_kincaid_grade : float grammar_errors_per_1000_words : float keyword_density : float structural_score : int # 0–5 based on element presence flagged_claims : list [ str ] overall_score :