标签：#parser

Hi everyone, I want to share textparser, a high-performance, lightweight text parsing and AST generation library written in pure C. 💡 The Core Idea & Why It's DifferentTraditional parser generators (like Flex/Bison) come with a steep learning curve and rigid code generation. On the other end, hand-writing recursive descent parsers or state machines becomes an unmaintainable mess as your language grows.textparser bridges this gap using a hybrid JSON + Python + C workflow:You define your tokens, syntax patterns, and color styles in a clean, human-readable JSON grammar file. A lightweight Python compiler tool processes the JSON, runs optimizations, and emits a dense, static C header array.The C runtime engine loads these pre-compiled arrays instantly. It processes raw text strings into an abstract token tree (textparser_token_item) using a highly optimized regular expression engine (crpe2).By offloading the grammar overhead and heavy state-machine parsing logic to the Python build step, the actual runtime C library stays incredibly lean, memory-efficient, and fast. 🚀 Features At A Glance30+ Languages Out-of-the-Box: Includes ready-to-use JSON grammars for C, C++, Rust, Python, JavaScript, HTML, SQL, and dozens more.Rich Token Metadata: Every parsed token tracks exact code coordinates, structural flags, and custom syntax styling options.Zero Bloat: Ideal for terminal text editors, syntax highlighters, custom linters, and lightweight static analysis tools where bringing in a massive compiler front-end is overkill.

I've been building a database engine from scratch in Rust, and I recently finished the lexer. The lexer itself wasn't the most interesting part. What I found more valuable was how my design evolved as I learned more about Rust and how compilers and database systems are typically implemented. My First Approach When I started, I stored the input as a Vec<char> . It felt straightforward because I could access characters directly without worrying about UTF-8 boundaries. I also represented identifiers like this: Ident ( String ) At first glance, this seems perfectly reasonable. Every identifier token carries its own text, making it easy for the parser to consume. The Problem As the lexer grew, I started asking myself a simple question: The identifier already exists in the original SQL query. Why am I allocating another string and copying the same data into every token? For a query like: SELECT username , email FROM users ; the source text already contains: username email users Creating separate String allocations for each identifier means duplicating data that already exists. I also learned an important detail about Rust enums. The size of an enum is influenced by its largest variant. Once variants start carrying additional data, every token instance becomes larger than it otherwise needs to be. Moving to a Span-Based Design Instead of storing identifier text directly inside tokens, I switched to storing only the token kind: Ident along with source location information: Span { start , end , line , column , } Now the token only answers two questions: What is this token? Where did it come from? If the parser needs the actual identifier text, it can recover it directly from the original SQL source using the stored byte range. Replacing Vec<char> with &str The second design change was moving away from: Vec < char > and operating directly on: & str using lifetimes. Instead of creating another collection containing the entire input, the lexer now walks over borrowed source tex