50 Million Records in Under One Second — Inside ZenQL’s New Collection Engine
With the release of version 1.7.9, ZenQL’s Collection API, Thor, received substantial performance improvements, largely driven by a series of memory optimization enhancements. These changes reduced unnecessary allocations, improved data handling efficiency, and significantly accelerated query execution, particularly when working with large in-memory datasets. From the early stages of development, we established a baseline benchmark to measure both correctness and performance consistently: filtering a slice of 50 million items and validating the result. The original Collection API completed this benchmark in approximately 9 seconds. Later, we introduced Thor as a replacement for the default Collection API, reducing the benchmark time to around 4 seconds. With the latest round of memory optimizations and internal improvements, Thor now completes the same benchmark in less than one second. benchmark: goos: linux goarch: amd64 pkg: github.com/malikhan-dev/zenql/collections/Thor cpu: 12th Gen Intel(R) Core(TM) i7-12700H BenchmarkQueryEngine BenchmarkQueryEngine-20 88 13636313 ns/op 22727310 B/op 0 allocs/op at Thor_Engine__test . go func BenchmarkQueryEngine ( b * testing . B ) { result := From ( & items ) . Where ( func ( search ComplexObjectToSearch ) bool { return search . Name == "Jane" && search . Flag == false }) . Collect () result2 := From ( & result ) . Any ( func ( search ComplexObjectToSearch ) bool { return ( search . Name != "Jane" ) || ( search . Flag != false ) }) . Assert () if result2 { b . Error ( "result should be false" ) } } Join the Journey We are committed to making ZenQL the fastest and most developer-friendly query engine for Go. As we continue to grow and push the boundaries of performance, we need your support! If you find ZenQL useful, please star our repository on GitHub. Your support helps us reach more developers and keep the project moving forward. Thank you!