Trifle – Open-source analytics that stores answers, not events

Trifle is an open-source time-series analytics library that aggregates nested counters instead of storing raw events. All in the database you already have. After rebuilding it twice over 10 years, it now tracks ~1B events a day at my day job.It started in 2015 as my own Rails APM. I plugged into ActiveSupport::Notifications, got a few small users, and one bigger one whose scraping app broke everything. That sparked the core idea: aggregate counters into pre-defined time buckets, so a single write increments multiple buckets at once. The APM eventually faded away without much traction.Later in 2021 I needed analytics at my day job. Instead of going for something out there I revised the idea of Trifle as a more generic analytics library, borrowing some data warehouse ideas. First used Redis, then Postgres, eventually MongoDB. Hence why Trifle::Stats comes with multiple drivers that keep the DSL unified while storage layer changes with your needs. In our case (huge write volume, some reads) PG read faster but slowed on large writes.The nested values are the whole trick here. Single: Trifle::Stats.track( key: 'requests::aws::s3_uploads', values: { count: 1, status: { request.response_code => 1 }, size: payload.bytes, duration: { sum: request.duration, count: 1 } } ) builds up counts for requests, success rate, result status codes, duration for multiple time buckets at once. Single bucket from 2am then looks like: { count: 14, status: { 200: 12, 500: 2 }, size: 5628341, duration: { sum: 43, count: 14 } } If request.duration is in seconds, then sum stored under duration would be in seconds as well.Success rate is never stored, but it is calculated by dividing 200s over total number of requests. Same with average duration: sum over count. You ask for a metrics key, granularity and timeframe and you get back aggregated values at each point. Ready for charts or to answer "Average response time over last 30 days&qu