Explore Oban's performance boundaries and system load through a journey to
processing millions of jobs a minute.
June 6, 2022 People frequently ask about Oban's performance characteristics, and we typically answer anecdotally—"on this one app in this one environment we run this many jobs a day and the load is around blah." That's real world usage data, but not exactly reproducible. The goal of this article is to fix that ambiguity with numbers and (some) math. You know—and we know that you know—you're here to see some schmancy charts that back up the claim of "one million jobs a minute." Good news! We'll jump right into benchmark results and charts. Afterwards, if you're feeling adventurous, stick around to dig into the nitty-gritty of how much load Oban places on a system, how it minimizes PostgreSQL overhead, and our plans for eking out even more jobs per second. The Benchmark This is the story of a benchmark that demonstrates the jobs-per-second throughput that Oban is capable of. The benchmark aims to process one million jobs a minute, that's 16,1667 jobs/sec, in a single queue on a single node. Running multiple queues or multiple nodes can achieve dramatically higher throughput, but where's the fun in that? Not to spoil the story, but it didn't take much to accomplish our goal. Oban is built on PostgreSQL, which has a stellar benchmarking story and widely documented results for write-heavy workloads, which is precisely Oban's profile. Our benchmark inserts one million jobs and then tracks the time it takes to complete processing them all with a throughput measurement every second. (Reporting and metrics aren't shown, but you can find the original script in this gist). # Start Oban without any queues Oban . start_link ( repo : Repo ) # Create an atomic counter and encode it to binary for use in jobs cnt = :counters . new ( 1 , [ ] ) :ok = :counters . put ( cnt , 1 , 0 ) bin_cnt = cnt |> :erlang . term_to_binary ( ) |> Base . encode64 ( ) # Insert 1m jobs in batches of 5k to avoid parameter limitations Task . async_stream ( 1 .. 200 , fn _ -> Oban . insert_all ( fn…