700 MB/s of Kafka throughput, on Postgres

2026-09-23T00:00:00+00:00

Kafgres is a Postgres extension that embeds a singleton broker directly inside Postgres, allowing both deep integration with DB transactions, as well as allowing external systems to interact with the database, just as they would with real Kafka. </blockquote>
In a previous article introducing Kafgres</a>, I had mentioned that Kafka becomes the suitable choice for demanding architectures of 100K+ evt/s. However, I knew that in theory, Kafgres should be able to handle this load easily, considering it has to do a fraction of the work of a Kafka cluster doing the same throughput: Postgres-style replication is used, rather than a quorum. With this in mind, I rented a large box from Hetzner, and got to work profiling and poking around the extension to drive up data throughput and records per second.
On an i9 box with PLP-enabled NVME drives (~$0.25/hr via Hetzner server auction), I was able to achieve smooth write throughput of ~700 MB/s, and 600k evt/s.
How it works</h2>
Kafgres lives inside Postgres, in that it is installed in the DB as a Postgres extension, and largely uses Postgres's extension APIs to interact with the OS. However, while taking advantage of a lot of the machinery Postgres provides (to avoid having to write our own WAL for metadata, or completely reimplement leader/follower failover), Kafgres largely works independently of the rest of the database. The vast majority of the Kafgres work is handled by an independent Postgres-spawned background worker, and under default configuration, actually writes topic bytes directly to disk, skipping Postgres internals and overhead for the vast majority of data. This allows us to achieve theoretical performance quite close to that of a regular Kafka singleton running on the same machine.
</figure>
Identifying the bottleneck</h2>
On the i9 box (and for a few of the profiling rounds, a cheaper i7 box), I drove multiple producers against Kafgres 0.1.0. At a throughput of 100-200MB/s this began to stall the engine (identified by a latency spike). The CPU core for Kafgres was nearing 100%, suggesting the stalling was not from blocking on the drive, but by CPU time.
We benchmarked the drive on this box, which achieved a throughput of ~1911MB/s, and an fsync rate of ~68K/s. Meanwhile, our broker was performing less than 600 fsyncs/s, which confirmed there was a large amount of drive-write headroom.
Profiling</h2>
In a previous blog</a>, I have detailed profiling a previous Postgres extension to improve throughput. I used a similar technique to profile the saturated Kafgres broker.
For the on-cpu time (the dominating bottleneck), 40.7% of our time was spent in parse/analyze/plan, which is quite a lot considering only metadata is carried in tables, and topic data gets written directly to disk.
For off-CPU, roughly a third of blocked time was spent in commit. This was an issue on the weaker machine I used, with drives with no power-loss-protection, but was less of an issue on the i9 box.
Pulling the levers</h2>
The first change I made was to attack the nearly half of CPU time spent repeating the same, cachable work. A relatively simple change was to use SPI_keepplan for the metadata statements, allowing Postgres to maintain a generic plan for our statements, to reuse rather than to plan per-execution.
- Spi::get_one_with_args::<i64>( + crate::plan::get_one::<i64>( "SELECT log_start_offset FROM kafgres_partitions WHERE topic_id = $1::oid AND partition = $2", &[topic.into(), partition.into()], ) </code></pre> The next change I made was to tackle the commit time. Technically, with the configuration we were testing, non-transactional produce did not guarantee the actual topic data had been flushed to drive by the time it acked. However, we were guaranteeing offsets had been synchronously committed. Taking a similar default-relaxed posture towards the offsets would allow more metadata to accumulate in the WAL before flushing to disk. Introducing the GUC relaxed_produce_commit</code> allows this to be turned on/off, and improves throughput.
pub fn relax_commit_durability() -> Result<(), pgrx::spi::Error> { pgrx::Spi::run("SET LOCAL synchronous_commit = off") } </code></pre>
An additional change we made was not to optimize throughput per se, but to decrease impact on regular Postgres usage. Reaching these high throughputs is less appealing when it's at the cost of your database performance. The i9 machine we were testing on had 2 NVMEs, so we added a config field to change the directory that kafgres topic data is written to. When that is moved to a different drive than the rest of the database, a throughput of 100 MB/s has very little (~1-5%) impact on the performance of pgbench running against the same database[1]</a>.
fn log_root() -> PathBuf { match crate::log_directory() { Some(dir) => { let p = PathBuf::from(dir); if p.is_absolute() { p } else { PathBuf::from(data_directory()).join(p) } } None => PathBuf::from(data_directory()).join(LOG_DIR), } } </code></pre> These changes (alongside a few others, not worth mentioning in depth) all modestly improved throughput, in total from 113 MB/s -> 197 MB/s. It was at this point I noticed there was a performance disparity between running the benchmark over docker proxy and hitting the Kafgres sockets directly via host networking. This led me to look into how I was handling work queuing up, and socket readiness. The naive approach opted for in 0.1.0 was to have a 5ms tick-loop, allowing bytes to accumulate during this 5ms of blocked time, and waking the background worker on a schedule to handle the bytes that had accumulated. The correct way to do this, as we move on from version 0.1.0 of Kafgres, was to use Postgres's own WaitEventSet</code>. This lets us block not until some period of time has passed, but instead until some socket is ready for us to start work. for &(fd, _) in want { pg_sys::AddWaitEventToSet(self.set, pg_sys::WL_SOCKET_READABLE, fd, null_mut(), null_mut()); } </code></pre> -// epoll set holds the latch self-pipe and postmaster-death pipe. Nothing else. -// Every request waits for the next timer edge. -while BackgroundWorker::wait_latch(Some(tick)) { +// Every client socket is in the wait set; `tick` is now a ceiling, not a cadence. +while readiness.wait(tick) { + readiness.sync(listener_fd, &conn_fds); </code></pre> This change, when we use host networking rather than the docker network, is responsible for the vast majority of the improvement in throughput. Change</th> 1 KiB</th> 256 KiB</th></tr></thead> main</code>, before any of it</td> 118.8</td> 113.3</td></tr> + cached SPI plans</td> 139.9</td> 135.6</td></tr> + relaxed commit, (& a few other tweaks)</td> 169.7</td> 197.3</td></tr> + socket readiness</td> 597.3</td> 703.4</td></tr> </tbody></table> </figure> With record sizes of 256 KiB, we have achieved a stable, low-latency throughput of 700 MB/s produce. In comparison</h2> With these major changes, alongside a handful of other performance tunings that landed, Kafgres now holds its own against vendor Kafka. According to many SaaS Kafka vendors, the vast majority of Kafka workloads are small, estimated at 56% having <= 1 MB/s throughput. Those normally scaling into hundreds of MB/s are mostly large enterprises, who can have a dedicated team for managing Kafka, or can pay fees to vendors to have their Kafka managed for them. On AWS MSK, a regular, 3-node (considered the floor to most) Kafka cluster that could handle ~700 MB/s would likely be made up of kafka.m5.8xlarge nodes, coming in at $3.36/hr each. This comes out to $7,357 monthly in broker-compute, before accounting for storage and data transfer costs, which at usual AWS rates would cost tens of thousands more ($18,000/mo in storage for just 24 hr retention for this data volume). Of course, where MSK (or really, any other vendor) wins is replication. Kafgres can failover to a secondary database, just like the Postgres DB itself, but cannot seamlessly shrug off a zonal outage in the way a 3-zone Kafka cluster can. Additionally, if one chooses to host their Postgres deployment in the cloud, many of the costs incurred for MSK (storage, networking) would still be incurred for the Postgres instance. Regardless, this strongly suggests that for the vast majority of use cases, the Postgres-resident Kafgres is sufficient. By the time you hit Kafgres's limits, an equivalent vendor-provided "real Kafka" deployment will cost hundreds of thousands per year. Kafgres 0.2.0</a> has been released, which vastly closes the gap between "real" Kafka 4.0 and Kafgres. [1]</a> Previously this just said <1%. However, pgbench perf is heavily confounded by other factors (producer core saturation, etc.), and later runs showed a range of 0.9%-5.2%. </div> Kafgres: Embedding a Kafka Broker into Postgres 2026-09-07T00:00:00+00:00 TL;DR “Just use Postgres” Mostly compelling</li> But why throw away the Kafka ecosystem</li> </ul> Dozens of blogs, listicles and wikis center around the concept of “just use Postgres’. Many have contrasted the operational burden and cost of managing a Kafka cluster against just using various techniques to get message queues or event logs out of your existing Postgres clusters. Proposed options range from using Postgres locks to handle competitive consumption from tables (that can be treated as logs). SELECT FOR UPDATE SKIP LOCKED </code></pre> To using purpose built extensions extending psql for message queues SELECT * from pgmq.send( queue_name => 'my_queue', msg => '{"foo": "bar1"}' ); </code></pre> These blog posts are broadly correct, in that Postgres (especially when extended) is sufficient for the vast majority of use-cases, especially at lower scales, and that relying on your existing database alleviates a large operational burden. One caveat not usually addressed is that relying on bespoke table-row-based queues or tailor-made Postgres extensions throws away the Kafka ecosystem, and leaves you with a smaller corpus of libraries to draw on during development. In many cases, that is a reasonable trade-off, but “writing wrappers around Postgres” is not most people’s end goal. I have been working on kafgres</code>, an extension embedding a performant singleton Kafka broker into Postgres. Not only does doing this let you reuse your database as your log broker and store, it allows tight coupling between your event log and database activity, without needing to spin up yet another deployment (i.e. debezium). $ psql -h Postgres -p 5432 -U Postgres -c "SELECT version()" PostgreSQL 16.15 (Debian 16.15-1.pgdg12+2) on aarch64-unknown-linux-gnu, compiled by gcc (Debian 12.2.0-14+deb12u1) 12.2.0, 64-bit $ kcat -b Postgres:9092 -L Metadata for all topics (from broker 1: Postgres:9092/1): 1 brokers: broker 1 at Postgres:9092 (controller) </code></pre> BEGIN; INSERT INTO orders (id, customer_id, total, status) VALUES (1001, 1, 249.99, 'placed'); SELECT kafgres_produce( 'order-events', 'order-1001', jsonb_build_object( 'event', 'OrderPlaced', 'order_id', 1001, 'customer', 'Acme Corp', 'total', 249.99 )::text) AS offset_assigned; COMMIT; </code></pre> Of course, Postgres already provides its own WAL, which is ultimately what debezium usually reads. If you’re more used to mapping events from the CDC than emitting via Postgres, you can do that as well. SELECT kafgres_add_mapping( 'orders-cdc', -- mapping name 'public.orders', -- source table 'orders-cdc', -- destination topic value_expr => $$ jsonb_build_object( 'op', op, 'order_id', new.id, 'total', new.total, 'status', new.status, 'customer', (SELECT jsonb_build_object('name', c.name, 'tier', c.tier) FROM customers c WHERE c.id = new.customer_id)) $$, key_expr => $$ new.id::text $$, filter_expr => $$ new.status <> 'draft' $$); </code></pre> Kafgres isn’t just a server pointed at your tables (doing this would be prone to causing WAL churn), it is built on a pluggable storage engine living in the Postgres engine, and it implements dozens of Kafka APIs, so that both Kafka clients and admin tools sees it as a regular, singleton Kafka broker. # 1. an ordinary Kafka client produces $ echo 'order-3001:{"event":"OrderPlaced","order_id":3001,"source":"Kafka-client"}' \ | kcat -b Postgres:9092 -t order-events -K: -P # 2. a SQL transaction produces to the same topic $ psql -h Postgres -c "BEGIN; INSERT INTO orders VALUES (3002, 4, 1250.00, 'placed'); SELECT kafgres_produce('order-events', 'order-3002', jsonb_build_object('event','OrderPlaced','order_id',3002,'source','sql-transaction')::text); COMMIT;" # 3. one consumer reads both, and cannot tell them apart $ kcat -b Postgres:9092 -t order-events -C -e -q -o beginning -f '%p:%o %k -> %s\n' 0:0 order-1001 -> {"event": "OrderPlaced", "total": 249.99, "customer": "Acme Corp", "order_id": 1001} 1:0 order-3001 -> {"event":"OrderPlaced","order_id":3001,"source":"Kafka-client"} 1:1 order-3002 -> {"event": "OrderPlaced", "source": "sql-transaction", "order_id": 3002} </code></pre> Redpanda Console is happy to administer kafgres as a Kafka broker</figcaption> </figure> On a machine with an NVME and an i7 from 2017, a Postgres instance with kafgres installed is stable and fluid at a throughput of 30k-70k event/s. The DB remains responsive during this time, and with the default storage engine, we only see a few percent-point hit in pgbench performance. Kafgres versus: Kafgres</th> Kafka</th></tr></thead> 0 new nodes</td> Often 3+ nodes, or SAAS subscription</td></tr> 30k-70k msg/s</td> 100k+ msg/s</td></tr> CDC out of the box</td> CDC via Debezium/etc</td></tr> Singleton, same HA posture as Postgres</td> Distributed</td></tr> "I want to use Kafka ecosystem, and don't need to worry about uber scale yet"</td> "We need a distributed log and/or to scale through 100k msg/s"</td></tr> </tbody></table> In an order pipeline demo</a>, we are able to reduce a 3-deployment event-driven architecture to one Postgres deployment, while still using standard Kafka libraries to build our microservices. Normally</th> With Kafgres</th></tr></thead> Postgres: the business database</td> the same Postgres</td></tr> Debezium: change capture off the WAL</td> kafgres's CDC mappings (kafgres_add_mapping</code>)</td></tr> Kafka: the broker services talk over</td> the same Postgres</td></tr> An outbox table plus a relay, to make a write and a publish atomic</td> kafgres_produce()</code> inside the business transaction</td></tr> </tbody></table> I have recently released Kafgres 0.1.0</a>, and will be continuing to iterate on the broker to chase down performance, safety, and utility gains. Misc. on the writing of Kafgres</h3> Kafgres is written using pgrx</code>, a framework that I am a big fan of, allowing you to write Postgres extensions in Rust. While performance is important for the extension, considering we want a performant broker, the main appeal of Rust for this project is reducing the footprint of code that could crash the broker. If Kafgres were super prone to crashing, the appeal of collapsing two deployments to one would be defeated by the risk of bringing down your one Postgres deploy. I transported a lot of learnings from my previous broker-in-Postgres project, pgmqtt</a>. Where we diverge from the current version of pgmqtt is leaving a lot more internals off of the WAL. Although this means we need to write a lot more code, it means we can achieve a much higher throughput without crowding the other Postgres workers. Q: If we keep our topic off the WAL, how do we get transactional kafgres_produce()? A: kafgres_produce uses a commit marker row that does land on the WAL, and is inserted in the caller's transaction. The produced message doesn't land in that transaction, but the row that tells the kafgres engine whether the message commits does. This allows us to replicate Kafka's own aborted-transaction protocol. Q: If Postgres is a singleton, how do we handle replication? A: Take cues from Postgres. We stream replication data to a standby (another kafgres, in another Postgres). Since we avoid using the WAL for topic data, the kafgres follower needs to pull the log using the Kafka fetch protocol (as if it were reading from another, regular Kafka broker). We replicate Kafka's epoch reconciliation to handle divergent data. How do you profile a pgrx Postgres extension? 2026-05-19T00:00:00+00:00 TL;DR - If you build with symbol tables intact, find your PID, and hook into it with perf record</code> and bpftrace</code>, you will have usable data dumps you can inspect with scripts and flamegraphs. I’ve been building an extension</a> that embeds an MQTT broker into Postgres, and spent time tuning performance and throughput. This post walks through the setup I used, and should be generic enough to apply to any pgrx extension. The goal: figure out where the extension is spending its time, both on-CPU and while blocked off-CPU. Build flags</h2> In your Cargo.toml, there are a few build directives that will ensure the built binary (the .so your compiler emits, and Postgres loads) will have the necessary symbols for the profiler to read & understand what is happening. [profile.release] # or whatever profile you're going to profile strip = "none" # keep symbol table (.symtab) debug = "line-tables-only" # keep the line table (.debug_line) </code></pre> strip = "none"</code> answers "what function contains this address" debug = "line-tables-only"</code> will answer "what source line did this instruction come from" Finding your target PID</h2> Since a pgrx extension isn’t just a binary you can execute, but runs as a process owned by Postgres, we can’t immediately reach for something like cargo flamegraph</code> pgrx extensions can execute in two places: SQL backends and background workers. SQL backend: your extension code executes as part of a client connection's backend. This is per-session, so you can find the PID from inside your session. You will keep this session open while you profile the process, and use it to run the workload you’re testing with. select pg_backend_pid(); </code></pre> Background worker: this is a long-lived process that Postgres starts, and runs independently/out-of-band with client sessions. Find it from SQL: -- from any connection SELECT pid, application_name, backend_type FROM pg_stat_activity WHERE backend_type = 'background worker'; -- You will need to pick the worker for *your* extension, there will be multiple returned. </code></pre> Or from your shell: ps -eo pid,cmd | grep 'your_extension_name' </code></pre> Exercising your extension</h2> For a useful profile, you will need to exercise your extension while you sample it. Importantly, this workload should be varied or at least representative of real conditions you want to improve. A typical Postgres extension might be bottlenecked in several different places, depending on the workload that it's running. Think through various scenarios you might want to observe: Steady state: What happens when throughput is sustained</li> Bursts: what happens when you get a sudden spike?</li> Large payloads: Do these cause an edge case with a lot of overhead?</li> Connection storms: Do you have a lot of overhead by connection? Can you weather this?</li> </ul> Profiling on-CPU</h2> On-CPU here refers to time spent where your extension's instructions are being actively executed by the CPU, not waiting on something like the disk. The Linux util perf record</code> will sample your process at a given rate, recording the instruction that is being executed (& the call stack for it). Note: in order to successfully use perf record, you may need to enable capabilities for it on your machine. Here’s a sample, though path might differ with your operating system: sudo setcap cap_perfmon,cap_sys_ptrace+ep /usr/lib/linux-tools-6.8.0-110/perf </code></pre> Using perf record is simple, once you have your process running with a workload and have its PID. PID=$your_target_pid perf record -F 99 -g --call-graph dwarf,16384 \ -o perf.data -p "$PID" -- sleep 30 </code></pre> About the arguments: -F 99</code>: Sample at 99 Hz - this is notably not 100 Hz, in case your extension does something at 100 Hz already, which might synchronize with the profiler.</li> --call-graph dwarf, 16384</code> tells perf to use .eh_frame</code>/.debug_frame</code> unwind table to read the stack. 16384</code> denotes the snapshot size for the sampled stack. 16384 should be large enough to get a useful stack from your extension.</li> </ul> We can then render this as a flamegraph with any number of flamegraph utils. I used brendangregg/FlameGraph</a>. perf script -i perf.data | \ stackcollapse-perf.pl | \ flamegraph.pl --title "my-extension (on-CPU)" > cpu.svg </code></pre> Flamegraphs are just one way of reading the perf data. To interpret these, take a look at the width of frames in the flamegraph - the wider they are, the more perf samples landed in these frames (implying more time was spent there). Frames are stacked on top of each other to represent call chains. Functions that take up a lot more width in the graph than you expect are what you should prioritize investigating. Profiling off-CPU</h2> Off-CPU time here refers to time your process wasn't running, and was waiting on something to resume execution on the CPU. In many extensions, especially those that involve interacting with data-on-disk, off-CPU time will dominate on-CPU time. perf</code> in its default configuration can’t do this, since there’s nothing to be sampled. The Linux util bpftrace</code> can use eBPF to hook into the kernel scheduler & capture when your process is moved on/off the CPU. The idea is that we can record the stack and timestamp when your process is descheduled, and when it gets rescheduled, we can compute how long it was off the CPU (blocked, waiting on something like disk write). The script we supply to bpftrace</code> below does exactly that. sudo bpftrace -p "$PID" -e ' tracepoint:sched:sched_switch / args->prev_pid == $1 / { @off_kstack[args->prev_pid] = kstack; @off_ustack[args->prev_pid] = ustack; @off_ts[args->prev_pid] = nsecs; } tracepoint:sched:sched_switch / args->next_pid == $1 && @off_ts[args->next_pid] != 0 / { $delta_us = (nsecs - @off_ts[args->next_pid]) / 1000; @offcpu[@off_kstack[args->next_pid], @off_ustack[args->next_pid]] = sum($delta_us); delete(@off_kstack[args->next_pid]); delete(@off_ustack[args->next_pid]); delete(@off_ts[args->next_pid]); }' "$PID" </code></pre> The output from this isn’t in a format that most tools are ready to understand: it’s a textdump of form: @offcpu[kstack, ustack]: microseconds</code>. Parsing and folding this output into something useful (like an aggregated table, or a flamegraph) will need some code to process it in. I have done this with a python script</a> bundled with profile.sh, which we’ll get to shortly. Profiling Together</h2> On-CPU and off-CPU profiling are complementary, and can and should be run together. For my extension, I wrapped the whole workflow into a shell script</a> to handle all of this. ./profile.sh \ --pid-pattern "pgmqtt" \ --mode both \ --duration 60 \ --workload "python3 simulate_load.py" \ --label baseline </code></pre> This will locate the PID for a background worker for you, run both on & off CPU profilings on it with your given workload, and will render this all into tables and flamegraphs for you to study. Here is an example of output from my off-CPU processing: [epoll] 1201654 us 70.1% [other] 344596 us 20.1% [fsync] 81566 us 4.8% [futex] 76557 us 4.5% [disk-io] 8651 us 0.5% </code></pre> Case study: my experience with pgmqtt</h2> I arrived at profile.sh by iterating over my own extension, trying to squeeze out higher throughputs while maintaining durability guarantees. Here’s what using profile.sh to make iterative improvements actually looks like. pgmqtt</code> is a PostgreSQL extension that embeds an MQTT broker, and for it to have some use beyond a CDC source (or event sink), I needed to ensure we had decent performance characteristics. Although there are plenty of MQTT deployments out there that do not see a very high throughput, it is definitely worth chasing some higher throughput to increase the ceiling on workloads that can use pgmqtt. In MQTT, “Quality of service 0” (QoS 0) is the most basic level for messages: there are effectively no guarantees. A publisher sends a message, and expects the subscriber to either get it or not get it. A best-effort attempt is made to deliver it to the subscriber, but usually this QoS is used in cases where its okay to miss some events. Quality of service 1 (QoS 1) is the level where guarantees start to matter. QoS 1 is typically referred to as "at least once", meaning if a publisher saw a PUBACK from the broker for their PUB, they know that any subscribers will be delivered the message at least once. This requires us to store the message in case the broker crashes before we can confirm the subscriber has received the message. I focused on these two QoS levels for my runs. A central feature for pgmqtt is Postgres CDC driving mqtt publishes, but I found CDC-driven publishes to be very fast already, and took a backseat to optimizing regular QoS0/QoS 1 throughput. Baseline run: QoS0</h2> On-CPU was quite fast, I observed no obvious bottlenecks. Skipping over this to off-CPU: I expected little off-CPU interaction for QoS0, since durability is not a requirement for QoS 0, and found this to be the case: Reason</th> Time (µs)</th> Share</th></tr></thead> [epoll]</code></td> 28,605,318</td> 99.1 %</td></tr> [other]</code></td> 262,519</td> 0.9 %</td></tr> [fsync]</code></td> 6,628</td> 0.0 %</td></tr> </tbody></table> 99.1% of the off-CPU time was epoll - this was the background worker waiting on the next tick to resume reading from clients & publishing. Baseline run: QoS 1</h2> Off-CPU was broadly similar to QoS0, but with fsync taking a much larger (but negligible compared to wait time) slice of time, from interactions with the filesystem (as QoS 1 messages must be persisted). Reason</th> Time (µs)</th> Share</th></tr></thead> [epoll]</code></td> 23,198,201</td> 99.2 %</td></tr> [fsync]</code></td> 129,983</td> 0.6 %</td></tr> [other]</code></td> 46,530</td> 0.2 %</td></tr> </tbody></table> Observed throughput: Publisher side: 228 msg/s</li> Subscriber side: 574 msg/s</li> </ul> We saw way lower throughput with QoS 1 than with QoS0: 228 msg/s was lower than I've seen in prior benchmarks - normally I would expect to see > 1000 msg/s when testing the extension. Looking at on-CPU, we see a lot of samples on: DecodeXLogRecord</code>: PostgreSQL's WAL record deserialiser</li> pg_comp_crc32c_armv8</code>: CRC verification of WAL records</li> WALRead</code>: reading bytes from WAL segment files</li> </ol> This means that WAL decoding dominated our CPU time, even though we weren't testing any CDC-driven messages in this profiling run. With this run, we were causing a lot of churn in the WAL from our persisted messages. This churn meant we were not able to keep up with our (unthrottled) publisher's rate, and once their buffer reached our 64kb-per-tick ceiling, the publishers were dropped. This caused their rate to zero out for the remainder of the run, causing the low average publish rate. We don't typically see this problem in our normal performance tests, since our publishers were throttled to not be disconnected by the broker. First Optimisations:</h2> Configure the tick interval, allowing us to handle more messages simply by waiting less</li> Configure buffer size limit for clients, giving us a new default of 256KiB</li> Read from client socket until WouldBlock</li> </ul> Run 2 - QoS0</h3> No changes to throughput, as expected. When there aren't any messages to persist, we don't cause WAL churn, and don't encounter the backpressure problem. Reason</th> Time (µs)</th> Share</th></tr></thead> [epoll]</code></td> 27,841,203</td> 99.0 %</td></tr> [other]</code></td> 218,447</td> 0.8 %</td></tr> [fsync]</code></td> 60,112</td> 0.2 %</td></tr> </tbody></table> We did notice an increase in fsync share for QoS0 - since we are ticking much more often, we also poll for CDC changes a lot more. This is still negligible for QoS0. Run 2 - QoS 1</h3> Observed throughput: Publisher side: 1,907 msg/s</li> Subscriber side: 6,843 msg/s</li> </ul> Way better - 8.4x improvement at the same publisher load. This was largely from us no longer seeing the publishers get dropped Reason</th> Time (µs)</th> Share</th> Run 1 share</th></tr></thead> [epoll]</code></td> 4,614,221</td> 19.7 %</td> 99.2 %</td></tr> [fsync]</code></td> 7,541,088</td> 32.2 %</td> 0.6 %</td></tr> [other]</code></td> 10,897,344</td> 46.5 %</td> 0.2 %</td></tr> [net]</code></td> 367,503</td> 1.6 %</td> 0.0 %</td></tr> </tbody></table> We see that we now spend only 19.7% of blocked time waiting on the WaitLatch (epoll, which is what WaitLatch ends up calling), meaning the broker is spending a lot more time actually working. Note: we kept this configurable, since it may be desirable for a DB admin to have the MQTT broker do less work, and give the rest of the server more CPU + disk time. [other] has become the dominant share at this point - at this point, it would be valuable to break this down further, if you want a better at-a-glance output for this. It would get repetitive to detail the half-dozen further iterations I went through. Instead, I will summarize some of the further fixes I made and why: What</th> Why</th></tr></thead> Gated hot-path logging</td> Increased blocked time for little gain</td></tr> Set transient/cache tables as UNLOGGED</td> Decreased WAL churn for internal tables that do not need durability</td></tr> Allow CDC tick frequency to be configurable</td> Compensate for the increase in WAL reads from a very low wait time - CDC is quite fast, even if only flushed every few ticks</td></tr> Prepare statements</td> Eliminate per-call planning cost</td></tr> Smarter socket writing (write_buf/try_write)</td> Don't block BGW on slow TCP send</td></tr> Backpressure clients (or, just stop reading their buffer) rather than dropping them</td> Not completely based off of profiling, but a more intelligent approach than dropping as soon as a buffer size is exceeded - inspired by our buffer increase to 256KiB</td></tr> Set replica identity nothing</code> on some internal tables</td> Save some WAL volume from tables that needed no logical WAL subscriptions</td></tr> Set client_id type to collate C</td> We only need UTF-8 byte comparisons on this ID, we won’t ever need to sort/etc. This saves us from spending a bunch of time on varstr_cmp</td></tr> Better gating on CDC processing check</td> Skipping earlier meant unrelated DB churn wouldn’t eat up CPU cycles</td></tr> </tbody></table> The biggest wins were from tackling the things that looked “worst” in the tables & flamegraph, like taking a look at [epoll] 99.2%</code> and thinking that's a lot of time to throw away while messages could be getting processed. Having profiling data makes chasing performance far more tractable.

Kafgres: Embedding a Kafka Broker into Postgres

How do you profile a pgrx Postgres extension?

rynr.dev blog

Sixteen Models Walk Into a Storefront

700 MB/s of Kafka throughput, on Postgres