RC RANDOM CHAOS

memcpy walks off the end of the receiver

A June 2026 statistical analysis of rsync releases finds the two carrying Claude-written commits are no buggier than the historical distribution.

· 4 min read
memcpy walks off the end of the receiver

A June 2026 analysis took every rsync release with bug data, 36 of them from v2.4.6 to v3.4.3, and asked one thing: do the releases containing Claude-written commits look unusually buggy? Two releases carry Claude commits. On the metric the analysis uses, severity-weighted bugs per 10 commits (sev/10c), v3.4.2 scores 0.00 (9 Claude commits out of 50, zero bugs past the severity filter) and v3.4.3 scores 3.29 (28 Claude commits out of 34, 17 bugs). One lands just below the historical interquartile range, the other just above it, and neither is an outlier.

That result exists because of a pile-on. In late May 2026 a Mastodon post tied a regression the poster hit on upgrade to the fact that the release carried Claude commits. The post offered no evidence for the link. It collected thousands of likes and boosts, picked up 58 replies, and moved to Hacker News. On May 30 someone opened a GitHub issue against the rsync repository titled “Please Do Not Vibe Fuck Up This Software,” attaching a screenshot of the Mastodon post and nothing technical. The thread passed 350 comments, a good share of it harassment, with the ugliest entries (including drawings of the maintainer being strangled) since deleted. rsync’s author put up a response essay; on Lobsters a couple of people asked whether anyone had actually charted regressions per release to see if the rate had moved. This analysis is that chart.

The metric sums a severity score for every bug in a release rather than counting bugs. Each report was scored 0 to 100 by Qwen 3 35B, prompted to act as a senior reliability engineer, at temperature 0 so the same input always yields the same number. Those scores are divided by 100, summed per release, divided by the release’s commit count, and multiplied by 10. Bugs come from GitHub issues, the rsync Bugzilla, and the mailing list; GitHub and list reports attach to the most recent release shipped before the report, Bugzilla entries to the version named in their Version field. Reports scored 0 (feature requests, spam, off-topic AI rants, empty submissions) are dropped. That filter matters for v3.4.2, which drew four GitHub issues, all scored 0. The data and plumbing scripts were written by GLM 5.1, but every number on the page is templated in directly by the analysis script, so the figures can’t drift from an LLM writing prose around them.

Two tests carry the argument. An exact permutation test enumerates every pair of two releases and asks how often a random pair scores as bad or worse than the Claude pair’s mean of 1.65 sev/10c. The answer is 272 of 595 pairs, a p-value of 46%. The historical mean is 2.95, about 1.8 times the Claude mean. A Fisher’s exact test splits releases at the historical median (0.74 sev/10c) and asks whether Claude releases are more likely to land above it; the p-value is 74% and the odds ratio is 1.06, which is as close to even as the data gets.

The strongest objection the analysis takes seriously is throughput: maybe Claude did not raise the defect rate per commit but let the maintainer land so much more code that absolute bug counts rose anyway. The numbers run the other way. Claude releases averaged 42 commits against 185 for the rest, and 5.6 severity-weighted bugs against 14.9. They did change far more code per release, 3756 lines against 696 (p=5%, the one test where Claude releases stand out), and that larger change set did not come with more bugs.

The last objection is regime. rsync’s history is bimodal: v2.x releases average 1.11 sev/10c, v3.x releases 4.23, so comparing two recent releases against the full history could hide them being bad for a quieter modern baseline. Restricting to v3.x reverses that. The v3.x era is the noisier one, and against it v3.4.3 sits inside the middle 50% while v3.4.2 is better than most. A Wald-Wolfowitz runs test on the 35 non-Claude releases returns p=0.060 (13 runs against an expected 18.5), marginal evidence of regime structure, which is why the v3.x-only comparison is worth running at all.

None of this proves future Claude releases will behave. Two releases is thin, the metric ignores commit complexity and security intensity, and the author is candid that it is a blunt instrument aimed at a blunt accusation. What it does show is that the two releases everyone pointed at are statistically unremarkable: if nobody had labeled them, nobody looking at the bug data would have picked them out.


Contains a referral link.

Share

Keep Reading

Latest on the Wire

Full wire →

New signal daily · RSS

Stay in the loop

New writing delivered when it's ready. No schedule, no spam.