Published onOctober 23, 2026 (10d ago)My Canaries Kept Passing and My Rollouts Kept FailingWeb-ScrapingWeb-CrawlerGoKubernetesPostmortemA worker rollout passed an 80-minute production canary at 72.5-75.5% yield, then its standing pods tripped their circuit breakers at 9.5%. T...
Published onOctober 23, 2026 (10d ago)I Probed a Million Sites for llms.txt. Half the 200s Were Junk, and My Accept Header Invented More.Web-ScrapingData-StudyWeb-CrawlerGoKubernetesllms.txtI fetched /llms.txt from the Tranco top 1,000,000 domains. The run agreed with its own smoke test to about 99% and was still wrong in three ...
Published onOctober 16, 2026 (17d ago)There Is No "Latest Chrome"Web-ScrapingTLS-FingerprintingAkamaiHTTP/2DebuggingGoOne Akamai 403 took four fixes in about a day. One was rolled back after 0 of 12 live requests passed, and one applied the lesson of my last...
Published onSeptember 18, 2026 (1mo ago)My TLS Fingerprint Benchmark Was Measuring Keep-AliveWeb-ScrapingTLS-FingerprintingCloudflareProxiesBenchmarkingGoI benchmarked 18 browser TLS fingerprints against a Cloudflare-challenged site and one of them won at 94%. Twenty-four minutes later I retra...
Published onSeptember 8, 2026 (1mo ago)How Much of the Web Blocks Bots? I Got the Number Wrong Four Times First.Web-ScrapingAnti-botWeb-CrawlerGoKubernetesData-StudyI scanned the Tranco top 1,000,000 domains to measure how much of the web runs managed bot defense. The final answer is 53.5% of reachable s...
Published onNovember 24, 2025 (11mo ago)After 2025, Can Your Google SERP Crawler Still Survive?Web-ScrapingGoogle-SearchSEOAnti-botAIAn in-depth look at how AI-driven anti-bot defenses, SERP reshuffling, and legal/ethical constraints are reshaping Google search crawling af...