homeblogtags
  • Published on
    October 23, 2026 (10d ago)

    My Canaries Kept Passing and My Rollouts Kept Failing

    Web-ScrapingWeb-CrawlerGoKubernetesPostmortem
    A worker rollout passed an 80-minute production canary at 72.5-75.5% yield, then its standing pods tripped their circuit breakers at 9.5%. T...
  • Published on
    October 23, 2026 (10d ago)

    I Probed a Million Sites for llms.txt. Half the 200s Were Junk, and My Accept Header Invented More.

    Web-ScrapingData-StudyWeb-CrawlerGoKubernetesllms.txt
    I fetched /llms.txt from the Tranco top 1,000,000 domains. The run agreed with its own smoke test to about 99% and was still wrong in three ...
  • Published on
    October 16, 2026 (17d ago)

    There Is No "Latest Chrome"

    Web-ScrapingTLS-FingerprintingAkamaiHTTP/2DebuggingGo
    One Akamai 403 took four fixes in about a day. One was rolled back after 0 of 12 live requests passed, and one applied the lesson of my last...
  • Published on
    September 18, 2026 (1mo ago)

    My TLS Fingerprint Benchmark Was Measuring Keep-Alive

    Web-ScrapingTLS-FingerprintingCloudflareProxiesBenchmarkingGo
    I benchmarked 18 browser TLS fingerprints against a Cloudflare-challenged site and one of them won at 94%. Twenty-four minutes later I retra...
  • Published on
    September 8, 2026 (1mo ago)

    How Much of the Web Blocks Bots? I Got the Number Wrong Four Times First.

    Web-ScrapingAnti-botWeb-CrawlerGoKubernetesData-Study
    I scanned the Tranco top 1,000,000 domains to measure how much of the web runs managed bot defense. The final answer is 53.5% of reachable s...
  • Published on
    November 24, 2025 (11mo ago)

    After 2025, Can Your Google SERP Crawler Still Survive?

    Web-ScrapingGoogle-SearchSEOAnti-botAI
    An in-depth look at how AI-driven anti-bot defenses, SERP reshuffling, and legal/ethical constraints are reshaping Google search crawling af...
© 2026 Built with 💖 by Tony Wang • With TypeScript, Next.js, Tailwind • Inspired by Leerob
Support me • Contact me
My projects: Crawlora • Namefy • BizHQ