Observability on wid's bloghttps://wid-blog.github.io/en/tags/observability/Recent content in Observability on wid's blogHugoenWed, 27 Nov 2024 00:00:00 +0000The Blind Spot in Deploy Change Verification — Campaign Cache Incident Retrospectivehttps://wid-blog.github.io/en/posts/career/dable/campaign-cache-miss-retrospective/Wed, 27 Nov 2024 00:00:00 +0000https://wid-blog.github.io/en/posts/career/dable/campaign-cache-miss-retrospective/The deploy was two days old, and the metrics had been calm the whole time. The moment we turned off the cache refresh batch, ad serving stopped. A retrospective on the missing verification of what a deploy actually changed.