We had rolled out a change for internal authentication of our services when they reach out to the customer workloads during assignment. As part of that rollout, the router which forwards all incoming requests to the customer workloads, including adb traffic, stopped stripping the. "?token=<instance token>" query parameter from the requests as part of cleanup. However, the HTTP library used to front that request started to fail to pair the requests with the correct path as it was not configured to do so, e.g. it was expecting exact match.

Because this was a staged rollout involving multiple components to make sure we enable the auth once all pieces are in place, it did not surface itself in staging where all components were already deployed which contained the fix for this issue. So, it was effectively failure of our rollout ordering that made us blind to the problem in staging and it made it to the production.

We've reverted all changes during the incident but it took more than we expected to untangle all the pieces. Once rollback is completed, all regions started to recover as the old versions of Android were deployed. We then applied the same rollout plan with fallbacks. The rollout was successful and it didn't need to fall back, later on we cleared those fallbacks and concluded the incident.

Android Debug Bridge (adb) tunnels started to return 404 or 401 to the clients.