Reports of login failures across production environments

Incident Report for Perion

Postmortem

Scope of the Issue

  • North-America production web application
  • North-America production iOS application

No other regions or services were disrupted, including ad-serving and APIs access across al regions.

‌

Timeline of Events

  • 2025-07-29 16:58 - Deployment to production North-America causes sessions to be invalidated and logins to fail
  • 2025-07-29 17:37 - The issue was found and a fix was executed
  • 2025-07-29 17:42 - The issue was resolved

‌

Context

We have been undergoing routine security hardening to comply with current industry standards. Part of this effort was to move to a model of authentication where we use shorter-lived auth tokens in conjunction with refresh tokens to seamlessly expire and update the tokens that users use upon each request to our servers via their web browsers. This change finally made its way to production on 2025-07-29 at approximately 4:58PM

‌

Phased Rollout

The changes described above were phased across our deployment environments ensuring months of testing and validation.

‌

Root Cause

Ultimately, a misconfiguration in our authentication provider’s platform meant that effectively all users of the production North-American platform were not considered in an authentication requests. More specifically, the database of users was not linked to the new authentication mechanism deployed to our environment in North-America. This resulted in all users failing to validate their sessions, and subsequently, or initially, login to the platform.

Posted Jul 29, 2025 - 22:18 UTC

Resolved

We have identified and resolved the issue. A misconfiguration with our authentication provider caused a short period where users' session were invalidated, and users were subsequently unable to login
Posted Jul 29, 2025 - 21:48 UTC

Investigating

We are currently investigating the issue

The issue only affects the web application. Ad-serving and API access remains operational
Posted Jul 29, 2025 - 21:31 UTC
This incident affected: DOOH (DOOH: Web Application & APIs).