Ship a security fix without locking out real users

· Maksym Lyskov

A tightened access check changes who gets a 403. You can count those callers before you merge, by name, and check the same list the day after deploy.

Everything below is a made-up example: the endpoint, the rule, the caller ids and the numbers. The code is TypeScript on Express 5 and the queries are Postgres. I ran both in a scratch project against a throwaway Postgres 17 with seeded data, and the output blocks are what they printed.

The endpoint is GET /projects/:id/export. Today anyone in the project’s organisation can export it. The fix says only members of that project can. That’s the classic object-level hole, OWASP API1:2023, and closing it is correct. It also turns away every caller who was relying on the looser rule, and some of them are legitimate.

Run the new rule next to the old one

Ship the new rule first in a mode where it decides nothing. Both rules run on every request, the old one still answers, and any disagreement gets logged.

// Old rule: anyone in the project's org can export it.
const oldRule = (c: Caller, projectId: string) =>
  projects.get(projectId)?.orgId === c.orgId;

// The fix: only members of that project can export it.
const newRule = (c: Caller, projectId: string) =>
  members.has(`${c.id}:${projectId}`);

export function canExport(req: Request, res: Response, next: NextFunction) {
  const mode = process.env.EXPORT_AUTHZ ?? "shadow"; // "shadow" | "enforce"
  const caller = res.locals.caller as Caller;
  const projectId = String(req.params.id);
  const before = oldRule(caller, projectId);
  const after = newRule(caller, projectId);

  if (before !== after) {
    console.log(JSON.stringify({
      event: "authz_shadow",
      rule: "export.project_member",
      mode,
      callerId: caller.id,
      callerKind: caller.kind,
      projectId,
      oldAllow: before,
      newAllow: after,
    }));
  }

  const allowed = mode === "enforce" ? after : before;
  if (!allowed) return res.sendStatus(403);
  next();
}

In my scratch run, a reporting API key that belongs to the org but to neither project still gets its 200s in shadow mode, and each request leaves a line:

{"event":"authz_shadow","rule":"export.project_member","mode":"shadow","callerId":"svc_reporting","callerKind":"api_key","projectId":"p1","oldAllow":true,"newAllow":false}
shadow key-report GET /projects/p1/export -> 200

With EXPORT_AUTHZ=enforce the same requests come back 403. So that env var doubles as the rollback switch later, and flipping it is faster than a revert.

I log both directions on purpose. A request the old rule refused and the new one allows means the new rule has a gap of its own, and you’d rather find that in the log than in an incident.

If you can’t deploy a shadow build, replay last week’s access logs through newRule instead. Two traps there. The log line has to carry the caller and the object id, and a path like /projects/p1/export does. And the replay checks against today’s memberships, so a user removed from a project on Tuesday shows up as a would-be rejection for Monday’s requests.

Count who it would turn away

Ship the shadow lines into a table (or query them wherever your logs live) and group by caller:

SELECT caller_id,
       caller_kind,
       count(*)                   AS would_reject,
       count(DISTINCT project_id) AS projects,
       count(DISTINCT ts::date)   AS days_seen,
       max(ts) - min(ts)          AS spread
FROM authz_shadow
WHERE rule = 'export.project_member'
  AND old_allow AND NOT new_allow
  AND ts >= now() - interval '7 days'
GROUP BY caller_id, caller_kind
ORDER BY would_reject DESC;

On the seeded week:

   caller_id   | caller_kind | would_reject | projects | days_seen |     spread
---------------+-------------+--------------+----------+-----------+-----------------
 u_eve         | user        |          312 |      290 |         1 | 00:36:17
 svc_reporting | api_key     |           21 |        3 |         7 | 6 days 00:02:00
 u_dana        | user        |            5 |        4 |         4 | 4 days 04:00:00

The projects and spread columns do most of the sorting for you. 290 different ids in 36 minutes is someone walking ids, and that’s the abuse the fix exists for. The same 3 projects every night for a week is a scheduled job. A handful of projects spread across 4 days is a person doing their job.

Every row gets a decision before you flip to enforce.

u_eve stays refused. svc_reporting is a legitimate integration, so it gets migrated first: give the key membership on those 3 projects, or a scoped export grant, and watch its row disappear from the next run. u_dana turns out to be an org admin, which means the new rule is missing a branch. Add the admin check to newRule and rerun the shadow week.

Write the refusals you expect into a table, because the after-deploy query reads it:

INSERT INTO authz_expected_refusals (rule, caller_id, reason) VALUES
  ('export.project_member', 'u_eve', 'walked 290 project ids in 36 minutes');

Seven days of shadow traffic catches daily and weekly callers. A job that runs on the first of the month only shows up if the 1st falls inside that week.

After deploy, compare per caller

The day after you flip to enforce, take every caller who got a 403 on the route since the deploy and put their history next to it:

WITH before AS (
  SELECT caller_id,
         count(*) FILTER (WHERE status < 400)  AS ok,
         count(*) FILTER (WHERE status = 403)  AS refused
  FROM access_log
  WHERE route = 'GET /projects/:id/export'
    AND ts >= :'deployed_at'::timestamptz - interval '35 days'
    AND ts <  :'deployed_at'::timestamptz
  GROUP BY caller_id
), after AS (
  SELECT caller_id,
         count(*) FILTER (WHERE status = 403)  AS refused
  FROM access_log
  WHERE route = 'GET /projects/:id/export'
    AND ts >= :'deployed_at'::timestamptz
  GROUP BY caller_id
), judged AS (
  SELECT a.caller_id,
         coalesce(b.ok, 0)      AS ok_before,
         coalesce(b.refused, 0) AS refused_before,
         a.refused              AS refused_after,
         CASE
           WHEN e.caller_id IS NOT NULL THEN 'expected'
           WHEN coalesce(b.ok, 0) > 0   THEN 'REGRESSION'
           ELSE 'new caller'
         END AS verdict
  FROM after a
  LEFT JOIN before b ON b.caller_id = a.caller_id
  LEFT JOIN authz_expected_refusals e
         ON e.caller_id = a.caller_id AND e.rule = 'export.project_member'
  WHERE a.refused > 0
)
SELECT * FROM judged
ORDER BY verdict = 'REGRESSION' DESC, refused_after DESC;

Run it with psql -v deployed_at="$DEPLOYED_AT" -f after-deploy.sql. The lookback is 35 days on purpose, so a monthly job counts as a caller that used to succeed.

    caller_id    | ok_before | refused_before | refused_after |  verdict
-----------------+-----------+----------------+---------------+------------
 svc_month_close |         6 |              0 |             6 | REGRESSION
 u_eve           |       312 |              0 |            57 | expected
 u_finn          |         0 |              0 |             2 | new caller

svc_month_close is the monthly job the shadow week couldn’t see. It exported 6 projects 30 days ago, ran again after the deploy and got 6 refusals. It never made the expected list, so the query calls it a regression.

When a REGRESSION row shows up, I flip EXPORT_AUTHZ back to shadow, migrate the caller, and flip it to enforce again. The exception is a hole someone is actively using. u_eve tried 57 more times after the deploy, and I wouldn’t reopen the route for everyone to unblock one job. In that case I leave enforce on and add a narrow allow for the one regressed caller until it’s migrated.

u_finn has no history on the route. That’s either an integration someone set up yesterday or a second person probing, and it takes one look at their requests to tell which.

Count the 403s where they’re recorded

Check that your access log records the 403s at all. In NestJS, guards run before interceptors, and a guard that returns false makes Nest throw a ForbiddenException, so a request logger built as an interceptor never sees the refusals. In Express, a logger mounted after canExport misses them too, since sendStatus(403) never calls next(). Pull the counts from the proxy or load balancer access log, or from a middleware mounted first that writes on the response’s finish event.

If you have an access-control fix waiting for review, wrap it in the shadow check today and run the first query when a week of traffic has gone through it.