Turning a job done twice into a nightly ticket

· Maksym Lyskov

You ran a job with an agent last week and it went fine. This week you typed almost the same prompt and it went fine again. Next week you’ll do it by hand a third time, and the first busy week after that, you skip it.

After two clean runs I’d automate it, as long as the automated version goes through the same review as everything else.

Both examples below are made up, and so is everything in them:

  • Speed rounds. Every so often you point an agent at slow requests and ask for speedups under one rule: responses stay identical, only faster.
  • Dead feature flags. Once a week someone hunts for flags that have sat fully on (or fully off) for ages and rips them out.

When a job is ready

One good run could be luck. After two that looked the same, the steps are basically sitting in your prompt history.

I check how much I changed between the two prompts. Same inputs (slow-request logs, the flag config), same rule, same proof at the end? Then there’s little left for a person to decide each time. If I had to argue scope with the agent or invent a rule halfway through, I’d run it by hand once more.

Diff your two prompts. Whatever you typed both times is your ticket template.

The command files a ticket

The scheduled command reads logs and code and writes one ticket into the normal queue. Its permissions don’t include editing files or pushing.

No one watches it at 3am, so its output should be something a person picks up in the morning. From there it gets a branch, tests, a PR and a reviewer, like any other ticket.

With Claude Code that’s a saved command run in headless mode (claude -p "/flag-sweep") from whatever scheduler you already have. Without Claude Code, any scheduled script works if it reads code and logs and can only open issues, plus the CI check below.

The flag job as a command file:

---
description: Find dead feature flags and file one cleanup ticket
allowed-tools: Bash(flagctl list:*), Bash(git grep:*), Bash(gh issue list:*), Bash(gh issue create:*), Edit(sweep-report.md)
---
$ARGUMENTS: empty, or "dry-run" to write the ticket to sweep-report.md and file nothing.

1. Run `flagctl list --json`. Statuses other than on/off/rollout → halt UNKNOWN_STATUS.
2. Keep flags at 0% or 100% for 30+ days. Drop any listed in keep-flags.txt.
3. Drop flags named in an open `flag-cleanup` ticket's "Flags:" line.
4. `git grep` each remaining key and record every file that reads it.
5. Nothing left → halt NOTHING_TO_FILE.
6. File ONE ticket from the template, label `flag-cleanup`.
7. Append to sweep-report.md: date, found, skipped, ticket url or halt code.

Halts: NOTHING_TO_FILE (worked, found nothing) · FLAGCTL_FAILED: <first error line> · UNKNOWN_STATUS: <status>

Keep the halt codes. When the log is quiet, you need to tell a run that found no dead flags from one where flagctl died before it looked.

What goes into the ticket

The ticket is written for the agent that builds it and the reviewer after. Both need the rule and the proof the PR owes, or the builder decides for itself what “done” means.

A ticket the speed job might file:

Title: Speed round: 2 slow reads in orders (filed by the nightly speed sweep)

Rule: identical responses. Same body, status code and error message
for every request below. Only the time changes.

Targets:
- GET /orders/:id/history  p50 900 ms. Loads every past order, then
  filters in memory. Likely fix: filter in the query.
- POST /orders/:id/notes   reads the order twice.

Out of scope: caching, response shape changes, writes other requests depend on.

Proof the PR must carry:
- Identity: replay fixtures/orders-replay/ against main and the branch,
  normalised. Paste `Identity: 50 requests, 0 diffs`.
- Timing: before and after per target, same data.
- A test per target that fails on main and passes on the branch.

Endpoints: GET /orders/:id/history, POST /orders/:id/notes

What matters in that ticket: one rule, exact targets, what’s off limits, and a proof line CI can grep for. The Endpoints: line lets the next night’s run skip anything an open ticket already claims.

The recorded requests replay against a database snapshot restored fresh before every run, and they only get re-recorded in a PR of their own, never as a side effect of a speed branch.

The flag ticket’s proof differs: git grep finds no reads, the code keeps the branch the flag served, and tests pass with the flag gone from config.

Identical, after normalising

Byte-for-byte identity breaks on the first real endpoint. Responses carry request ids, generatedAt timestamps, and lists in whatever order the query plan picked, so two correct builds differ every time.

So the replay normalises both sides before it compares, and the normaliser lists every field allowed to differ:

# normalise.sh: the only differences the identity check forgives
jq -S 'del(.. | .requestId?, .generatedAt?) | .items |= sort_by(.id)' "$1"

Sort a list only when the API doesn’t promise its order. And keep data timestamps in, like an order’s createdAt; the snapshot pins those, so they should match. I’d make any change to this file need a human reviewer, since every line there is a difference the check will let through.

Make the proof a required check

If the PR doesn’t carry the proof, it goes back, even if the diff looks fine.

Put that in CI and mark it required in branch protection. It applies to any PR with the label, whether the nightly job filed the ticket or a person did.

# .github/workflows/proof.yml
on:
  pull_request:
    types: [opened, edited, synchronize, labeled, unlabeled]
jobs:
  proof:
    runs-on: ubuntu-latest
    steps:
      - if: contains(github.event.pull_request.labels.*.name, 'speed')
        env:
          BODY: ${{ github.event.pull_request.body }}
        run: printf '%s\n' "$BODY" | grep -q '^Identity: .* 0 diffs$'

It runs on every PR and skips the step without the label, so the required status never hangs as pending.

A deploy step you control can also compare the new build’s first minutes with the previous one and print a revert command when the targets got slower. A person runs it.

When the ticket path is too heavy

A single dead flag read in 3 lines isn’t worth a ticket, an agent run and a review.

Two things help. The command files at most one ticket per run, so small finds get batched into it. And below a size you pick (say, under 10 changed lines, total), it files nothing and writes a “Held” section in the report. Anything held for two weeks goes into the next ticket, whatever its size.

The first weeks: every decline becomes a rule

The first version will file things you’d never file by hand. So for a few weeks, read every ticket, and give every decline a reason in one line.

Say the sweep files a ticket to remove legacy_tax_fallback, at 100% for 90 days. The reviewer declines it: that flag is the kill switch for falling back when the tax provider is down, and it’s supposed to sit at 100% forever. The reason turns into a line in keep-flags.txt:

legacy_tax_fallback  # kill switch for the tax provider outage path, never remove

Next night, step 2 drops it. Every decline should end like that, as a skip list entry, a threshold or a line under “Out of scope”. If a decline doesn’t change the command, the same ticket shows up again next week.

From day one, run in dry-run for a week and read the would-be tickets, and keep the report line even when nothing is found.

Noticing when the job has drifted

The job changes under the command. The flag tool grows a new status, the slow endpoints move, the identity rule needs a new exception. The command keeps running, and its output goes wrong without anything failing.

What I’d watch:

  • Unknown input shape halts. Step 1 above stops on a status it doesn’t know. Silently ignoring it would read as “no dead flags”.
  • Main against main. Before each speed run, replay the fixtures against main twice. Any diff means the recordings or the normaliser went stale; halt and fix those first.
  • A month of NOTHING_TO_FILE. Could be true, could be a broken filter. Check one by hand.
  • Shipped tickets that didn’t help. The next speed run measures last week’s targets. A fix that moved nothing points at the wrong bottleneck.
  • The decline rate. If most tickets still get declined after a month of tuning, I’d switch it off and run the job by hand again.