Agent-harness ergonomics: non-blocking default, distinct exit codes, JSON errors, status preservation #2

New issue

Closed

opened 2026-07-30 08:43:26 +00:00 by jercik · 0 comments

jercik commented

2026-07-30 08:43:26 +00:00

Owner

The operational rule that teaches agents to drive this CLI safely (rules/tropkod/tropkod-usage.md in j4k/align) grew to ~17.5 KB across 15 commits of adversarial review (setup-atlas#25, align#153/#161), and a forensic classification of all 103 review findings showed most of that prose exists to compensate for client behaviors, not to describe the service. Fixing them here would let consumers delete most of the rule. Every citation below was verified against v1.0.0 (current main has no src/ drift from the tag).

1. Blocking default `--wait-ms 120000`

src/cli.ts:42 defaults --wait-ms to 120_000; the submit path awaits a single long-poll POST /v1/queries (src/remote-query-client.ts:24-29). That default exactly equals Claude Code's default Bash tool timeout, so an agent that omits the flag is killed mid-wait. The job id only arrives in the response, and with no recovery route (item 4) the paid analysis is orphaned while the server runs it to completion on the single-slot service.

Proposal: default to 0 (submit-and-return) — or at minimum when stdout is not a TTY.

2. Exit 1 conflates every failure source

src/cli.ts:87-90 is a catch-all that exits 1 for: pre-dispatch validation (src/resolve-submission-request.ts:26,33, src/read-remote-client-options.ts:14,19, src/parse-integer-option.ts:7), transport failure and timeout (src/remote-query-client.ts:64,66), server refusal — 4xx and 5xx alike (src/remote-query-client.ts:69-72), schema-parse failure on an accepted 200 (src/remote-query-client.ts:74), and a failed job (src/format-submission.ts:47-55). The safety-relevant question for a consumer is "could a job have been created?" — the answer differs per source (validation: no; refusal: no; transport after dispatch: unknowable; parse-after-accept: yes, id lost), yet all exit 1, forcing consumers to triage by stderr wording, which is version-fragile.

Proposal: distinct exit codes for at least pre-dispatch validation, transport/timeout, server refusal, and parse-failure-after-accept.

3. Errors are plain text on stderr even under `--json`

The try/catch sits outside the commander action (src/cli.ts:85-90), where options.json isn't even in scope; errors always go through formatErrorMessage (src/format-error-message.ts:2-5) as plain text on stderr. Piping --json output into jq therefore yields empty stdout on every failure, and the pipeline reports jq's exit status, not the client's.

Proposal: with --json, emit a JSON error envelope on stdout carrying a kind discriminator (which would also deliver item 2's partition machine-readably).

4. No recovery path for a lost job id

The client has exactly two routes: POST /v1/queries (src/remote-query-client.ts:27) and GET /v1/jobs/<id> (:38). --session is submit-only input (src/cli.ts:36,68); nothing consumes it for lookup. A killed submit that never printed its id is a dead end — the consumer must report-and-stop while a paid analysis runs invisibly.

Proposal: a job-listing or session-lookup route (server + client), or at minimum documented orphan semantics.

5. `readErrorMessage` drops the HTTP status on recognized JSON error bodies

src/remote-query-client.ts:87-104: when the body parses as either recognized JSON error shape, the returned message is the body's text with status unused (:92, :97); the status survives only in the :103 fall-through. A 5xx whose body happens to read like a request refusal is indistinguishable from a real 4xx refusal — and the consumer's resubmit decision hinges on exactly that distinction, where a misread buys a duplicate paid analysis.

Proposal: always include the numeric status in the message (and in item 3's envelope).

Items 1, 2, and 4 are jointly why the rule's "accounting test" section exists; item 3 is why it mandates capture-before-parse; item 5 is why it teaches reading refusals "narrowly". Each fix retires its block of prose. A --poll mode was also suggested in review (align#153, review 7441) — worth considering, with the caveat that any long-running foreground mode inherits item 1's harness-timeout hazard, so heartbeat output and resumability matter more than the loop itself.

The operational rule that teaches agents to drive this CLI safely (`rules/tropkod/tropkod-usage.md` in j4k/align) grew to ~17.5 KB across 15 commits of adversarial review (setup-atlas#25, align#153/#161), and a forensic classification of all 103 review findings showed most of that prose exists to compensate for client behaviors, not to describe the service. Fixing them here would let consumers delete most of the rule. Every citation below was verified against `v1.0.0` (current `main` has no `src/` drift from the tag). ## 1. Blocking default `--wait-ms 120000` `src/cli.ts:42` defaults `--wait-ms` to `120_000`; the submit path awaits a single long-poll `POST /v1/queries` (`src/remote-query-client.ts:24-29`). That default exactly equals Claude Code's default Bash tool timeout, so an agent that omits the flag is killed mid-wait. The job id only arrives in the response, and with no recovery route (item 4) the paid analysis is orphaned while the server runs it to completion on the single-slot service. **Proposal:** default to `0` (submit-and-return) — or at minimum when stdout is not a TTY. ## 2. Exit 1 conflates every failure source `src/cli.ts:87-90` is a catch-all that exits 1 for: pre-dispatch validation (`src/resolve-submission-request.ts:26,33`, `src/read-remote-client-options.ts:14,19`, `src/parse-integer-option.ts:7`), transport failure and timeout (`src/remote-query-client.ts:64,66`), server refusal — 4xx and 5xx alike (`src/remote-query-client.ts:69-72`), schema-parse failure on an accepted 200 (`src/remote-query-client.ts:74`), and a `failed` job (`src/format-submission.ts:47-55`). The safety-relevant question for a consumer is *"could a job have been created?"* — the answer differs per source (validation: no; refusal: no; transport after dispatch: unknowable; parse-after-accept: yes, id lost), yet all exit 1, forcing consumers to triage by stderr wording, which is version-fragile. **Proposal:** distinct exit codes for at least pre-dispatch validation, transport/timeout, server refusal, and parse-failure-after-accept. ## 3. Errors are plain text on stderr even under `--json` The `try/catch` sits outside the commander action (`src/cli.ts:85-90`), where `options.json` isn't even in scope; errors always go through `formatErrorMessage` (`src/format-error-message.ts:2-5`) as plain text on stderr. Piping `--json` output into `jq` therefore yields empty stdout on every failure, and the pipeline reports jq's exit status, not the client's. **Proposal:** with `--json`, emit a JSON error envelope on stdout carrying a `kind` discriminator (which would also deliver item 2's partition machine-readably). ## 4. No recovery path for a lost job id The client has exactly two routes: `POST /v1/queries` (`src/remote-query-client.ts:27`) and `GET /v1/jobs/<id>` (`:38`). `--session` is submit-only input (`src/cli.ts:36,68`); nothing consumes it for lookup. A killed submit that never printed its id is a dead end — the consumer must report-and-stop while a paid analysis runs invisibly. **Proposal:** a job-listing or session-lookup route (server + client), or at minimum documented orphan semantics. ## 5. `readErrorMessage` drops the HTTP status on recognized JSON error bodies `src/remote-query-client.ts:87-104`: when the body parses as either recognized JSON error shape, the returned message is the body's text with `status` unused (`:92`, `:97`); the status survives only in the `:103` fall-through. A 5xx whose body happens to read like a request refusal is indistinguishable from a real 4xx refusal — and the consumer's resubmit decision hinges on exactly that distinction, where a misread buys a duplicate paid analysis. **Proposal:** always include the numeric status in the message (and in item 3's envelope). --- Items 1, 2, and 4 are jointly why the rule's "accounting test" section exists; item 3 is why it mandates capture-before-parse; item 5 is why it teaches reading refusals "narrowly". Each fix retires its block of prose. A `--poll` mode was also suggested in review (align#153, review 7441) — worth considering, with the caveat that any long-running foreground mode inherits item 1's harness-timeout hazard, so heartbeat output and resumability matter more than the loop itself.

jercik referenced this issue from a commit

2026-07-30 09:45:55 +00:00

feat!: non-blocking default, distinct exit codes, JSON error envelopes

jercik referenced this issue from a pull request that will close it,

2026-07-30 09:46:10 +00:00

feat!: non-blocking default, distinct exit codes, JSON error envelopes #3

jercik closed this issue

2026-07-31 11:24:33 +00:00

jercik referenced this issue from a commit

2026-07-31 11:24:34 +00:00

feat!: non-blocking default, distinct exit codes, JSON error envelopes (#3)