The reliability layer of the Softcery conversational AI stack records every call as correlated traces, logs, and metrics, derives latency and failure signals from the record, and alerts on them. It runs on customer infrastructure with no external service. Measurements below come from the reference deployment: the agent operating softcery.com.

1. Design principles

1. Capture first. A signal not recorded during the call cannot be examined after it.
2. Complete accounting. Derived timings must sum to the measured total. An unexplained gap is a defect.
3. Root cause over proximate cause. A failed turn is labeled with the fault requiring operator action.
4. Effect over acknowledgment. A command that changes no state returns an error.
5. Configuration as code. Dashboards, alert rules, and probes are versioned and installed from the repository. Server divergence is a defect.

2. Signal model

The core emits OpenTelemetry (OTLP) for traces, logs, and metrics; the deployment persists them. All three carry trace id and turn number, so one call is queryable across signals.

2.1 Traces

2.2 Logs

One JSON stream: application logs, third-party library logs, every model-provider HTTP request with retry and status. Each line carries trace id, span id, turn number.

2.3 Metrics

Span-derived metrics, host metrics, provider response counters by HTTP status, discard counters (§6.1). Histograms carry exemplars linking to the source trace.

3. Query paths

• Dashboard to call: data point, exemplar, trace, transcript, logs.
• Call to cause: failure label, stage marks, provider log lines under one trace id.
• Fleet: per-call table with outcome, turn count, tool count, trace link.

4. Latency model

Response time = caller's last word to first agent audio, decomposed into five stages:

Invariants: stages sum to the total, residual gap is a defect; one clock per layer, no stage spans two clocks; names state what is measured. stt_wait measures waiting, not transcription speed; when the streaming transcript finalizes before end of speech, it reads zero.

Measured on the reference deployment, the agent operating softcery.com (one turn, rented-inference configuration): 573ms = llm 418 + opener 23 + tts 131 + 1 residual; stt_wait 0.

5. Failure taxonomy

A turn producing no audio is an error with exactly one label, selected from a per-call fault ledger:

Precedence: a turn interrupted while holding a rate-limit error is rate_limited, not superseded.

6. Silent-failure closure

Two failure classes emit no error signal.

6.1 Message discard

Condition: message dropped, no effect, no log (observed: event name unknown to the receiver after a protocol change). Closure: every discard increments a reason-labeled counter (unknown_event, unparseable_journal, inbox_overflow); any non-zero count alerts.

6.2 Accepted no-op

Condition: command accepted, success returned, no state change (observed: field-name mismatch between sender and receiver schema). Closure: input validated against the published schema before dispatch, unknown fields rejected; a command changing nothing returns an error containing the current state.

7. Evaluation

Operational: transcript replay through the pipeline; word-error-rate harness against a domain corpus. Recognizer-only WER on the reference corpus: 10.1%, reproducible to the digit across runs.

Not operational: automated scoring of production conversations against human-checked labels; release gates on measured conversation quality.

Same record feeds knowledge-retrieval evaluation, per-language scoring, cost accounting. The layer also operates against third-party agents (audits).

8. Alerting

Rules are code (§1.5), severities page and ticket, routed to an on-call webhook. Reference-deployment budgets:

9. Health probes

Three layers; each covers a failure the layer below reads as healthy:

10. Storage and data custody

Tempo (traces), Loki (logs), Mimir (metrics), Grafana (dashboards, alerting), all on customer infrastructure. Custody rule: recordings and transcripts are customer data; a boundary prohibiting conversations to third parties prohibits telemetry about them too. The layer requires no external service. Wire format is standard OTLP; a corporate monitoring backend can receive the same signals where policy permits.

11. Known limitations

1. Health endpoint returns 200 without checking pipeline state; a hung pipeline reads as up on probe layers 1 and 2. Readiness check planned.
2. Provider error counters cover HTTP only; websocket speech-provider auth/billing failures are not counted.
3. Tool arguments and results truncate at 256 characters.
4. No conversation-quality release gate exists (§7).