AI content checks in Runtime
Nimi runs content checks directly in the Runtime on the user's device. Every app that calls text.generate gets the same inspection for prompts, generated output, and complete tool-call arguments—whether the request runs on a local model or a cloud provider.
Suppose a research app reads a file containing a private key and includes it in the next prompt, or a tool returns a token that gets fed back into the conversation. If that content matches one of the rules below, Runtime rejects the request before it reaches the model. Model output is checked too. Apps keep their own workflows without each maintaining a separate scanner.
Developer preview
Content checks are available in current Runtime source builds. For prebuilt downloads, check the release notes for your version. Apps do not need new request fields, registration steps, or configuration to use them.
How local inspection works
Runtime already resolves caller identity, holds provider credentials, and dispatches model requests. Running deterministic format checks in the same place means protection does not depend on whether a model obeys safety instructions in a prompt.
Inspection runs entirely on the user's machine with no network dependency and no external scanning service. When an app uses a cloud route, requests that pass inspection are still sent to the selected model provider.
Responsibilities stay distinct across the stack:
| Responsibility | Owner |
|---|---|
| Inspect business text in model inputs, outputs, and complete tool arguments | Runtime AI |
| Verify identity, permissions, credential custody, and resource access | Existing Runtime resource services |
| Execute tools, manage multi-turn state, render output, and obtain business approval | Your App host |
Passing a content check does not grant permission, and model output such as approved: true never establishes authorization. Integration business results and Delegation approval policies work as before.
What gets checked and when
Current inspection covers text.generate in SYNC and STREAM execution. Candidate generation and RuntimeAgent calls are included whenever they use that path. Other AI capabilities, raw image/audio/video bytes, opaque continuity payloads, and calls outside text.generate are not covered.
| Stage | Checked content | Result of a match |
|---|---|---|
| Model input | Normalized business text and JSON string values: system text, selected history and summaries, tool descriptions and schemas, tool results, and visible media-reference strings | Refused before model dispatch, media materialization, request-body capture, or new Job creation |
| Streaming output | Generated text and visible reasoning summaries, checked incrementally | Stops undelivered output and reports failure through the existing stream and Job path |
| Tool call | Complete arguments after decoding JSON escapes into actual values | Refused before forwarding the tool call to the App |
| Final result | Naturally complete text, visible summaries, and tool arguments | Refused before flushing final buffered output, completing normally, or storing a successful result |
Normal output keeps its existing chunking and buffering behavior. Inspection adds no timers, full-step buffering, or new output-length limits. Because a transport chunk boundary is not always a value boundary, Runtime can confirm some matches only when a delimiter or the end of a logical item arrives. Normal text continues to stream in the meantime.
In streaming mode, text already delivered before a match is confirmed cannot be recalled. Content checks help catch accidental credential exposure, but they do not guarantee zero leakage, act as an operating-system sandbox, or prevent all prompt injection, malicious model behavior, or data exfiltration.
Current rules
The initial rules target specific, well-defined credential formats rather than blocking every sensitive word or high-entropy string.
| Rule | Match condition | Inspection mode |
|---|---|---|
pem-private-key | Complete, unencrypted PEM private keys that parse as PKCS8 (PRIVATE KEY), PKCS1 (RSA PRIVATE KEY), or SEC1 (EC PRIVATE KEY) | Complete values only |
github-classic-pat | ghp_ followed by exactly 36 ASCII letters or digits, with identifier boundaries on both sides | Complete values and incremental text |
For github-classic-pat, ASCII letters, digits, underscores (_), and non-ASCII bytes continue an identifier, so a match only triggers when the token stands on its own. Matching is purely structural—Runtime does not verify whether a credential was actually issued or remains active.
Other token formats, encrypted or OpenSSH private keys, and arbitrary encodings are not covered. Placeholders and ordinary technical text—such as token: string, password: <redacted>, empty values, hashes, standalone PEM headers, public keys, certificates, generic Bearer wording, sensitive field names, high-entropy strings, or prompt-injection phrases—do not trigger refusal on their own.
Local and cloud routes apply the same rules. A local request containing a private key in one of the covered formats is refused even for legitimate analysis. You can supply a redacted copy through your App’s existing input controls; Runtime never silently rewrites your input.
Handle a refusal in your App
When Runtime blocks content, it returns gRPC FailedPrecondition with reason AI_CONTENT_FILTER_BLOCKED (and exposes ai-content-filter-blocked on the protected App carrier). Branch on the typed error or event from your SDK rather than parsing user-facing error messages.
- Input refusal: The call fails before Runtime creates a Job or opens an output stream. Handle it as a call error; there is no stream
failedevent or successful result. - Output refusal: An active stream reports failure through the existing
failedpath (and records Job failure) when the connection can still deliver it. Treat the entire model turn as failed, and do not execute any tool calls received earlier in that failed batch. - User guidance: Tell the user that content filtering blocked the request or response, and ask them to review and redact the input or selected context before submitting again. Do not automatically retry, replay tools, or switch models, providers, or local/cloud routes.
A clear user-facing message is:
This request or response did not pass the content check. Review the input and selected context, then remove or redact restricted content before submitting again.
Cancellation, timeouts, permission errors, Runtime shutdown, and transport failures keep their own error codes and priority. Content-check diagnostics record the rule, stage, and related information without storing the matched secret. Known usage may remain in the existing Job record; missing usage does not mean a failed call was free.
History and conversation recovery
Runtime never globally rewrites or scrubs stored App history, and a diagnostic event does not mean earlier records were cleaned. Old turns remain readable, but any history or summary selected into a new request is checked again. If matched material stays in the selected context, even a harmless follow-up question will be refused.
Apps can let users remove the offending material through their existing context-editing or new-conversation controls. Reopening a RuntimeAgent conversation reuses its anchor rather than clearing context, and Runtime does not add automatic history cleanup, history exemptions, or a new reset protocol. If a conversation still includes blocked context and your App has no existing recovery entry point to remove it, tell the user clearly that inference cannot continue in that conversation.