Search & RAG — reference
Exhaustive reference for content search and the three streaming inference surfaces: every parameter, field, mode, limit, error code, the stream event vocabulary, and a "Notes & limits" section stating what each feature does not do.
For request/response wire detail at the endpoint level, see the generated API reference (the OpenAPI / Scalar spec). This page documents the SDK-level surface and the behavior that the spec alone does not capture.
Version note. The API spec is at
0.50.0. Nothing on the search side of this page is 0.26-only, so it works on any current client. The optional inference flagallowGlobalRegion(region serving, below) is a 0.27 addition, and the optionalreasoningfield and thereasoning_deltaevent are a 0.48 addition. The milli-cent amounts (inferenceBalanceCentsChargedMilli, andbalanceCentsMilliin the usage report),cachePricingand the cache counts need a 0.49 client. Where the SDK method name differs from a raw field name, the SDK name is given.
Content search — client.search.content(req)
Unified search across documents and records. Requires the search:r scope to call. Results
are additionally enforced per row against what the calling token could read directly: a
scoped token sees only results it holds the read grant for — documents:r for documents,
records:r for records, each optionally narrowed to a single type with a qualifier
(records:r:<type>, documents:r:<type>) — and only within its ownership data scope. A
document's type is the type of the schema it is bound to; an untyped document matches only an
unqualified documents:r grant. A token with search:r but no read grant returns an empty result set.
The same per-row enforcement applies to RAG retrieval (/v1/rag).
Request parameters
| Parameter | Type | Default | Notes |
|---|---|---|---|
query | string | — (required unless filters holds a $near) | Natural-language or keyword query. Optional when filters holds a $near: with no query, a $near returns the matches within its radius, nearest first (Distance search below). |
mode | 'TEXT' | 'SEMANTIC' | 'HYBRID' | HYBRID | Keyword relevance / meaning-based similarity / fused ranking. |
limit | integer | 20 | Max results. Valid range 1–100; out-of-range is rejected with 400. |
offset | integer | 0 | Results to skip — paging (see Paging below). Valid range 0 to 200; out-of-range is rejected with 400. |
contentTypes | ('documents' | 'records')[] | both | Narrow to one content type. Omitted, empty, or both values ⇒ unified. |
typeName | string | — | Restrict hits to one schema type — for example patient or runbook. Applies to both documents and records: any item whose bound schema type matches. On its own it narrows both content types; combine it with contentTypes to narrow within one (e.g. contentTypes: ['documents'] + typeName: 'runbook' ⇒ runbook documents). Untyped content (no schema) never matches. Documents ingested before this facet shipped must be reindexed to be matchable — see the note below. |
folderId | string (uuid) | — | Restrict to content in this exact folder. |
rootFolderId | string (uuid) | — | Restrict to this folder and all descendants. Use instead of folderId. |
userId | string (uuid) | — | Ownership filter — content owned by this user. |
scope | string | — | Ownership filter — content owned by this entity, as <namespace>:<value> (org:..., client:..., or a namespace you registered). One entry per query; pair it with userId for a second dimension, or use scopeFilters (below) for more than one scope namespace at once. Mutually exclusive with scopeFilters. |
scopeFilters | string[] | — | Ownership filter for MORE THAN ONE scope namespace in a single query — e.g. ["org:<id>", "client:<id>"] to narrow to one specific client within one specific org. Each entry uses the same <namespace>:<value> form as scope. Naming the same namespace twice is rejected (400), as is supplying both scope and scopeFilters. Capped at 16 entries. |
filters | object | — | Field-level metadata filters; AND-combined across keys. See Filter grammar. |
createdAfter | string (ISO-8601) | — | Content created at/after this UTC timestamp. |
createdBefore | string (ISO-8601) | — | Content created at/before this UTC timestamp. |
uniqueDocuments | boolean | true | When true (the default), at most one hit per source document; set false for chunk-level results, where several chunks of one document can each be a hit. |
minSimilarity | number 0.0–1.0 | 0.15 | Minimum semantic similarity; hits below are excluded (semantic / hybrid). In HYBRID the floor drops only a hit that matched on meaning alone: with uniqueDocuments: true, a hit the keyword leg also matched is kept. Send 0 to turn the floor off. |
minTextRelevance | number 0.0–1.0 | 0.3 | Relative keyword-relevance floor, as a fraction of the top hit's score (e.g. 0.5 keeps hits at least half as relevant as the best). Applies to HYBRID only (TEXT mode does not apply it). Send 0 to keep all keyword matches. |
textMode | 'OR' | 'AND' | 'PHRASE' | 'COMPLEX' | PHRASE in HYBRID, OR in TEXT | Keyword sub-mode (see Keyword sub-modes). Applies to TEXT / HYBRID. |
slop | integer ≥0 | 3 when textMode is omitted in HYBRID; otherwise 0 | Phrase-match slop: intervening positions tolerated between terms when textMode='PHRASE'. Ignored otherwise. |
requireComplete | boolean | false | Fail-closed override: return 503 instead of degraded partial results when a leg is unavailable. |
sortBy | 'distance' | — (relevance order) | Nearest first. Requires a $near filter, and either no query or mode: 'TEXT'; rejected with 400 for HYBRID and SEMANTIC. See Distance search. |
Scoping documents by type — reindex note. The document side of the
typeNamefacet relies on an index key that is written when a document is indexed. Records have always carried it; documents ingested before this facet shipped do not, so they will not match atypeNamefilter until they are reindexed. New and updated documents pick it up automatically. To make older documents matchable, re-index them (re-ingest, or touch them so they re-index). Records are unaffected.
Keyword sub-modes (textMode)
For TEXT and HYBRID searches, textMode controls how query terms combine in the
keyword leg:
OR(default inTEXT) — match any term; broadest recall.AND— require all terms; higher precision.PHRASE(default inHYBRID, with aslopof 3) — require terms as a contiguous sequence (tunable withslop).COMPLEX— full keyword query syntax (boolean operators, field-scoped clauses, range filters). Use only when you need expression-level control; the query is parsed as a structured expression rather than a bag of terms.
Filter grammar (filters)
Each top-level key is a metadata field declared filterable on the schema (or a built-in document field). Top-level keys are AND-combined. Each value is one of:
- Scalar (string / number / boolean) — equality, e.g.
{ "status": "open" }. - Array of scalars — OR-set (match any), e.g.
{ "tag": ["red", "blue"] }. - Operator map — closed set of operators:
- Scalar operand:
$eq,$ne,$gt,$gte,$lt,$lte. Operators in one map are AND-combined, so{ "price": { "$gte": 100, "$lte": 500 } }is a closed range. - Array operand:
$in,$nin. Cannot be combined with other operators in the same map.
- Scalar operand:
Numbers and booleans match typed (the field must have been ingested under a typed schema).
A numeric filter operand is held to the same signed 64-bit range as stored values — an
out-of-range number in the request body is rejected with 400; match a large id you stored
as a string with string equality, not a numeric operator.
Dates may be ISO-8601 strings or epoch millis. Filter keys are validated (^[A-Za-z_] [A-Za-z0-9_-]*$); unknown operators, non-scalar operands, malformed keys, and any attempt
to filter on a reserved tenancy/ownership key are rejected with 400. You cannot widen your
access through the filter map — ownership scope is enforced separately.
Distance search ($near)
A field of type geo (a limited pilot: geo fields are refused unless the pilot is enabled for your account) takes one
more operator, $near:
{ "filters": { "location": { "$near": { "lat": 47.6062, "lon": -122.3321, "radiusMeters": 5000 } } } }
- It keeps only content whose location is within
radiusMetersof the point. Distance is great-circle distance on a sphere (mean Earth radius 6,371,008.8 m), inclusive. It is a spherical model, so it differs slightly from a distance measured on the WGS84 ellipsoid. latmust be in [-90, 90],lonin [-180, 180],radiusMetersin [1, 2,000,000]; all three are required numbers.$nearmust be the only operator in its map, and at most one$nearis allowed per request. Anything else is rejected with400.- Every result then carries
distanceMeters(rounded to 0.1 m). - Three ways to use it:
- No
query: the matches within the radius, nearest first (equal distances in a stable order). It runs on the keyword index whatevermodesays, and is billed as aTEXTsearch. mode: 'TEXT'withsortBy: 'distance': the keyword matches within the radius, nearest first.- A search ordered by distance (either of the two above) returns at most one result per item;
uniqueDocumentsdoes not apply. - A query with no
sortBy, any mode: results keep their relevance order, filtered to the radius. - A
TEXTsearch with a$near(with or withoutsortBy) carries no keyword-highlightedsnippet; it falls back to an excerpt. - Content indexed with
indexMode: SEMANTICis in the semantic index only, so a search with noquery, or aTEXTsearch, does not find it; aHYBRIDorSEMANTICsearch with a query does.
- No
- A search ordered by distance is exact within fixed per-request limits. When a very dense
area exceeds them, the response sets
incompleteResults: true: the results are still in exact distance order, but nearer matches may exist that were not returned. Narrow the radius or add filters. In such a searchtotalResultsis a lower bound. - Content whose stored location could not be read is indexed without it and is not found by
$near; itsindexFailure.codeisLOCATION_NOT_INDEXED(see the operations-trust reference). - Without
typeName, a$nearon a field that is not of typegeomatches nothing: the result is empty, not an error. - With
typeName, a$nearon a field that no schema of that type declares asgeois a400. This check does not apply to a key that reaches every app context: that key gets an empty result. requireComplete: truealso returns a503instead of a search ordered by distance that would carryincompleteResults: true.$nearis accepted by content search only. RAG inference rejects it with400.
Response shape
search.content returns a flat object (it is not wrapped in the { data, nextCursor }
cursor envelope used by list/lookup endpoints — see Paging):
| Field | Type | Notes |
|---|---|---|
results | SearchResult[] | Matched chunks, ranked (highest score first). May be empty. |
totalResults | integer | Approximate matching-pool size. 0 for a miss. Not a reliable paging signal — use hasMore instead. |
hasMore | boolean | True when more matching results exist past this page; false once you've reached the last page or the offset maximum (200). |
searchTimeMs | integer | Server-side execution time. Reported even on an empty result. |
degraded | boolean | True when one leg was unavailable and results came from the survivor only. |
degradedLegs | string[] | Which legs were unavailable: "text" (keyword) and/or "vector" (semantic). Empty when not degraded. |
incompleteResults | boolean | Present and true only on a search ordered by distance that reached its per-request limits before it could confirm the page (see Distance search). Absent otherwise. |
Each SearchResult:
| Field | Type | Notes |
|---|---|---|
documentId | string (uuid) | Source entity id. Use with getDocument / getRecord. (This is the source id, not an internal index id.) |
externalId | string | null | The matched item's externalId as you supplied it, alongside documentId — lets you correlate a hit back to your own record identity without a follow-up lookup. Null when the item was ingested without one, or was indexed before this field existed and hasn't been reindexed since. |
sourceType | 'PartnerDocument' | 'GenericRecord' | Document vs. record discriminator — the two literal strings the API returns; branch on it when rendering mixed results. |
score | number | Primary sort key, higher is more relevant. Its scale depends on mode — see Notes below; a HYBRID score is not comparable to a TEXT- or SEMANTIC-only score. |
textScore | number | Keyword (relevance) sub-score. Non-zero in HYBRID when the keyword leg contributed. In TEXT-only mode it's rank-derived (see Notes below) — meaningful for ordering within one response, not a raw BM25 magnitude. |
semanticScore | number | Semantic similarity sub-score. Non-zero in SEMANTIC / HYBRID. |
chunkText | string | The specific chunk that matched. Feed this (or contextText) to a model. |
contextText | string | The wider surrounding passage containing the chunk — better grounding context. |
snippet | string | Highlighted excerpt with query terms emphasized, for display. May be null for a semantic-only hit (use chunkText). |
metadata | object | Metadata supplied at ingest (title, folderId, custom fields). |
createdAt | string (ISO-8601) | Source content creation time. |
distanceMeters | number | Great-circle distance in meters (rounded to 0.1 m) from the $near point. Present only when the request has a $near. |
Paging
search.content has no nextCursor — it is not enveloped. Page by combining limit
with offset:
const page1 = await client.search.content({ query, mode, limit: 20, offset: 0 });
const page2 = await client.search.content({ query, mode, limit: 20, offset: 20 });
limit is capped at 100; consecutive pages are disjoint. This differs from list/lookup
endpoints, which return { data, nextCursor } and are drained by feeding nextCursor back
as startFrom. For pulling recent content deterministically, prefer a createdAfter
window over deep offset paging.
Notes & limits — search
- Index/search mode must line up. Content indexed for one strategy only will not appear in a search requiring the other. Indexing mode is a property of the content (set at ingest / on the schema); search mode is a property of the query.
- Only searchable fields participate in keyword relevance. A query matching only a non-searchable field returns nothing — by design.
- Sensitive fields are excluded from the index. Their values are not written to the search index, so search does not match, rank, or return them (index-time exclusion).
textScoreinTEXT-only mode is rank-derived, not a raw BM25 magnitude. That path returns highlighted snippets and already-correctly-ordered hits but never exposes the raw per-hit keyword score, sotextScorethere reflects the hit's relative rank — meaningful for ordering within one response, not comparable across requests or againstHYBRID/SEMANTIC.HYBRID'sscoreis a Reciprocal Rank Fusion (RRF) value, not a 0–1 confidence — its magnitude looks nothing likeTEXT/SEMANTIC's. RRF combines the keyword and semantic legs by RANK POSITION only (never by raw score), as1 / (k + rank)per leg, summed across legs it appears in, with the standardk = 60. That makes everyHYBRIDscore small and tightly clustered near the same constant: a hit ranked top on a single leg scores1/61 ≈ 0.0164; ranked top on both legs,2/61 ≈ 0.033— near the practical ceiling. This is expected, not a sign every result is a weak match: RRF is designed to fuse two differently-scaled rankings by ORDER, not to reproduce either leg's own scale, so don't read aHYBRIDscore of0.02as "2% relevant," and don't set a similarity-style threshold (e.g.score > 0.5) against it — it will discard everything. By contrast,TEXT- orSEMANTIC-only mode'sscoreis the search engine's own native score for that leg (roughly a 0–1 cosine similarity forSEMANTIC) — a genuinely different scale, which is why the two modes' scores are never comparable to each other or toHYBRID.minTextRelevanceapplies only toHYBRID;minSimilarityapplies only toSEMANTIC/HYBRID. Both have a default floor (see the parameter table), so a result set can be smaller than the matching pool even when you set neither.- A miss is a
200, not a404. Emptyresults,totalResults: 0. - Degradation is silent unless you check. Inspect
degraded/degradedLegs, or setrequireComplete: trueto turn a degraded leg into a503. INDEXEDmeans indexing is complete for both search strategies; if a search still misses freshly written content, suspect your rate limit before indexing lag. The keyword index is immediately consistent — a document is matchable byTEXTthe moment it isINDEXED— and in practice the semantic (vector) index is normally immediately queryable too. A rate-limited request (yours or a retry the SDK made on your behalf) can silently take some time to actually run, which looks identical to "my data isn't there yet" — see the rate limits guide. A freshly indexed document that matches on text is designed to surface inHYBRIDandRAGat keyword speed (its semantic score fills in once the vectors catch up), but this fallback exists in code only foruniqueDocuments: false, and like/v1/search's default (uniqueDocuments: true, which RAG's grounding search also uses) it rests only on a unit test with mocked inputs, not a live-verified guarantee; treat it as best-effort on either path. A query that can match a document only semantically (pure-SEMANTIC, or a hybrid query sharing no words with it) has no fallback at all. See explanation.md § "Freshly indexed content".- Cross-content search returns mixed types. Always branch on
sourceTypewhen rendering unified results.
Inference surfaces — client.inference.*
Three streaming surfaces, all returning an async iterable of SSE events, all requiring the
inference:r scope, all sharing one pre-flight check sequence. By default, inference runs
against AWS-hosted models inside the Vectros perimeter (in-perimeter for the data plane). This
describes the platform-hosted model path. A call you route to your own model provider
(providerAlias, which needs a risk-acceptance waiver on your account) is processed where your
provider processes it and leaves the Vectros perimeter; residency settings do not apply to it.
Streaming model (shared)
Each surface returns an async iterable; iterate it to consume events in arrival order. Every
event carries an event field naming its type, so a consumer can dispatch on one key. On the
wire it is standard Server-Sent Events (event: <type> / data: <json> framed by a blank
line) — any compliant SSE reader works; the SDK presents it as an async iterator.
Comment lines — lines that start with a colon, such as : keepalive — may appear anywhere in a
stream, including before the first event and between events, and compliant readers ignore them. The
platform sends one as the stream opens, one about every 2 seconds for the first 10 seconds, and then one about every 15 seconds,
so a connection through a proxy or load balancer with an idle timeout is not cut while the model is still
working. A hand-written parser must skip any line that starts with a colon and must not treat a block with
no data: line as an event. The SDKs already do both. A stream can run for up to 5 minutes in total; a call that reaches that bound ends with an
error event and is not billed.
Set your HTTP client's read (idle) timeout to at least 20 seconds. The first call after a quiet period can take several seconds before its first event, and a reasoning model can think for longer than that between frames. A client that gives up after a few seconds of silence (some HTTP libraries default to 5 seconds) can fail on a call that would have succeeded.
Shared event vocabulary:
| Event | Fields | When |
|---|---|---|
reasoning_delta | delta (string) | One chunk of the model's reasoning. Sent only when the request sets reasoning: "stream" and the model reasons; always before the first content_delta. It is not part of the answer: do not append it to the answer text. |
content_delta | delta (string) | One chunk of generated text. Append each to build the answer. |
done | inputTokens, outputTokens, model, platformCreditsCharged, inferenceBalanceCentsChargedMilli, optionally cacheReadTokens / cacheCreateTokens | Terminal event with token counts, resolved model id, and per-call cost. platformCreditsCharged is the call's platform fee in credits (1 credit = 1 cent), charged to your monthly credit allowance: 0.2 for a call of up to 60 seconds, then 1 credit per 5 minutes of the call's duration, counted from the start of the request to the end of the model's response (a 120-second call is 0.4). inferenceBalanceCentsChargedMilli is the exact amount charged for the call's tokens, an integer in thousandths of a cent (1 cent = 1000). Exactly one. |
error | message, code | A failure after the stream has opened (for example code inference_error). A failure before the stream opens is a plain HTTP error response instead (see Error codes). |
Surface-specific events are listed under each surface below.
Pre-flight checks (shared, fixed order)
Run before any model invocation; cheaper checks first:
- Action scope →
403. A scoped token must carryinference:r(or a wildcard). Rootsk_*keys carry wildcard scope and pass by construction. The403does not enumerate the missing action. - Monthly credit limit →
402. Once the period's cumulative credits reach the account's monthly ceiling (the plan's, or a higher one agreed for the account), inference rejects until the period rolls or the ceiling is raised. - Burst rate limit →
429. Per-tenant request-rate ceiling, scaling with plan tier. - Inference billing gate →
402. In balance mode (default on lower tiers), a per-account pre-funded balance must be positive, else402 Insufficient inference balance. In usage mode (Enterprise-shaped, post-billed), accumulated usage is checked against a contractual cap, else402when the cap is reached.
When a call is routed to your own model provider with providerAlias, checks 2 and 4 (monthly credit limit and inference billing) are skipped for
that call; see Routing a call to your own model provider.
After these checks, once the prompt is assembled (on RAG, after the retrieval), a balance-mode call
reserves the most it could cost against the pre-paid balance, and the unused part is released when the
call finishes. If the reservation cannot be made the call is refused before the stream opens: 402 when
the balance is short; 429 with errorCode BALANCE_BUSY when the balance is momentarily busy with
other concurrent requests of yours; 503 with errorCode BALANCE_UNAVAILABLE when a temporary
service fault interrupted the step. The last two are not a shortage: do not top up, and retry after
the seconds in the Retry-After header.
The token cost of a call is metered and recorded on stream finalization (after the stream closes). A transient accounting failure does not fail the in-flight response — the balance on the next call may briefly lag.
Grounded corpus answers — client.inference.ragInference(req)
Retrieve-then-generate over your indexed content.
Request parameters:
| Parameter | Type | Default | Notes |
|---|---|---|---|
query | string | — (required) | The question to ground and answer. The request body, instructions and every other field included, is limited to 8 KB (a larger body is refused with 413). |
model | string | claude-haiku-5-5 | Model alias. See Model catalog. When providerAlias is set, names the model on that provider instead. |
maxTokens | integer | 1024 | Output cap. Capped at 4096 (tighter than chat — retrieved context shares the input budget). |
temperature | number | 0.3 | Sampling temperature. Ignored for claude-haiku-5-5, claude-sonnet-5, claude-sonnet-5-5 and claude-opus-5-5, which sample at their own default. |
instructions | string | — | Optional extra instructions for the answer. Counts toward the 8 KB request-body limit above — put large context in your indexed content and retrieve it with search instead. |
providerAlias | string | — | Route this call through your own model provider. See Routing a call to your own model provider. |
reasoning | 'omit' | 'stream' | omit | stream sends the model's reasoning as reasoning_delta events; see the reasoning notes under Notes & limits — inference. Any other value is rejected with 400. Ignored when providerAlias is set. |
search | object | — | Retrieval params (below). |
search sub-object mirrors content search: mode (default HYBRID), limit
(default 10, valid range 1–100; a value outside that range is rejected with 400, never
silently clamped — this is the RAG topK), userId, scope, scopeFilters,
folderId, rootFolderId, typeName, filters, contentTypes, createdAfter,
createdBefore, requireComplete. scope/scopeFilters behave exactly as documented in the
content-search parameter table above — scope for one ownership namespace, scopeFilters (an
array, mutually exclusive with scope) for more than one.
Event sequence: search_results → optional truncation_warning → optional reasoning_delta* (only
with reasoning: "stream") → content_delta+ → done.
| Event | Fields | Notes |
|---|---|---|
search_results | results[], totalResults, searchTimeMs, degraded, degradedLegs, hasMore | Always emitted, even when results is empty. hasMore is true when more matching results may exist than results returned — raise search.limit (maximum 100) or narrow your filters. Each entry: documentId, externalId, score, textScore, semanticScore, chunkText, contextText, snippet, metadata, sourceType, typeName, createdAt. These are your citations. |
truncation_warning | resultsRequested, resultsUsed, truncatedCount, noContentCount, reason | Emitted before the answer if any retrieved passage was dropped from grounding — either because it didn't fit the context budget (truncatedCount) or because it had no groundable text at all (noContentCount, rare in practice). reason is "context_window_budget", "no_groundable_content", or "context_window_budget_and_no_content" when both occurred; the two count fields let you attribute the drop precisely without parsing reason. |
Behavior:
- With
search.requireComplete: true, a degraded retrieval leg causes the call to reject before the stream opens (503) instead of grounding on partial results. - An empty retrieval still emits
search_results(empty) and still streams an answer (typically stating that nothing relevant was found). - A scoped token's data scope is enforced on the retrieval step.
Single-document Q&A — client.inference.documentAsk(req)
Ask one question against one document's full text. No retrieval step. Requires inference:r
(to run the model) and documents:r (to read the document) — a scoped token must hold
documents:r for the document (a grant qualified to a type, documents:r:<type>, covers only
documents of that type), within its data scope; inference:r alone returns 404.
Request parameters:
| Parameter | Type | Default | Notes |
|---|---|---|---|
id | string (uuid) | — (required) | The document to ask against: the {id} in the path POST /v1/documents/{id}/ask (the SDKs take it as a request field). |
prompt | string | — (required) | The question. |
instructions | string | built-in document-analyst prompt | Optional system instructions for the answer. When omitted or blank, a default prompt asks the model to answer only from the document text and to say so when the document does not contain the answer. |
model | string | claude-haiku-5-5 | Model alias. When providerAlias is set, names the model on that provider instead. |
maxTokens | integer | 2048 | Output cap. Capped at 8192. |
temperature | number | 0.2 | Sampling temperature. Ignored for claude-haiku-5-5, claude-sonnet-5, claude-sonnet-5-5 and claude-opus-5-5, which sample at their own default. |
providerAlias | string | — | Route this call through your own model provider. See Routing a call to your own model provider. |
reasoning | 'omit' | 'stream' | omit | stream sends the model's reasoning as reasoning_delta events; see the reasoning notes under Notes & limits — inference. Any other value is rejected with 400. Ignored when providerAlias is set. |
Event sequence: document_context → optional reasoning_delta* (only with reasoning: "stream") →
content_delta+ → done.
| Event | Fields | Notes |
|---|---|---|
document_context | documentId, title, textBytes, model | The document loaded, its size, and the resolved model. Fires before any generated text. |
Errors:
409(before the stream opens) — the document is not askable yet: it is still processing (not yet fully indexed), it failed ingest, no text could be extracted from its file (itsextraction.outcomesays why), or its text is not retained (a file uploaded withstoreText: false— the extracted text is discarded after indexing, so there is no full text to load). Freshly-ingested documents commonly return409until indexing completes.413(before the stream opens, no credits charged) — the document's estimated input size exceeds the cap (32,000 input tokens, ~25 pages). Payload:message,estimatedTokens,limitTokens. Branch on this and re-route to RAG.404— the document does not exist, belongs to another tenant, is out of your token's data scope, or your token lacksdocuments:rfor it. All return the identical404; the endpoint never reveals existence outside your read access.
Stateless chat — client.inference.chatInference(req)
Single-turn completion. No retrieval, no stored state.
Request parameters:
| Parameter | Type | Default | Notes |
|---|---|---|---|
messages | { role, content }[] | — (required) | Conversation. A system role message becomes the system prompt; user / assistant messages pass through. A conversation that ends with an assistant message (a prefill) is refused with 400 on claude-haiku-5-5, claude-sonnet-5, claude-sonnet-5-5 and claude-opus-5-5: end with a user message, or name claude-haiku-4-5 in model to prefill the reply. |
model | string | claude-haiku-5-5 | Model alias. When providerAlias is set, names the model on that provider instead. |
maxTokens | integer | 2048 | Output cap. Capped at 8192. |
temperature | number | 0.7 | Sampling temperature. Ignored for claude-haiku-5-5, claude-sonnet-5, claude-sonnet-5-5 and claude-opus-5-5, which sample at their own default. |
topP | number | — | Nucleus-sampling parameter (optional). Used only when temperature is 0; ignored for the same models as temperature. |
providerAlias | string | — | Route this call through your own model provider. See Routing a call to your own model provider. |
reasoning | 'omit' | 'stream' | omit | stream sends the model's reasoning as reasoning_delta events; see the reasoning notes under Notes & limits — inference. Any other value is rejected with 400. Ignored when providerAlias is set. |
Event sequence: optional reasoning_delta* (only with reasoning: "stream") → content_delta+ → done.
Chat stores nothing. For multi-turn, append the assistant's reply to your messages array
and re-send the whole array next turn.
Model catalog — client.inference.listInferenceModels()
Lists every model in the catalog; availableOn names the plans that can call each one.
Response:
| Field | Type | Notes |
|---|---|---|
models | Model[] | Every model in the catalog. availableOn names the plans that can call it; a call to a model outside your plan returns 402. |
defaultModel | string | The alias used when a call omits model: claude-haiku-5-5 (claude-haiku-4-5 stays in the catalog and is still served to a call that names it). Available on every plan; on a balance-billed account, calling it still needs a funded inference balance. |
Each Model:
| Field | Type | Notes |
|---|---|---|
id | string | Alias: claude-haiku-4-5, claude-haiku-5-5, claude-sonnet-5, claude-sonnet-5-5, claude-opus-5-5, amazon-nova-lite, amazon-nova-2-lite, meta-llama-4-scout, meta-llama-4-maverick. Which of these your key can reach depends on its plan. |
name | string | Display name. |
provider | string | Model provider. |
contextWindow | integer | Context window size in tokens. |
inputCreditsPer1kTokens | number | Base (non-US) input rate — equal to regionPricing.base.input. Default-served (US) requests are billed regionPricing.us.input instead; see below. |
outputCreditsPer1kTokens | number | Base (non-US) output rate — equal to regionPricing.base.output. Default-served (US) requests are billed regionPricing.us.output instead; see below. |
regionPricing | object | Per-region rate breakdown for this model — base, us, regionPremiumFactor, globalAvailable. See "Region pricing" below. |
availableOn | string[] | Plan tiers that may call this model (e.g. free, starter, pro, scale, enterprise). |
longContext | object | Optional; present only for a model priced by prompt length (claude-haiku-5-5). promptTokensOver (100000), multiplier (5.0), and base / us credit rates per 1K input and output tokens. See "Prompt-length pricing" below. |
cachePricing | object | Optional; present only for a model with prompt-cache prices (the five Claude models). base and us, each with read, write5m and write1h: credits per 1K input tokens read from the prompt cache, written to it with a 5-minute lifetime, and written with a 1-hour lifetime. A model without the field has no cache prices. See "Prompt-cache pricing" below. |
Requesting a model your plan does not include returns a 402 pointing to upgrade; an alias
that is not in the catalog returns 400. A lighter model is available on every tier; more capable
models require higher tiers.
Prompt-length pricing
A model with a longContext object is priced by the length of the PROMPT of each request. A request whose prompt is up to
longContext.promptTokensOver tokens is charged the standard rates of the model (inputCreditsPer1kTokens,
outputCreditsPer1kTokens, regionPricing). A request whose prompt is LONGER is charged longContext.multiplier times those rates
on every token of that request, input and output (longContext.base and longContext.us carry the resulting rates). A prompt of
exactly promptTokensOver tokens is not over the line. The prompt's length is the one the model provider reports when the call
completes, so a request near the line is charged at
the rate its real size falls in then; the balance hold taken before the call assumes the higher rate when the prompt looks near or
over the line, and the unused part is refunded (a balance-mode account whose balance covers the real cost but not the held amount can
be refused with 402).
Token cost is charged in milli-cents (thousandths of a cent) and is not rounded to whole cents: every call that uses tokens draws its exact cost from your inference balance, rounded to the nearest thousandth of a cent, in addition to the per-call platform fee.
Prompt-cache pricing
A model with a cachePricing object charges the input tokens its provider reports as read from, or written to, its prompt cache
at the cache rates instead of the input rate (cacheReadTokens and cacheCreateTokens on the done event). The rates are the
input rate times a weight: a cache read is 0.10 of the input rate (0.05 on claude-sonnet-5-5 and claude-opus-5-5), a write with
a 5-minute lifetime is 1.25, and a write with a 1-hour lifetime is 2.0. cachePricing.base is built on the base input rate and
cachePricing.us on the US-served input rate, so us is base times regionPricing.regionPremiumFactor. Credits per 1K input
tokens, base rates:
| Model | Input | Cache read | Cache write, 5 minutes | Cache write, 1 hour |
|---|---|---|---|---|
claude-haiku-4-5 | 0.1 | 0.01 | 0.125 | 0.2 |
claude-haiku-5-5 | 0.01 | 0.001 | 0.0125 | 0.02 |
claude-sonnet-5 | 0.2 | 0.02 | 0.25 | 0.4 |
claude-sonnet-5-5 | 0.2 | 0.01 | 0.25 | 0.4 |
claude-opus-5-5 | 0.4 | 0.02 | 0.5 | 0.8 |
amazon-nova-lite, amazon-nova-2-lite, meta-llama-4-scout and meta-llama-4-maverick have no cache prices: their cachePricing
is absent, and a cache count on one is not charged on its own. Read the exact figures from GET /v1/models.
For claude-haiku-5-5, the prompt-length multiplier applies to the cache rates too, and cache read and cache write tokens count
toward the length of the prompt. On a prepaid balance, the amount held before a call on a model with cache prices covers the input at the
5-minute cache-write rate (1.25 times the input rate) and the unused part is released when the call ends, so a call that touches no
cache is charged what it would be without cache prices; a balance close to the held amount can be refused with 402. Agent runs do not use
the cache rates: a run charges any cache token at the input rate. A call routed with providerAlias is not charged for tokens.
Region serving (allowGlobalRegion)
All three inference surfaces (chat, RAG, document-ask) accept an optional boolean
allowGlobalRegion in the request body.
| Field | Type | Default | Meaning |
|---|---|---|---|
allowGlobalRegion | boolean | tenant residency default | Opt this request into the lower-cost global (non-US) region path. |
- The tenant's residency default is US serving, applied when the flag is omitted. US serving is the fail-closed default and carries a region premium.
- Setting
allowGlobalRegion: truelets an entitled tenant serve the request from the global region at a lower rate. Entitlement is gated on a signed global-processing waiver (dev portal → Settings → Data Residency & Region; see the operations & trust reference for the residency posture this entitlement governs). - If
allowGlobalRegion: trueis sent by a tenant that is not entitled, the request is rejected with403— it is not silently downgraded or upgraded. Region choice never changes which content is retrieved, only where the model runs and the price. - If Vectros requires a newer version of the global-processing waiver, an earlier signature no longer
covers it once the date announced to account owners has passed: the tenant returns to US-only
serving, and
allowGlobalRegion: trueis rejected with403, until an account owner signs the new version. allowGlobalRegionhas no effect on a call routed withproviderAlias; see Routing a call to your own model provider.
Region pricing (regionPricing)
Every model returned by the model catalog
carries a regionPricing object so you can compute the exact billed rate for either region
without a client-side multiply:
| Field | Type | Notes |
|---|---|---|
regionPricing.base | { input, output } | Per-1K credit rates billed when a request is served from the global (non-US) region — i.e. allowGlobalRegion: true on an entitled tenant. Equal to the model's inputCreditsPer1kTokens / outputCreditsPer1kTokens. |
regionPricing.us | { input, output } | Per-1K credit rates billed when a request is served from the US region — the default. Equal to base multiplied by regionPremiumFactor. |
regionPricing.regionPremiumFactor | number | Multiplier applied to base to get us. 1.0 means no premium (base == us); a Bedrock cross-region model typically carries a premium above 1.0 (e.g. 1.1). |
regionPricing.globalAvailable | boolean | Whether this model can be served from the global region at all. Every model in the catalog currently reports true. |
A default-served (no allowGlobalRegion, or an unentitled tenant) request is always billed
regionPricing.us; an entitled tenant opting in with allowGlobalRegion: true is billed
regionPricing.base. Use this object to show accurate per-region pricing in a model picker
before the caller decides whether to opt into global serving.
Routing a call to your own model provider (providerAlias)
All three inference surfaces (chat, RAG, document-ask) accept an optional providerAlias string in
the request body. It names a provider config that the account owner has set up (a
customer-supplied Anthropic key, or an OpenAI-compatible endpoint such as a self-hosted vLLM or TGI
cluster) and routes that one call through it instead of the platform-hosted models.
- Alias. A provider alias is 1 to 64 lowercase letters, digits,
-or_, unique within the tenant. A call must send it exactly as stored: the match is case-sensitive. The rule applies when a config is created and when its alias is changed; a config that already holds an alias outside the rule keeps working with it. - Endpoint and other fields. A config you set up yourself takes a plain
httpsendpoint on port 443 (no credentials, query or fragment; the platform adds the provider type's standard API path, so leave it out), thebyo-model-inference:customer-keywaiver type, and theCUSTOMER_DIRECTbilling mode, which is the default when omitted. The provider key is printable ASCII of at most 4096 characters, and a tenant can hold at most 25 configs. A request outside these rules is rejected with400; a value a config already holds is accepted. A save that loses a race with another change answers409. - Provider type. A request that changes the
providerTypeof an existing config, in either direction, must state theendpointtoo (an empty string means the provider's default), or it is rejected with400: a stored endpoint overrides the Anthropic default at call time, so a type change that left it unsaid would keep sending calls to the old host. A stated endpoint on a different host, which includes""on a config that has one, needs the provider key again. - Model. When
providerAliasis set,modelnames the model on that provider's own id space (an opaque string to Vectros, not a catalog alias) and falls back to the provider config's default model when omitted. A request that sets neither is rejected with400. The plan gate on the model catalog does not apply to a provider-side model id. - Entitlement. Your account must hold the matching risk-acceptance waiver. An unknown or inactive
alias, or a missing waiver, is rejected with
403. An account owner signs the waiver for an environment in the developer portal (Model Providers). If Vectros requires a newer version of the waiver, an earlier signature stops counting once the date announced to account owners has passed, until an owner signs the current version. - Business Associate Agreement attestation. If your account holds a Business Associate Agreement
with Vectros (one accepted in the developer portal; an agreement made outside it does not make the
account a holder here), a provider config is usable only while an owner has attested, for that config, that
the account holds a business associate agreement, or another agreement that HIPAA requires, with
the provider at the config's endpoint, covering the information the account will send to it, and
that the account is permitted to direct that disclosure. The owner gives the attestation per
config, in the developer portal (Model Providers) or the developer API, and the attestation is
kept as a signed record. Through the API the request states the provider type and endpoint being
attested (an empty endpoint means the provider's default); a request that omits them is refused
with
400, and one that no longer matches the config is refused with409and records nothing. It applies to the test and live environments alike, and to a config set up for you by Vectros. It belongs to the config's provider type and endpoint, exactly as saved: any change to either, including a change of letter case or a trailing slash, withdraws it in the same change, and calls through the config are then rejected with403until an owner attests again (normally within moments of the change). If Vectros cannot establish whether the account holds the agreement, it treats the account as a holder. An account that does not hold the agreement is not asked for an attestation. - Region.
allowGlobalRegionand your residency settings do not apply to a routed call; your provider decides where it processes the call. - Perimeter. A routed call is processed where your provider processes it and leaves the Vectros
perimeter. The in-perimeter description elsewhere in this guide is for the platform-hosted model
path. For a routed RAG call, the retrieval still runs inside the perimeter (a semantic or hybrid
search embeds your query with a Vectros-hosted model); the query and the retrieved passages are
then sent to your provider. Vectros delivers to the endpoint saved on the config (for an Anthropic
config with no endpoint saved, Anthropic's published API host), with the provider
type's standard API path added (for example
/v1/messages; save the endpoint without it, and a trailing slash is trimmed), only overhttpswith the server certificate validated, and without following redirects. - Billing. The provider bills you directly; Vectros does not price a routed call's tokens. The call
skips the monthly credit-limit and inference-balance checks and does not draw the inference balance;
the per-call platform fee still applies, and the
doneevent'sinferenceBalanceCentsChargedMilliis0. - Management. The account owner manages provider configs in the developer portal (Model
Providers), the developer API and the CLI's
vectros provider-configcommands. A call withoutproviderAliasis served by the platform-hosted models.
Error codes (inference)
| Code | Surface | Meaning |
|---|---|---|
400 | all | Malformed request (e.g. missing query / messages / prompt, bad filter key, an alias outside the model catalog, or a providerAlias call with no model and no default model on the provider config). RAG also returns 400 for a search.limit outside 1–100. POST /v1/chat also answers 400 with errorCode: CONTEXT_WINDOW_EXCEEDED when the conversation plus maxTokens cannot fit the model's context window. It also answers 400 when the last message has role assistant on a model that does not accept a prefill (see messages). Any surface answers 400 when the body is not valid JSON (Malformed JSON body) and when reasoning is anything other than omit or stream. |
402 | all | Monthly credit limit exceeded, usage cap reached, or a requested model the plan does not include (errorCode SUBSCRIPTION_LIMIT_EXCEEDED); insufficient inference balance, or a balance that cannot cover the call's reservation (errorCode INSUFFICIENT_BALANCE). A funded balance that is momentarily busy or interrupted is 429 / 503 below, never 402. |
403 | all | Token scope does not permit inference (inference:r missing); or allowGlobalRegion: true was sent but this tenant is not entitled to global-region serving (no signed global-processing waiver); or providerAlias names an alias that is unknown or inactive for your account, or your account lacks the risk-acceptance waiver for it, in which case errorCode says which: WAIVER_NOT_SIGNED (the waiver has not been signed; an account owner signs it in the developer portal, Model Providers), WAIVER_TEXT_CHANGED (the version of the waiver in force for your account is older than the version Vectros requires; an owner must review and sign the current version; editing a waiver's text refuses nothing), WAIVER_REVOKED (an owner revoked it; an owner can sign it again); or, on an account that has accepted the Business Associate Agreement with Vectros in the developer portal, the alias's provider config has no current attestation; or the alias's provider config holds a retired billing mode (move it to CUSTOMER_DIRECT). |
404 | all | App context not found. when the app context of the call does not exist or is being deleted. On document-ask also: document not found / cross-tenant / out-of-scope (uniform — existence never revealed). |
409 | document-ask | Document not askable yet — still processing (not yet indexed), failed ingest, or text not retained (a file uploaded with storeText: false). Returned before the stream opens. |
413 | all | The request body is too large: over 1 MiB (1,048,576 bytes) on chat and document-ask, or over 8 KB on RAG; the value is PAYLOAD_TOO_LARGE in the field code (answered by the edge) or errorCode (answered by the platform). Document-ask also: the document exceeds the input-token cap. Before the stream opens; no credits charged. |
429 | all | Burst rate limit exceeded; or, with errorCode BALANCE_BUSY, your pre-paid inference balance is momentarily busy with other concurrent requests of yours. That is not a shortage (that is 402) and not a rate-limit breach: do not top up, retry after the seconds in the Retry-After header. |
500 | RAG | The retrieval step failed (Search failed) before the stream opened. |
503 | RAG, search | A retrieval/search leg was unavailable and requireComplete / requireComplete: true was set. On RAG this 503 carries no errorCode. |
503 | all inference calls (errorCode BALANCE_UNAVAILABLE) | A temporary service fault interrupted the pre-paid balance step of this call. It is not a shortage (that is 402): do not top up. Retry after the seconds in the Retry-After header. On RAG, branch on errorCode to tell it from the requireComplete case. |
503 | all inference calls, with providerAlias | The stored key of the provider config cannot be opened (the platform's encryption service is unavailable). A provider key you enter from release 0.48.0 on is stored encrypted; a key stored before that is read as it is and cannot produce this 503. Nothing is sent to the provider and the reserved balance is released. The response carries no errorCode and no Retry-After: retry shortly, and if it persists enter the key again in the developer portal. |
Notes & limits — inference
- Hard output caps per surface: chat 8192, RAG 4096, document-ask 8192 output tokens.
Document-ask additionally caps input at 32,000 tokens (~25 pages) with a
413before the stream opens. AmaxTokensabove a surface's cap is floored to the cap. claude-haiku-5-5,claude-sonnet-5,claude-sonnet-5-5andclaude-opus-5-5can reason before they answer, and when they do,maxTokensincludes the reasoning. The reasoning counts againstmaxTokenstogether with the answer and is billed as output whether or not you receive it (onclaude-opus-5-5, the one measured, the usage figure counts both and the split is not reported), so a smallmaxTokens, or a request that makes the model reason at length, can return little or no text: the stream ends withdoneand an output-token count at or nearmaxTokens. Leave headroom (amaxTokensof 1024 or more for a short answer).reasoning: "stream"delivers the reasoning as its own events; the default does not. Withreasoningomitted or"omit", the stream carries only the answer, exactly as before. With"stream", a model that reasons sendsreasoning_deltaevents before the firstcontent_delta(on RAG aftersearch_resultsand anytruncation_warning; on document-ask afterdocument_context), and a model that does not reason sends none.doneis unchanged, and the token counts and charge are the same as without the option. Vectros does not log or store the reasoning. It can quote your documents or messages, so treat it as you treat the answer. The field applies to platform-hosted models: a call routed withproviderAliasignores it. A value other thanomitorstreamis rejected with400before anything is charged.- Request body: at most 1 MiB (1,048,576 bytes) on chat and document-ask; at most 8 KB on RAG. A larger chat or
document-ask body is refused with
413andPAYLOAD_TOO_LARGEbefore anything is charged (the edge firewall answers it in the fieldcode, the platform inerrorCode; read either); shorten the conversation or send fewer or shorter messages. 1 MiB is roughly 250,000 to 400,000 tokens (it depends on the text and the model), which covers the 100,000-token prompt-length price tier (see "Prompt-length pricing"). A prompt beyond 1 MiB is not supported (a body so large that the API gateway itself refuses it, about 6 MB, is answered by the gateway, not by this413). A RAG body over 8 KB is refused with413by the edge, so a large RAG context belongs in indexed content. The413the edge returns names the fieldcode; the one from the platform names iterrorCode; both carry the valuePAYLOAD_TOO_LARGE. A prompt must also fit the model'scontextWindow(see the model catalog for each model:claude-haiku-4-5has 200,000 tokens, the default modelclaude-haiku-5-51,000,000).POST /v1/chatrefuses a conversation whose estimated size plusmaxTokensexceeds the window with400anderrorCode: CONTEXT_WINDOW_EXCEEDED, before anything is charged (a call routed to your own provider withproviderAliasis not checked). The estimate is approximate: a conversation within a few percent of the window can be refused although it would have fit, or accepted and still rejected by the model provider, in which case the call ends with an error event and nothing is charged. - RAG topK is bounded at 100.
search.limitdefaults to 10 and accepts 1–100; a value outside that range is rejected with400rather than clamped. WhenhasMoreon thesearch_resultsevent istrue, more matching results may exist than were retrieved. Retrieved context that does not fit the model's input budget is dropped from grounding and reported in atruncation_warningevent. queryandinstructionsare for a question and prompt-shaping text. The request body is limited to 8 KB (413),queryandinstructionsincluded. Put large context in your indexed content and retrieve it with search.- Chat is stateless; there is no managed conversation state. No server-side thread
store, no assistants registry. Multi-turn is the caller's responsibility — re-send the
messagesarray each turn and budget the history against the model's context window. - Document-ask is single-document. No multi-document Q&A endpoint. For multi-document
grounding, use RAG (retrieval picks relevant passages across the corpus) or stitch
multiple
/askcalls at the application layer. - Cost is recorded on finalization. The
doneevent carriesplatformCreditsChargedandinferenceBalanceCentsChargedMilli; an accounting hiccup will not fail an in-flight response, so the next call's balance may briefly lag. - Disconnecting does not stop the model call. A call is billed on the provider's reported usage. The platform does not currently stop the model call when the client disconnects, so a call you abandon can still run to completion and be billed.
- Cache tokens are charged at the model's cache rates.
donemay carrycacheReadTokens/cacheCreateTokens. On a model that listscachePricinginGET /v1/modelsthey are charged at its cache read and write rates and included ininferenceBalanceCentsChargedMilli; on a model without it they are not charged on their own. These endpoints send no cache checkpoint, so a non-zero count comes only from a provider that caches on its own. A call routed throughproviderAliasis not charged. - In-perimeter scope. The in-perimeter description (no third-party model-vendor egress)
is for the data plane on the platform-hosted model path — the content you store and
retrieve through Vectros. A call you route to your own model provider (
providerAlias, which needs a risk-acceptance waiver on your account) is processed where your provider processes it and leaves the Vectros perimeter. Specific compliance coverage terms are addressed in the security and compliance documentation, not asserted here. - The model catalog is the source of truth. Handlers gate on the live catalog at request
time, so a model going generally available or being retired takes effect immediately; the plans in
availableOnare enforced when you call the model.
Where to go next
- how-to.md — runnable guides for each call on this page.
- explanation.md — the concepts behind the modes, grounding context, and the three inference surfaces.
- ../data-model/reference.md — schema field declarations (searchable, filterable, sensitive) that govern what search indexes.
- ../operations-trust/compliance.md — sensitive-field protections, isolation, and the in-perimeter inference posture.