Search & RAG — reference

Exhaustive reference for content search and the three streaming inference surfaces: every parameter, field, mode, limit, error code, the stream event vocabulary, and a "Notes & limits" section stating what each feature does not do.

For request/response wire detail at the endpoint level, see the generated API reference (the OpenAPI / Scalar spec). This page documents the SDK-level surface and the behavior that the spec alone does not capture.

Version note. The API spec is at 0.50.0. Nothing on the search side of this page is 0.26-only, so it works on any current client. The optional inference flag allowGlobalRegion (region serving, below) is a 0.27 addition, and the optional reasoning field and the reasoning_delta event are a 0.48 addition. The milli-cent amounts (inferenceBalanceCentsChargedMilli, and balanceCentsMilli in the usage report), cachePricing and the cache counts need a 0.49 client. Where the SDK method name differs from a raw field name, the SDK name is given.


Content search — client.search.content(req)

Unified search across documents and records. Requires the search:r scope to call. Results are additionally enforced per row against what the calling token could read directly: a scoped token sees only results it holds the read grant for — documents:r for documents, records:r for records, each optionally narrowed to a single type with a qualifier (records:r:<type>, documents:r:<type>) — and only within its ownership data scope. A document's type is the type of the schema it is bound to; an untyped document matches only an unqualified documents:r grant. A token with search:r but no read grant returns an empty result set. The same per-row enforcement applies to RAG retrieval (/v1/rag).

Request parameters

ParameterTypeDefaultNotes
querystring— (required unless filters holds a $near)Natural-language or keyword query. Optional when filters holds a $near: with no query, a $near returns the matches within its radius, nearest first (Distance search below).
mode'TEXT' | 'SEMANTIC' | 'HYBRID'HYBRIDKeyword relevance / meaning-based similarity / fused ranking.
limitinteger20Max results. Valid range 1–100; out-of-range is rejected with 400.
offsetinteger0Results to skip — paging (see Paging below). Valid range 0 to 200; out-of-range is rejected with 400.
contentTypes('documents' | 'records')[]bothNarrow to one content type. Omitted, empty, or both values ⇒ unified.
typeNamestring—Restrict hits to one schema type — for example patient or runbook. Applies to both documents and records: any item whose bound schema type matches. On its own it narrows both content types; combine it with contentTypes to narrow within one (e.g. contentTypes: ['documents'] + typeName: 'runbook' ⇒ runbook documents). Untyped content (no schema) never matches. Documents ingested before this facet shipped must be reindexed to be matchable — see the note below.
folderIdstring (uuid)—Restrict to content in this exact folder.
rootFolderIdstring (uuid)—Restrict to this folder and all descendants. Use instead of folderId.
userIdstring (uuid)—Ownership filter — content owned by this user.
scopestring—Ownership filter — content owned by this entity, as <namespace>:<value> (org:..., client:..., or a namespace you registered). One entry per query; pair it with userId for a second dimension, or use scopeFilters (below) for more than one scope namespace at once. Mutually exclusive with scopeFilters.
scopeFiltersstring[]—Ownership filter for MORE THAN ONE scope namespace in a single query — e.g. ["org:<id>", "client:<id>"] to narrow to one specific client within one specific org. Each entry uses the same <namespace>:<value> form as scope. Naming the same namespace twice is rejected (400), as is supplying both scope and scopeFilters. Capped at 16 entries.
filtersobject—Field-level metadata filters; AND-combined across keys. See Filter grammar.
createdAfterstring (ISO-8601)—Content created at/after this UTC timestamp.
createdBeforestring (ISO-8601)—Content created at/before this UTC timestamp.
uniqueDocumentsbooleantrueWhen true (the default), at most one hit per source document; set false for chunk-level results, where several chunks of one document can each be a hit.
minSimilaritynumber 0.0–1.00.15Minimum semantic similarity; hits below are excluded (semantic / hybrid). In HYBRID the floor drops only a hit that matched on meaning alone: with uniqueDocuments: true, a hit the keyword leg also matched is kept. Send 0 to turn the floor off.
minTextRelevancenumber 0.0–1.00.3Relative keyword-relevance floor, as a fraction of the top hit's score (e.g. 0.5 keeps hits at least half as relevant as the best). Applies to HYBRID only (TEXT mode does not apply it). Send 0 to keep all keyword matches.
textMode'OR' | 'AND' | 'PHRASE' | 'COMPLEX'PHRASE in HYBRID, OR in TEXTKeyword sub-mode (see Keyword sub-modes). Applies to TEXT / HYBRID.
slopinteger ≥03 when textMode is omitted in HYBRID; otherwise 0Phrase-match slop: intervening positions tolerated between terms when textMode='PHRASE'. Ignored otherwise.
requireCompletebooleanfalseFail-closed override: return 503 instead of degraded partial results when a leg is unavailable.
sortBy'distance'— (relevance order)Nearest first. Requires a $near filter, and either no query or mode: 'TEXT'; rejected with 400 for HYBRID and SEMANTIC. See Distance search.

Scoping documents by type — reindex note. The document side of the typeName facet relies on an index key that is written when a document is indexed. Records have always carried it; documents ingested before this facet shipped do not, so they will not match a typeName filter until they are reindexed. New and updated documents pick it up automatically. To make older documents matchable, re-index them (re-ingest, or touch them so they re-index). Records are unaffected.

Keyword sub-modes (textMode)

For TEXT and HYBRID searches, textMode controls how query terms combine in the keyword leg:

  • OR (default in TEXT) — match any term; broadest recall.
  • AND — require all terms; higher precision.
  • PHRASE (default in HYBRID, with a slop of 3) — require terms as a contiguous sequence (tunable with slop).
  • COMPLEX — full keyword query syntax (boolean operators, field-scoped clauses, range filters). Use only when you need expression-level control; the query is parsed as a structured expression rather than a bag of terms.

Filter grammar (filters)

Each top-level key is a metadata field declared filterable on the schema (or a built-in document field). Top-level keys are AND-combined. Each value is one of:

  • Scalar (string / number / boolean) — equality, e.g. { "status": "open" }.
  • Array of scalars — OR-set (match any), e.g. { "tag": ["red", "blue"] }.
  • Operator map — closed set of operators:
    • Scalar operand: $eq, $ne, $gt, $gte, $lt, $lte. Operators in one map are AND-combined, so { "price": { "$gte": 100, "$lte": 500 } } is a closed range.
    • Array operand: $in, $nin. Cannot be combined with other operators in the same map.

Numbers and booleans match typed (the field must have been ingested under a typed schema). A numeric filter operand is held to the same signed 64-bit range as stored values — an out-of-range number in the request body is rejected with 400; match a large id you stored as a string with string equality, not a numeric operator. Dates may be ISO-8601 strings or epoch millis. Filter keys are validated (^[A-Za-z_] [A-Za-z0-9_-]*$); unknown operators, non-scalar operands, malformed keys, and any attempt to filter on a reserved tenancy/ownership key are rejected with 400. You cannot widen your access through the filter map — ownership scope is enforced separately.

Distance search ($near)

A field of type geo (a limited pilot: geo fields are refused unless the pilot is enabled for your account) takes one more operator, $near:

{ "filters": { "location": { "$near": { "lat": 47.6062, "lon": -122.3321, "radiusMeters": 5000 } } } }
  • It keeps only content whose location is within radiusMeters of the point. Distance is great-circle distance on a sphere (mean Earth radius 6,371,008.8 m), inclusive. It is a spherical model, so it differs slightly from a distance measured on the WGS84 ellipsoid.
  • lat must be in [-90, 90], lon in [-180, 180], radiusMeters in [1, 2,000,000]; all three are required numbers. $near must be the only operator in its map, and at most one $near is allowed per request. Anything else is rejected with 400.
  • Every result then carries distanceMeters (rounded to 0.1 m).
  • Three ways to use it:
    • No query: the matches within the radius, nearest first (equal distances in a stable order). It runs on the keyword index whatever mode says, and is billed as a TEXT search.
    • mode: 'TEXT' with sortBy: 'distance': the keyword matches within the radius, nearest first.
    • A search ordered by distance (either of the two above) returns at most one result per item; uniqueDocuments does not apply.
    • A query with no sortBy, any mode: results keep their relevance order, filtered to the radius.
    • A TEXT search with a $near (with or without sortBy) carries no keyword-highlighted snippet; it falls back to an excerpt.
    • Content indexed with indexMode: SEMANTIC is in the semantic index only, so a search with no query, or a TEXT search, does not find it; a HYBRID or SEMANTIC search with a query does.
  • A search ordered by distance is exact within fixed per-request limits. When a very dense area exceeds them, the response sets incompleteResults: true: the results are still in exact distance order, but nearer matches may exist that were not returned. Narrow the radius or add filters. In such a search totalResults is a lower bound.
  • Content whose stored location could not be read is indexed without it and is not found by $near; its indexFailure.code is LOCATION_NOT_INDEXED (see the operations-trust reference).
  • Without typeName, a $near on a field that is not of type geo matches nothing: the result is empty, not an error.
  • With typeName, a $near on a field that no schema of that type declares as geo is a 400. This check does not apply to a key that reaches every app context: that key gets an empty result.
  • requireComplete: true also returns a 503 instead of a search ordered by distance that would carry incompleteResults: true.
  • $near is accepted by content search only. RAG inference rejects it with 400.

Response shape

search.content returns a flat object (it is not wrapped in the { data, nextCursor } cursor envelope used by list/lookup endpoints — see Paging):

FieldTypeNotes
resultsSearchResult[]Matched chunks, ranked (highest score first). May be empty.
totalResultsintegerApproximate matching-pool size. 0 for a miss. Not a reliable paging signal — use hasMore instead.
hasMorebooleanTrue when more matching results exist past this page; false once you've reached the last page or the offset maximum (200).
searchTimeMsintegerServer-side execution time. Reported even on an empty result.
degradedbooleanTrue when one leg was unavailable and results came from the survivor only.
degradedLegsstring[]Which legs were unavailable: "text" (keyword) and/or "vector" (semantic). Empty when not degraded.
incompleteResultsbooleanPresent and true only on a search ordered by distance that reached its per-request limits before it could confirm the page (see Distance search). Absent otherwise.

Each SearchResult:

FieldTypeNotes
documentIdstring (uuid)Source entity id. Use with getDocument / getRecord. (This is the source id, not an internal index id.)
externalIdstring | nullThe matched item's externalId as you supplied it, alongside documentId — lets you correlate a hit back to your own record identity without a follow-up lookup. Null when the item was ingested without one, or was indexed before this field existed and hasn't been reindexed since.
sourceType'PartnerDocument' | 'GenericRecord'Document vs. record discriminator — the two literal strings the API returns; branch on it when rendering mixed results.
scorenumberPrimary sort key, higher is more relevant. Its scale depends on mode — see Notes below; a HYBRID score is not comparable to a TEXT- or SEMANTIC-only score.
textScorenumberKeyword (relevance) sub-score. Non-zero in HYBRID when the keyword leg contributed. In TEXT-only mode it's rank-derived (see Notes below) — meaningful for ordering within one response, not a raw BM25 magnitude.
semanticScorenumberSemantic similarity sub-score. Non-zero in SEMANTIC / HYBRID.
chunkTextstringThe specific chunk that matched. Feed this (or contextText) to a model.
contextTextstringThe wider surrounding passage containing the chunk — better grounding context.
snippetstringHighlighted excerpt with query terms emphasized, for display. May be null for a semantic-only hit (use chunkText).
metadataobjectMetadata supplied at ingest (title, folderId, custom fields).
createdAtstring (ISO-8601)Source content creation time.
distanceMetersnumberGreat-circle distance in meters (rounded to 0.1 m) from the $near point. Present only when the request has a $near.

Paging

search.content has no nextCursor — it is not enveloped. Page by combining limit with offset:

const page1 = await client.search.content({ query, mode, limit: 20, offset: 0 });
const page2 = await client.search.content({ query, mode, limit: 20, offset: 20 });

limit is capped at 100; consecutive pages are disjoint. This differs from list/lookup endpoints, which return { data, nextCursor } and are drained by feeding nextCursor back as startFrom. For pulling recent content deterministically, prefer a createdAfter window over deep offset paging.

  • Index/search mode must line up. Content indexed for one strategy only will not appear in a search requiring the other. Indexing mode is a property of the content (set at ingest / on the schema); search mode is a property of the query.
  • Only searchable fields participate in keyword relevance. A query matching only a non-searchable field returns nothing — by design.
  • Sensitive fields are excluded from the index. Their values are not written to the search index, so search does not match, rank, or return them (index-time exclusion).
  • textScore in TEXT-only mode is rank-derived, not a raw BM25 magnitude. That path returns highlighted snippets and already-correctly-ordered hits but never exposes the raw per-hit keyword score, so textScore there reflects the hit's relative rank — meaningful for ordering within one response, not comparable across requests or against HYBRID/SEMANTIC.
  • HYBRID's score is a Reciprocal Rank Fusion (RRF) value, not a 0–1 confidence — its magnitude looks nothing like TEXT/SEMANTIC's. RRF combines the keyword and semantic legs by RANK POSITION only (never by raw score), as 1 / (k + rank) per leg, summed across legs it appears in, with the standard k = 60. That makes every HYBRID score small and tightly clustered near the same constant: a hit ranked top on a single leg scores 1/61 ≈ 0.0164; ranked top on both legs, 2/61 ≈ 0.033 — near the practical ceiling. This is expected, not a sign every result is a weak match: RRF is designed to fuse two differently-scaled rankings by ORDER, not to reproduce either leg's own scale, so don't read a HYBRID score of 0.02 as "2% relevant," and don't set a similarity-style threshold (e.g. score > 0.5) against it — it will discard everything. By contrast, TEXT- or SEMANTIC-only mode's score is the search engine's own native score for that leg (roughly a 0–1 cosine similarity for SEMANTIC) — a genuinely different scale, which is why the two modes' scores are never comparable to each other or to HYBRID.
  • minTextRelevance applies only to HYBRID; minSimilarity applies only to SEMANTIC / HYBRID. Both have a default floor (see the parameter table), so a result set can be smaller than the matching pool even when you set neither.
  • A miss is a 200, not a 404. Empty results, totalResults: 0.
  • Degradation is silent unless you check. Inspect degraded / degradedLegs, or set requireComplete: true to turn a degraded leg into a 503.
  • INDEXED means indexing is complete for both search strategies; if a search still misses freshly written content, suspect your rate limit before indexing lag. The keyword index is immediately consistent — a document is matchable by TEXT the moment it is INDEXED — and in practice the semantic (vector) index is normally immediately queryable too. A rate-limited request (yours or a retry the SDK made on your behalf) can silently take some time to actually run, which looks identical to "my data isn't there yet" — see the rate limits guide. A freshly indexed document that matches on text is designed to surface in HYBRID and RAG at keyword speed (its semantic score fills in once the vectors catch up), but this fallback exists in code only for uniqueDocuments: false, and like /v1/search's default (uniqueDocuments: true, which RAG's grounding search also uses) it rests only on a unit test with mocked inputs, not a live-verified guarantee; treat it as best-effort on either path. A query that can match a document only semantically (pure-SEMANTIC, or a hybrid query sharing no words with it) has no fallback at all. See explanation.md § "Freshly indexed content".
  • Cross-content search returns mixed types. Always branch on sourceType when rendering unified results.

Inference surfaces — client.inference.*

Three streaming surfaces, all returning an async iterable of SSE events, all requiring the inference:r scope, all sharing one pre-flight check sequence. By default, inference runs against AWS-hosted models inside the Vectros perimeter (in-perimeter for the data plane). This describes the platform-hosted model path. A call you route to your own model provider (providerAlias, which needs a risk-acceptance waiver on your account) is processed where your provider processes it and leaves the Vectros perimeter; residency settings do not apply to it.

Streaming model (shared)

Each surface returns an async iterable; iterate it to consume events in arrival order. Every event carries an event field naming its type, so a consumer can dispatch on one key. On the wire it is standard Server-Sent Events (event: <type> / data: <json> framed by a blank line) — any compliant SSE reader works; the SDK presents it as an async iterator.

Comment lines — lines that start with a colon, such as : keepalive — may appear anywhere in a stream, including before the first event and between events, and compliant readers ignore them. The platform sends one as the stream opens, one about every 2 seconds for the first 10 seconds, and then one about every 15 seconds, so a connection through a proxy or load balancer with an idle timeout is not cut while the model is still working. A hand-written parser must skip any line that starts with a colon and must not treat a block with no data: line as an event. The SDKs already do both. A stream can run for up to 5 minutes in total; a call that reaches that bound ends with an error event and is not billed.

Set your HTTP client's read (idle) timeout to at least 20 seconds. The first call after a quiet period can take several seconds before its first event, and a reasoning model can think for longer than that between frames. A client that gives up after a few seconds of silence (some HTTP libraries default to 5 seconds) can fail on a call that would have succeeded.

Shared event vocabulary:

EventFieldsWhen
reasoning_deltadelta (string)One chunk of the model's reasoning. Sent only when the request sets reasoning: "stream" and the model reasons; always before the first content_delta. It is not part of the answer: do not append it to the answer text.
content_deltadelta (string)One chunk of generated text. Append each to build the answer.
doneinputTokens, outputTokens, model, platformCreditsCharged, inferenceBalanceCentsChargedMilli, optionally cacheReadTokens / cacheCreateTokensTerminal event with token counts, resolved model id, and per-call cost. platformCreditsCharged is the call's platform fee in credits (1 credit = 1 cent), charged to your monthly credit allowance: 0.2 for a call of up to 60 seconds, then 1 credit per 5 minutes of the call's duration, counted from the start of the request to the end of the model's response (a 120-second call is 0.4). inferenceBalanceCentsChargedMilli is the exact amount charged for the call's tokens, an integer in thousandths of a cent (1 cent = 1000). Exactly one.
errormessage, codeA failure after the stream has opened (for example code inference_error). A failure before the stream opens is a plain HTTP error response instead (see Error codes).

Surface-specific events are listed under each surface below.

Pre-flight checks (shared, fixed order)

Run before any model invocation; cheaper checks first:

  1. Action scope → 403. A scoped token must carry inference:r (or a wildcard). Root sk_* keys carry wildcard scope and pass by construction. The 403 does not enumerate the missing action.
  2. Monthly credit limit → 402. Once the period's cumulative credits reach the account's monthly ceiling (the plan's, or a higher one agreed for the account), inference rejects until the period rolls or the ceiling is raised.
  3. Burst rate limit → 429. Per-tenant request-rate ceiling, scaling with plan tier.
  4. Inference billing gate → 402. In balance mode (default on lower tiers), a per-account pre-funded balance must be positive, else 402 Insufficient inference balance. In usage mode (Enterprise-shaped, post-billed), accumulated usage is checked against a contractual cap, else 402 when the cap is reached.

When a call is routed to your own model provider with providerAlias, checks 2 and 4 (monthly credit limit and inference billing) are skipped for that call; see Routing a call to your own model provider.

After these checks, once the prompt is assembled (on RAG, after the retrieval), a balance-mode call reserves the most it could cost against the pre-paid balance, and the unused part is released when the call finishes. If the reservation cannot be made the call is refused before the stream opens: 402 when the balance is short; 429 with errorCode BALANCE_BUSY when the balance is momentarily busy with other concurrent requests of yours; 503 with errorCode BALANCE_UNAVAILABLE when a temporary service fault interrupted the step. The last two are not a shortage: do not top up, and retry after the seconds in the Retry-After header.

The token cost of a call is metered and recorded on stream finalization (after the stream closes). A transient accounting failure does not fail the in-flight response — the balance on the next call may briefly lag.


Grounded corpus answers — client.inference.ragInference(req)

Retrieve-then-generate over your indexed content.

Request parameters:

ParameterTypeDefaultNotes
querystring— (required)The question to ground and answer. The request body, instructions and every other field included, is limited to 8 KB (a larger body is refused with 413).
modelstringclaude-haiku-5-5Model alias. See Model catalog. When providerAlias is set, names the model on that provider instead.
maxTokensinteger1024Output cap. Capped at 4096 (tighter than chat — retrieved context shares the input budget).
temperaturenumber0.3Sampling temperature. Ignored for claude-haiku-5-5, claude-sonnet-5, claude-sonnet-5-5 and claude-opus-5-5, which sample at their own default.
instructionsstring—Optional extra instructions for the answer. Counts toward the 8 KB request-body limit above — put large context in your indexed content and retrieve it with search instead.
providerAliasstring—Route this call through your own model provider. See Routing a call to your own model provider.
reasoning'omit' | 'stream'omitstream sends the model's reasoning as reasoning_delta events; see the reasoning notes under Notes & limits — inference. Any other value is rejected with 400. Ignored when providerAlias is set.
searchobject—Retrieval params (below).

search sub-object mirrors content search: mode (default HYBRID), limit (default 10, valid range 1–100; a value outside that range is rejected with 400, never silently clamped — this is the RAG topK), userId, scope, scopeFilters, folderId, rootFolderId, typeName, filters, contentTypes, createdAfter, createdBefore, requireComplete. scope/scopeFilters behave exactly as documented in the content-search parameter table above — scope for one ownership namespace, scopeFilters (an array, mutually exclusive with scope) for more than one.

Event sequence: search_results → optional truncation_warning → optional reasoning_delta* (only with reasoning: "stream") → content_delta+ → done.

EventFieldsNotes
search_resultsresults[], totalResults, searchTimeMs, degraded, degradedLegs, hasMoreAlways emitted, even when results is empty. hasMore is true when more matching results may exist than results returned — raise search.limit (maximum 100) or narrow your filters. Each entry: documentId, externalId, score, textScore, semanticScore, chunkText, contextText, snippet, metadata, sourceType, typeName, createdAt. These are your citations.
truncation_warningresultsRequested, resultsUsed, truncatedCount, noContentCount, reasonEmitted before the answer if any retrieved passage was dropped from grounding — either because it didn't fit the context budget (truncatedCount) or because it had no groundable text at all (noContentCount, rare in practice). reason is "context_window_budget", "no_groundable_content", or "context_window_budget_and_no_content" when both occurred; the two count fields let you attribute the drop precisely without parsing reason.

Behavior:

  • With search.requireComplete: true, a degraded retrieval leg causes the call to reject before the stream opens (503) instead of grounding on partial results.
  • An empty retrieval still emits search_results (empty) and still streams an answer (typically stating that nothing relevant was found).
  • A scoped token's data scope is enforced on the retrieval step.

Single-document Q&A — client.inference.documentAsk(req)

Ask one question against one document's full text. No retrieval step. Requires inference:r (to run the model) and documents:r (to read the document) — a scoped token must hold documents:r for the document (a grant qualified to a type, documents:r:<type>, covers only documents of that type), within its data scope; inference:r alone returns 404.

Request parameters:

ParameterTypeDefaultNotes
idstring (uuid)— (required)The document to ask against: the {id} in the path POST /v1/documents/{id}/ask (the SDKs take it as a request field).
promptstring— (required)The question.
instructionsstringbuilt-in document-analyst promptOptional system instructions for the answer. When omitted or blank, a default prompt asks the model to answer only from the document text and to say so when the document does not contain the answer.
modelstringclaude-haiku-5-5Model alias. When providerAlias is set, names the model on that provider instead.
maxTokensinteger2048Output cap. Capped at 8192.
temperaturenumber0.2Sampling temperature. Ignored for claude-haiku-5-5, claude-sonnet-5, claude-sonnet-5-5 and claude-opus-5-5, which sample at their own default.
providerAliasstring—Route this call through your own model provider. See Routing a call to your own model provider.
reasoning'omit' | 'stream'omitstream sends the model's reasoning as reasoning_delta events; see the reasoning notes under Notes & limits — inference. Any other value is rejected with 400. Ignored when providerAlias is set.

Event sequence: document_context → optional reasoning_delta* (only with reasoning: "stream") → content_delta+ → done.

EventFieldsNotes
document_contextdocumentId, title, textBytes, modelThe document loaded, its size, and the resolved model. Fires before any generated text.

Errors:

  • 409 (before the stream opens) — the document is not askable yet: it is still processing (not yet fully indexed), it failed ingest, no text could be extracted from its file (its extraction.outcome says why), or its text is not retained (a file uploaded with storeText: false — the extracted text is discarded after indexing, so there is no full text to load). Freshly-ingested documents commonly return 409 until indexing completes.
  • 413 (before the stream opens, no credits charged) — the document's estimated input size exceeds the cap (32,000 input tokens, ~25 pages). Payload: message, estimatedTokens, limitTokens. Branch on this and re-route to RAG.
  • 404 — the document does not exist, belongs to another tenant, is out of your token's data scope, or your token lacks documents:r for it. All return the identical 404; the endpoint never reveals existence outside your read access.

Stateless chat — client.inference.chatInference(req)

Single-turn completion. No retrieval, no stored state.

Request parameters:

ParameterTypeDefaultNotes
messages{ role, content }[]— (required)Conversation. A system role message becomes the system prompt; user / assistant messages pass through. A conversation that ends with an assistant message (a prefill) is refused with 400 on claude-haiku-5-5, claude-sonnet-5, claude-sonnet-5-5 and claude-opus-5-5: end with a user message, or name claude-haiku-4-5 in model to prefill the reply.
modelstringclaude-haiku-5-5Model alias. When providerAlias is set, names the model on that provider instead.
maxTokensinteger2048Output cap. Capped at 8192.
temperaturenumber0.7Sampling temperature. Ignored for claude-haiku-5-5, claude-sonnet-5, claude-sonnet-5-5 and claude-opus-5-5, which sample at their own default.
topPnumber—Nucleus-sampling parameter (optional). Used only when temperature is 0; ignored for the same models as temperature.
providerAliasstring—Route this call through your own model provider. See Routing a call to your own model provider.
reasoning'omit' | 'stream'omitstream sends the model's reasoning as reasoning_delta events; see the reasoning notes under Notes & limits — inference. Any other value is rejected with 400. Ignored when providerAlias is set.

Event sequence: optional reasoning_delta* (only with reasoning: "stream") → content_delta+ → done.

Chat stores nothing. For multi-turn, append the assistant's reply to your messages array and re-send the whole array next turn.


Model catalog — client.inference.listInferenceModels()

Lists every model in the catalog; availableOn names the plans that can call each one.

Response:

FieldTypeNotes
modelsModel[]Every model in the catalog. availableOn names the plans that can call it; a call to a model outside your plan returns 402.
defaultModelstringThe alias used when a call omits model: claude-haiku-5-5 (claude-haiku-4-5 stays in the catalog and is still served to a call that names it). Available on every plan; on a balance-billed account, calling it still needs a funded inference balance.

Each Model:

FieldTypeNotes
idstringAlias: claude-haiku-4-5, claude-haiku-5-5, claude-sonnet-5, claude-sonnet-5-5, claude-opus-5-5, amazon-nova-lite, amazon-nova-2-lite, meta-llama-4-scout, meta-llama-4-maverick. Which of these your key can reach depends on its plan.
namestringDisplay name.
providerstringModel provider.
contextWindowintegerContext window size in tokens.
inputCreditsPer1kTokensnumberBase (non-US) input rate — equal to regionPricing.base.input. Default-served (US) requests are billed regionPricing.us.input instead; see below.
outputCreditsPer1kTokensnumberBase (non-US) output rate — equal to regionPricing.base.output. Default-served (US) requests are billed regionPricing.us.output instead; see below.
regionPricingobjectPer-region rate breakdown for this model — base, us, regionPremiumFactor, globalAvailable. See "Region pricing" below.
availableOnstring[]Plan tiers that may call this model (e.g. free, starter, pro, scale, enterprise).
longContextobjectOptional; present only for a model priced by prompt length (claude-haiku-5-5). promptTokensOver (100000), multiplier (5.0), and base / us credit rates per 1K input and output tokens. See "Prompt-length pricing" below.
cachePricingobjectOptional; present only for a model with prompt-cache prices (the five Claude models). base and us, each with read, write5m and write1h: credits per 1K input tokens read from the prompt cache, written to it with a 5-minute lifetime, and written with a 1-hour lifetime. A model without the field has no cache prices. See "Prompt-cache pricing" below.

Requesting a model your plan does not include returns a 402 pointing to upgrade; an alias that is not in the catalog returns 400. A lighter model is available on every tier; more capable models require higher tiers.

Prompt-length pricing

A model with a longContext object is priced by the length of the PROMPT of each request. A request whose prompt is up to longContext.promptTokensOver tokens is charged the standard rates of the model (inputCreditsPer1kTokens, outputCreditsPer1kTokens, regionPricing). A request whose prompt is LONGER is charged longContext.multiplier times those rates on every token of that request, input and output (longContext.base and longContext.us carry the resulting rates). A prompt of exactly promptTokensOver tokens is not over the line. The prompt's length is the one the model provider reports when the call completes, so a request near the line is charged at the rate its real size falls in then; the balance hold taken before the call assumes the higher rate when the prompt looks near or over the line, and the unused part is refunded (a balance-mode account whose balance covers the real cost but not the held amount can be refused with 402).

Token cost is charged in milli-cents (thousandths of a cent) and is not rounded to whole cents: every call that uses tokens draws its exact cost from your inference balance, rounded to the nearest thousandth of a cent, in addition to the per-call platform fee.

Prompt-cache pricing

A model with a cachePricing object charges the input tokens its provider reports as read from, or written to, its prompt cache at the cache rates instead of the input rate (cacheReadTokens and cacheCreateTokens on the done event). The rates are the input rate times a weight: a cache read is 0.10 of the input rate (0.05 on claude-sonnet-5-5 and claude-opus-5-5), a write with a 5-minute lifetime is 1.25, and a write with a 1-hour lifetime is 2.0. cachePricing.base is built on the base input rate and cachePricing.us on the US-served input rate, so us is base times regionPricing.regionPremiumFactor. Credits per 1K input tokens, base rates:

ModelInputCache readCache write, 5 minutesCache write, 1 hour
claude-haiku-4-50.10.010.1250.2
claude-haiku-5-50.010.0010.01250.02
claude-sonnet-50.20.020.250.4
claude-sonnet-5-50.20.010.250.4
claude-opus-5-50.40.020.50.8

amazon-nova-lite, amazon-nova-2-lite, meta-llama-4-scout and meta-llama-4-maverick have no cache prices: their cachePricing is absent, and a cache count on one is not charged on its own. Read the exact figures from GET /v1/models.

For claude-haiku-5-5, the prompt-length multiplier applies to the cache rates too, and cache read and cache write tokens count toward the length of the prompt. On a prepaid balance, the amount held before a call on a model with cache prices covers the input at the 5-minute cache-write rate (1.25 times the input rate) and the unused part is released when the call ends, so a call that touches no cache is charged what it would be without cache prices; a balance close to the held amount can be refused with 402. Agent runs do not use the cache rates: a run charges any cache token at the input rate. A call routed with providerAlias is not charged for tokens.


Region serving (allowGlobalRegion)

All three inference surfaces (chat, RAG, document-ask) accept an optional boolean allowGlobalRegion in the request body.

FieldTypeDefaultMeaning
allowGlobalRegionbooleantenant residency defaultOpt this request into the lower-cost global (non-US) region path.
  • The tenant's residency default is US serving, applied when the flag is omitted. US serving is the fail-closed default and carries a region premium.
  • Setting allowGlobalRegion: true lets an entitled tenant serve the request from the global region at a lower rate. Entitlement is gated on a signed global-processing waiver (dev portal → Settings → Data Residency & Region; see the operations & trust reference for the residency posture this entitlement governs).
  • If allowGlobalRegion: true is sent by a tenant that is not entitled, the request is rejected with 403 — it is not silently downgraded or upgraded. Region choice never changes which content is retrieved, only where the model runs and the price.
  • If Vectros requires a newer version of the global-processing waiver, an earlier signature no longer covers it once the date announced to account owners has passed: the tenant returns to US-only serving, and allowGlobalRegion: true is rejected with 403, until an account owner signs the new version.
  • allowGlobalRegion has no effect on a call routed with providerAlias; see Routing a call to your own model provider.

Region pricing (regionPricing)

Every model returned by the model catalog carries a regionPricing object so you can compute the exact billed rate for either region without a client-side multiply:

FieldTypeNotes
regionPricing.base{ input, output }Per-1K credit rates billed when a request is served from the global (non-US) region — i.e. allowGlobalRegion: true on an entitled tenant. Equal to the model's inputCreditsPer1kTokens / outputCreditsPer1kTokens.
regionPricing.us{ input, output }Per-1K credit rates billed when a request is served from the US region — the default. Equal to base multiplied by regionPremiumFactor.
regionPricing.regionPremiumFactornumberMultiplier applied to base to get us. 1.0 means no premium (base == us); a Bedrock cross-region model typically carries a premium above 1.0 (e.g. 1.1).
regionPricing.globalAvailablebooleanWhether this model can be served from the global region at all. Every model in the catalog currently reports true.

A default-served (no allowGlobalRegion, or an unentitled tenant) request is always billed regionPricing.us; an entitled tenant opting in with allowGlobalRegion: true is billed regionPricing.base. Use this object to show accurate per-region pricing in a model picker before the caller decides whether to opt into global serving.


Routing a call to your own model provider (providerAlias)

All three inference surfaces (chat, RAG, document-ask) accept an optional providerAlias string in the request body. It names a provider config that the account owner has set up (a customer-supplied Anthropic key, or an OpenAI-compatible endpoint such as a self-hosted vLLM or TGI cluster) and routes that one call through it instead of the platform-hosted models.

  • Alias. A provider alias is 1 to 64 lowercase letters, digits, - or _, unique within the tenant. A call must send it exactly as stored: the match is case-sensitive. The rule applies when a config is created and when its alias is changed; a config that already holds an alias outside the rule keeps working with it.
  • Endpoint and other fields. A config you set up yourself takes a plain https endpoint on port 443 (no credentials, query or fragment; the platform adds the provider type's standard API path, so leave it out), the byo-model-inference:customer-key waiver type, and the CUSTOMER_DIRECT billing mode, which is the default when omitted. The provider key is printable ASCII of at most 4096 characters, and a tenant can hold at most 25 configs. A request outside these rules is rejected with 400; a value a config already holds is accepted. A save that loses a race with another change answers 409.
  • Provider type. A request that changes the providerType of an existing config, in either direction, must state the endpoint too (an empty string means the provider's default), or it is rejected with 400: a stored endpoint overrides the Anthropic default at call time, so a type change that left it unsaid would keep sending calls to the old host. A stated endpoint on a different host, which includes "" on a config that has one, needs the provider key again.
  • Model. When providerAlias is set, model names the model on that provider's own id space (an opaque string to Vectros, not a catalog alias) and falls back to the provider config's default model when omitted. A request that sets neither is rejected with 400. The plan gate on the model catalog does not apply to a provider-side model id.
  • Entitlement. Your account must hold the matching risk-acceptance waiver. An unknown or inactive alias, or a missing waiver, is rejected with 403. An account owner signs the waiver for an environment in the developer portal (Model Providers). If Vectros requires a newer version of the waiver, an earlier signature stops counting once the date announced to account owners has passed, until an owner signs the current version.
  • Business Associate Agreement attestation. If your account holds a Business Associate Agreement with Vectros (one accepted in the developer portal; an agreement made outside it does not make the account a holder here), a provider config is usable only while an owner has attested, for that config, that the account holds a business associate agreement, or another agreement that HIPAA requires, with the provider at the config's endpoint, covering the information the account will send to it, and that the account is permitted to direct that disclosure. The owner gives the attestation per config, in the developer portal (Model Providers) or the developer API, and the attestation is kept as a signed record. Through the API the request states the provider type and endpoint being attested (an empty endpoint means the provider's default); a request that omits them is refused with 400, and one that no longer matches the config is refused with 409 and records nothing. It applies to the test and live environments alike, and to a config set up for you by Vectros. It belongs to the config's provider type and endpoint, exactly as saved: any change to either, including a change of letter case or a trailing slash, withdraws it in the same change, and calls through the config are then rejected with 403 until an owner attests again (normally within moments of the change). If Vectros cannot establish whether the account holds the agreement, it treats the account as a holder. An account that does not hold the agreement is not asked for an attestation.
  • Region. allowGlobalRegion and your residency settings do not apply to a routed call; your provider decides where it processes the call.
  • Perimeter. A routed call is processed where your provider processes it and leaves the Vectros perimeter. The in-perimeter description elsewhere in this guide is for the platform-hosted model path. For a routed RAG call, the retrieval still runs inside the perimeter (a semantic or hybrid search embeds your query with a Vectros-hosted model); the query and the retrieved passages are then sent to your provider. Vectros delivers to the endpoint saved on the config (for an Anthropic config with no endpoint saved, Anthropic's published API host), with the provider type's standard API path added (for example /v1/messages; save the endpoint without it, and a trailing slash is trimmed), only over https with the server certificate validated, and without following redirects.
  • Billing. The provider bills you directly; Vectros does not price a routed call's tokens. The call skips the monthly credit-limit and inference-balance checks and does not draw the inference balance; the per-call platform fee still applies, and the done event's inferenceBalanceCentsChargedMilli is 0.
  • Management. The account owner manages provider configs in the developer portal (Model Providers), the developer API and the CLI's vectros provider-config commands. A call without providerAlias is served by the platform-hosted models.

Error codes (inference)

CodeSurfaceMeaning
400allMalformed request (e.g. missing query / messages / prompt, bad filter key, an alias outside the model catalog, or a providerAlias call with no model and no default model on the provider config). RAG also returns 400 for a search.limit outside 1–100. POST /v1/chat also answers 400 with errorCode: CONTEXT_WINDOW_EXCEEDED when the conversation plus maxTokens cannot fit the model's context window. It also answers 400 when the last message has role assistant on a model that does not accept a prefill (see messages). Any surface answers 400 when the body is not valid JSON (Malformed JSON body) and when reasoning is anything other than omit or stream.
402allMonthly credit limit exceeded, usage cap reached, or a requested model the plan does not include (errorCode SUBSCRIPTION_LIMIT_EXCEEDED); insufficient inference balance, or a balance that cannot cover the call's reservation (errorCode INSUFFICIENT_BALANCE). A funded balance that is momentarily busy or interrupted is 429 / 503 below, never 402.
403allToken scope does not permit inference (inference:r missing); or allowGlobalRegion: true was sent but this tenant is not entitled to global-region serving (no signed global-processing waiver); or providerAlias names an alias that is unknown or inactive for your account, or your account lacks the risk-acceptance waiver for it, in which case errorCode says which: WAIVER_NOT_SIGNED (the waiver has not been signed; an account owner signs it in the developer portal, Model Providers), WAIVER_TEXT_CHANGED (the version of the waiver in force for your account is older than the version Vectros requires; an owner must review and sign the current version; editing a waiver's text refuses nothing), WAIVER_REVOKED (an owner revoked it; an owner can sign it again); or, on an account that has accepted the Business Associate Agreement with Vectros in the developer portal, the alias's provider config has no current attestation; or the alias's provider config holds a retired billing mode (move it to CUSTOMER_DIRECT).
404allApp context not found. when the app context of the call does not exist or is being deleted. On document-ask also: document not found / cross-tenant / out-of-scope (uniform — existence never revealed).
409document-askDocument not askable yet — still processing (not yet indexed), failed ingest, or text not retained (a file uploaded with storeText: false). Returned before the stream opens.
413allThe request body is too large: over 1 MiB (1,048,576 bytes) on chat and document-ask, or over 8 KB on RAG; the value is PAYLOAD_TOO_LARGE in the field code (answered by the edge) or errorCode (answered by the platform). Document-ask also: the document exceeds the input-token cap. Before the stream opens; no credits charged.
429allBurst rate limit exceeded; or, with errorCode BALANCE_BUSY, your pre-paid inference balance is momentarily busy with other concurrent requests of yours. That is not a shortage (that is 402) and not a rate-limit breach: do not top up, retry after the seconds in the Retry-After header.
500RAGThe retrieval step failed (Search failed) before the stream opened.
503RAG, searchA retrieval/search leg was unavailable and requireComplete / requireComplete: true was set. On RAG this 503 carries no errorCode.
503all inference calls (errorCode BALANCE_UNAVAILABLE)A temporary service fault interrupted the pre-paid balance step of this call. It is not a shortage (that is 402): do not top up. Retry after the seconds in the Retry-After header. On RAG, branch on errorCode to tell it from the requireComplete case.
503all inference calls, with providerAliasThe stored key of the provider config cannot be opened (the platform's encryption service is unavailable). A provider key you enter from release 0.48.0 on is stored encrypted; a key stored before that is read as it is and cannot produce this 503. Nothing is sent to the provider and the reserved balance is released. The response carries no errorCode and no Retry-After: retry shortly, and if it persists enter the key again in the developer portal.

Notes & limits — inference

  • Hard output caps per surface: chat 8192, RAG 4096, document-ask 8192 output tokens. Document-ask additionally caps input at 32,000 tokens (~25 pages) with a 413 before the stream opens. A maxTokens above a surface's cap is floored to the cap.
  • claude-haiku-5-5, claude-sonnet-5, claude-sonnet-5-5 and claude-opus-5-5 can reason before they answer, and when they do, maxTokens includes the reasoning. The reasoning counts against maxTokens together with the answer and is billed as output whether or not you receive it (on claude-opus-5-5, the one measured, the usage figure counts both and the split is not reported), so a small maxTokens, or a request that makes the model reason at length, can return little or no text: the stream ends with done and an output-token count at or near maxTokens. Leave headroom (a maxTokens of 1024 or more for a short answer).
  • reasoning: "stream" delivers the reasoning as its own events; the default does not. With reasoning omitted or "omit", the stream carries only the answer, exactly as before. With "stream", a model that reasons sends reasoning_delta events before the first content_delta (on RAG after search_results and any truncation_warning; on document-ask after document_context), and a model that does not reason sends none. done is unchanged, and the token counts and charge are the same as without the option. Vectros does not log or store the reasoning. It can quote your documents or messages, so treat it as you treat the answer. The field applies to platform-hosted models: a call routed with providerAlias ignores it. A value other than omit or stream is rejected with 400 before anything is charged.
  • Request body: at most 1 MiB (1,048,576 bytes) on chat and document-ask; at most 8 KB on RAG. A larger chat or document-ask body is refused with 413 and PAYLOAD_TOO_LARGE before anything is charged (the edge firewall answers it in the field code, the platform in errorCode; read either); shorten the conversation or send fewer or shorter messages. 1 MiB is roughly 250,000 to 400,000 tokens (it depends on the text and the model), which covers the 100,000-token prompt-length price tier (see "Prompt-length pricing"). A prompt beyond 1 MiB is not supported (a body so large that the API gateway itself refuses it, about 6 MB, is answered by the gateway, not by this 413). A RAG body over 8 KB is refused with 413 by the edge, so a large RAG context belongs in indexed content. The 413 the edge returns names the field code; the one from the platform names it errorCode; both carry the value PAYLOAD_TOO_LARGE. A prompt must also fit the model's contextWindow (see the model catalog for each model: claude-haiku-4-5 has 200,000 tokens, the default model claude-haiku-5-5 1,000,000). POST /v1/chat refuses a conversation whose estimated size plus maxTokens exceeds the window with 400 and errorCode: CONTEXT_WINDOW_EXCEEDED, before anything is charged (a call routed to your own provider with providerAlias is not checked). The estimate is approximate: a conversation within a few percent of the window can be refused although it would have fit, or accepted and still rejected by the model provider, in which case the call ends with an error event and nothing is charged.
  • RAG topK is bounded at 100. search.limit defaults to 10 and accepts 1–100; a value outside that range is rejected with 400 rather than clamped. When hasMore on the search_results event is true, more matching results may exist than were retrieved. Retrieved context that does not fit the model's input budget is dropped from grounding and reported in a truncation_warning event.
  • query and instructions are for a question and prompt-shaping text. The request body is limited to 8 KB (413), query and instructions included. Put large context in your indexed content and retrieve it with search.
  • Chat is stateless; there is no managed conversation state. No server-side thread store, no assistants registry. Multi-turn is the caller's responsibility — re-send the messages array each turn and budget the history against the model's context window.
  • Document-ask is single-document. No multi-document Q&A endpoint. For multi-document grounding, use RAG (retrieval picks relevant passages across the corpus) or stitch multiple /ask calls at the application layer.
  • Cost is recorded on finalization. The done event carries platformCreditsCharged and inferenceBalanceCentsChargedMilli; an accounting hiccup will not fail an in-flight response, so the next call's balance may briefly lag.
  • Disconnecting does not stop the model call. A call is billed on the provider's reported usage. The platform does not currently stop the model call when the client disconnects, so a call you abandon can still run to completion and be billed.
  • Cache tokens are charged at the model's cache rates. done may carry cacheReadTokens / cacheCreateTokens. On a model that lists cachePricing in GET /v1/models they are charged at its cache read and write rates and included in inferenceBalanceCentsChargedMilli; on a model without it they are not charged on their own. These endpoints send no cache checkpoint, so a non-zero count comes only from a provider that caches on its own. A call routed through providerAlias is not charged.
  • In-perimeter scope. The in-perimeter description (no third-party model-vendor egress) is for the data plane on the platform-hosted model path — the content you store and retrieve through Vectros. A call you route to your own model provider (providerAlias, which needs a risk-acceptance waiver on your account) is processed where your provider processes it and leaves the Vectros perimeter. Specific compliance coverage terms are addressed in the security and compliance documentation, not asserted here.
  • The model catalog is the source of truth. Handlers gate on the live catalog at request time, so a model going generally available or being retired takes effect immediately; the plans in availableOn are enforced when you call the model.

Where to go next