Capabilities¶

Every capability the broker exposes follows the same shape: an agent calls a broker endpoint with its scoped token, the broker enforces quota and policy, calls the upstream provider with the real credential, and returns a normalized result. The agent never sees the credential.

Web search — the provider ladder¶

Raw web search is a chain: Nous Fast Search (included, free) → Tavily → SearXNG (self-hosted, free) → Serper → Brave, skipping rungs that are unconfigured or circuit-open. The caller can force an order or subset; the response reports which rungs were attempted before the winner, so failures are visible, not silent.

{
  "q": "current energy tariffs bogota",
  "count": 5,
  "country": "co",
  "lang": "es",
  "providers": ["searxng", "serper"]
}

{
  "provider": "serper",
  "query": "...",
  "results": [{"title": "", "url": "", "description": "", "age": null}],
  "attempted": ["searxng"]
}

The first rung has zero incremental cost, Tavily is the default quality rung, and the self-hosted SearXNG rung keeps queries working without paid APIs — a concrete example of the lab's cost discipline.

Research with citations — Perplexity¶

Synthesized answers with citations (model strictly sonar / sonar-pro), billed against a per-tenant unit budget (completion tokens):

{
  "q": "what are the best practices for k8s secret rotation",
  "model": "sonar",
  "max_tokens": 1024
}

Scraping — Crawl4AI → Firecrawl → Nous Firecrawl¶

Page-to-markdown conversion, chained crawl4ai (self-hosted) → firecrawl → nous-firecrawl, with an SSRF-guarded URL policy (public addresses only, no embedded credentials, redirect re-check) and a 2 MiB content cap:

{"url": "https://example.com/docs", "format": "markdown", "timeout_seconds": 60}

Speech-to-text — ElevenLabs → Deepgram → OpenAI¶

Voice notes are transcribed with ElevenLabs first, then Deepgram, then OpenAI Whisper as the last fallback. The broker normalizes decodable audio (16 kHz mono WAV via ffmpeg), probes metadata, and returns specific errors rather than a generic failure:

Error	Meaning
`empty_or_silent_audio`	Empty, zero-duration, or decoded-silent audio
`empty_transcript_provider`	Provider returned HTTP 2xx but no transcript
`provider_timeout`	Upstream STT provider timed out
`budget_exceeded`	Broker quota denied the request
`all_providers_failed`	No provider produced a usable transcript

A request ID is threaded through gateway → broker → provider so a voice note can be correlated end-to-end.

Text-to-speech¶

TTS chains Deepgram → OpenAI by default; selected agents use ElevenLabs first. It returns base64 audio with configurable voice, language, speed, and format:

{
  "text": "Hello, this is a reminder from your assistant.",
  "voice": "carina",
  "language": "en",
  "speed": 1.0,
  "format": "mp3"
}

Voice calls — Telnyx call control¶

The broker dials phone calls through Telnyx and plays a TTS message:

{
  "to": "<E.164 destination>",
  "message": "Reminder from your assistant",
  "voice": "female",
  "language": "en-US",
  "hangup_after": 60
}

A durable call row is created before dialing; the initial response only means "accepted for dialing".
Delivery is confirmed by Telnyx Call Control webhooks, verified with Ed25519 signatures and idempotent by event id.
Anti-spam policy is enforced per call: quiet hours, max calls per day, minimum interval, and importance/preferences from tenant settings.
Status and metrics endpoints track states from call_queued through call_completed, no_answer, busy, and failed.

Browser automation¶

Browser work runs only on the self-hosted browser service: a persistent, human-watchable browser for logins, captchas, and payments. Cloud browsers are forbidden by policy — the broker's routing endpoint always reports cloud_allowed: false. The broker proxies tasks, handoffs, and payment-intent state (payment authorization stays human-only), and ownership is enforced server-side per (tenant, agent, user).

The broker rejects card data outright: a 13–19 digit run is blocked only when it passes the Luhn checksum, so bill references and NIU numbers still work, and CVV mentions are blocked unconditionally.

Image generation — Higgsfield¶

/v1/media/higgsfield/generate runs the pinned Higgsfield CLI with broker-owned credentials. If the CLI or its credentials are missing, the capability reports as degraded (HTTP 200, listed in degraded_capabilities) rather than taking down the whole broker — optional capabilities must never remove core routing from the Service.

MCP proxy — brokered services¶

Registered internal MCP servers are exposed as /v1/mcp/<service> (streamable HTTP), so gateways can configure MCP servers that point at the broker instead of the raw service:

Services must be explicitly registered with capability, kind, allowed methods, byte caps, and timeout.
Service hosts must match the broker's allowlist.
Authorization/Cookie/provider-key-style headers are never forwarded; optional upstream auth is injected from broker env.
Requests and responses are capped and audited.

Example registration shape:

{
  "services": {
    "scrapling": {
      "url": "http://internal-mcp:8000/mcp",
      "capability": "mcp:scrapling",
      "provider": "scrapling-mcp",
      "kind": "mcp-streamable-http",
      "allowed_methods": ["POST", "GET"],
      "max_body_bytes": 8388608,
      "max_response_bytes": 16777216,
      "timeout_seconds": 120
    }
  }
}

Integrations — Composio (least privilege)¶

One fixed, reviewed read operation (repository listing) is exposed through Composio with an HMAC-derived identity layer and a reservation ledger that limits attempts per minute/day/total. It is the pattern for "how to expose an external agent platform to tenants without opening a generic proxy".

Circuit breaker & introspection¶

The broker tracks upstream failures per provider/capability and opens circuits so a dead rung is skipped fast. /v1/capabilities, /v1/services, /v1/limits/status, and /v1/circuit/status let operators and agents introspect the live state — with the detail that core_errors produce HTTP 503 while degraded_capabilities do not.