Managed LLM APIs

Managed LLM APIs — call a top model without running one

Mental model: a managed LLM API is one HTTP endpoint that serves a top-tier
model for you. You send a request; the provider runs the inference on their
GPUs and streams the answer back. You never buy, provision, or operate the
hardware. Because you call it through your own cloud account, the request stays
inside your IAM, your networking, and your compliance boundary — the model comes
to your data instead of your data leaving for an outside service.

Why it works this way: inference for a frontier model needs expensive GPU
capacity kept warm and scaled. A managed API amortizes that across every tenant
and hands you a single call. Your job shrinks to three decisions — which model,
which region, and how many tokens you spend.

1 · The single call (SDK shape)
    A managed messages/create call takes the model, a token cap, and the
    conversation, and returns both the text and a usage meter:

    response = client.messages.create(
        model="claude-haiku-4-5",     # model choice = capability vs price
        max_tokens=1024,              # hard ceiling on output length
        messages=[{"role": "user", "content": "What is a token?"}],
    )
    print(response.content[0].text)
    print(response.usage.input_tokens, response.usage.output_tokens)

2 · Model choice is a capability/price trade-off
    Over that one endpoint you pick the model. A bigger model is more capable
    but costs more per token; a smaller one is cheaper and faster. Match the
    model to the job — don't pay flagship prices for a classification task.

3 · Billing is per token, input + output — the real numbers
    You are billed on both sides. A real Claude API call metered:

      usage: input_tokens=30  output_tokens=51   (from a live response)

    On the cheap Haiku model, input is $1 per million tokens and output is
    $5 per million. The cost of that call:

      input  ->  30  tokens  x $1/1,000,000  = $0.000030
      output ->  51  tokens  x $5/1,000,000  = $0.000255
      total  ->                              = $0.000285   (~$0.0003)

    So prompt length and response length drive cost directly. The token meter
    isn't just billing — it's how you track usage, forecast spend, and spot a
    prompt that's quietly getting expensive.

4 · Region + quotas
    You pick a region so data is processed where your rules require it (data
    residency). You also stay under the account's quotas and rate limits — if
    you exceed them, the endpoint throttles you (HTTP 429) rather than serving
    the request.

The big three managed platforms (covered in later lessons):
    AWS Bedrock         -> unified runtime API across model providers
    Google Vertex AI    -> managed GenAI platform, Gemini + Model Garden
    Azure AI Foundry    -> models inside the Azure boundary (Azure OpenAI)
    Each wraps the same mechanics: one endpoint, IAM boundary, per-token billing.

Verify:
    Read response.usage.input_tokens and response.usage.output_tokens after any
    call, then multiply by the model's per-million input/output rates to get the
    exact cost — e.g. 30 in + 51 out on Haiku = $0.000285.

Gotcha:
    Cost scales with BOTH sides of the meter. A short prompt with a long,
    rambling response can cost more than a long prompt with a terse answer,
    because output tokens are the pricier side (here $5/M vs $1/M).