LLM Fundamentals & Prompting

LLM Fundamentals & Prompting — how a model reads, bills, and generates

Mental model: a language model never sees your words. It sees tokens — the model's
unit of text, and the unit you're billed on. Every call the model reads your whole
input as tokens, predicts the next token one at a time, and streams them back until it
signals it's done. Four levers follow from that: tokens (unit + cost), the context
window (how many tokens fit), temperature (how the next token is picked), and the
stop reason (how it knows to stop).

The SDK shape (@anthropic-ai/sdk). Every call is one POST to /v1/messages:

    // Node — npm install @anthropic-ai/sdk
    import Anthropic from "@anthropic-ai/sdk";
    const client = new Anthropic();          // reads ANTHROPIC_API_KEY

    const res = await client.messages.create({
      model: "claude-opus-4-8",
      max_tokens: 1024,                       // hard ceiling on OUTPUT tokens
      system: "You name fantasy taverns.",    // counted as input every call
      messages: [{ role: "user", content: "Name one tavern." }],
    });

    # Python — pip install anthropic
    from anthropic import Anthropic
    client = Anthropic()
    res = client.messages.create(
        model="claude-opus-4-8",
        max_tokens=1024,
        system="You name fantasy taverns.",
        messages=[{"role": "user", "content": "Name one tavern."}],
    )

1 · Count tokens BEFORE you send  (billed before you ever call create)
    // countTokens takes the same shape as create; returns input_tokens
    const c = await client.messages.countTokens({
      model: "claude-opus-4-8",
      messages: [{ role: "user", content: "..." }],
    });
    // c.input_tokens -> 21

    Real run: countTokens returned input_tokens = 21. That is the exact number of
    tokens the model would read for that prompt — the same unit you pay for.
    (Python: client.messages.count_tokens(...).input_tokens — never tiktoken; that's
    a different tokenizer and mis-counts Claude.)

2 · Meter a real call  (usage tells you what you actually paid)
    Real run of messages.create on a "what is a token" prompt:
      usage.input_tokens  = 30
      usage.output_tokens = 41
      stop_reason         = end_turn

    input_tokens (30) > the 21 above because this call carried a system prompt too.
    The system prompt + any few-shot examples are re-sent and re-billed on EVERY
    call — they are input tokens every single time, not a one-time setup cost.
    output_tokens (41) is what the model generated. stop_reason = end_turn means the
    model finished on its own (see step 5).

3 · The context window — the fixed token budget
    input_tokens + output_tokens must fit inside the model's context window: a fixed
    token budget for everything the model attends to at once — your prompt, the
    conversation history, and the output it's generating. It is the real constraint
    you design around. Fill it and something has to be dropped. This is why the
    re-billing in step 2 matters: history and few-shot examples don't just cost money,
    they consume the window on every turn.
    #1 gotcha: the running cost/space isn't "your latest question" — it's your latest
    question PLUS the system prompt PLUS all prior turns, re-counted each call.

4 · Temperature — the determinism lever
    Reading that context, the model predicts the single most likely next token.
    Temperature tunes how that pick is made: low = focused/repeatable, high = broader
    sampling. Same prompt ("give me tavern names"), two temperatures:

      temperature 0  ->  ["The Crimson Kraken", "The Crimson Kraken"]
                         (asked twice, identical both times — focused, repeatable)
      temperature 1  ->  ["The Copper Dragon", "The Wandering Griffin", "The Rusty Griffin"]
                         (three runs, three different names — wider sampling, varies)

    Rule of thumb: temperature 0 for extraction/classification/anything you want
    stable; temperature ~1 for brainstorming/variety.

5 · One token at a time, streamed, until it stops
    The model generates the answer one token at a time: it appends each predicted
    token, feeds the whole thing back in, and predicts the next — repeating until it
    emits an end-of-turn signal. That's stop_reason = end_turn (seen in step 2 —
    the model decided it was done). Streaming shows those tokens as they're produced
    instead of waiting for the whole response:

      const stream = client.messages.stream({ model: "claude-opus-4-8",
        max_tokens: 1024, messages: [...] });
      for await (const e of stream)
        if (e.type === "content_block_delta" && e.delta.type === "text_delta")
          process.stdout.write(e.delta.text);   // tokens appear live

    Other stop_reasons you'll meet: max_tokens (hit your output ceiling — the answer
    was cut off, raise max_tokens or stream), tool_use (the model is asking you to
    run a tool), refusal (declined for safety).

What each part does
    token          the model's unit of text AND the billed unit (input + output)
    countTokens    price a prompt before sending — returned input_tokens = 21 here
    usage          what a real call actually metered — input=30, output=41
    system prompt  re-sent and re-billed as input tokens on every call
    few-shot       same — examples count as input tokens every call
    context window fixed token budget for prompt + history + output combined
    temperature    0 = focused/repeatable, 1 = wider sampling (see the real runs)
    stop_reason    end_turn = model finished; max_tokens = cut off at the cap

Notes
    - temperature 0 is LOW-VARIANCE, not a byte-identical guarantee. It returned
      "The Crimson Kraken" twice here, but "0" means greedy-ish sampling, not a
      promise the bytes match across every run, model version, or machine. Treat it
      as "as stable as it gets," not "deterministic."
    - You pay for input AND output. output tokens are typically priced higher than
      input — long generations cost more than long prompts of the same length.
    - There is no server-side memory between calls. The API is stateless; "history"
      exists only because you re-send it (and re-pay for it) each turn.
    - max_tokens caps OUTPUT only. It does not limit how big your input can be — that
      is the context window's job.
    - Prompting, not temperature, is the primary steering tool. Some current models
      remove temperature entirely; shape behavior with the system prompt and examples.

Verify
    Re-run token counting on any prompt and confirm the unit:
      const c = await client.messages.countTokens({
        model: "claude-opus-4-8",
        messages: [{ role: "user", content: "Name one tavern." }],
      });
      console.log(c.input_tokens);            // the tokens you'll be billed for
    Then make one real call and inspect what it actually metered:
      const r = await client.messages.create({ model: "claude-opus-4-8",
        max_tokens: 1024, system: "You name taverns.",
        messages: [{ role: "user", content: "Name one tavern." }] });
      console.log(r.usage.input_tokens, r.usage.output_tokens, r.stop_reason);
    Add a system prompt or a few-shot example and re-check usage.input_tokens — it
    goes UP, proving those tokens are re-billed every call. Run the same prompt twice
    at temperature 0, then twice at temperature 1, and compare: temperature 0 repeats,
    temperature 1 varies.