Making Disagreement Computable: The Engineering Behind Munazara

A live Munazara debate streaming to a structured verdict

Ask a single language model a contested question and you get one confident answer with no seams — no view of what it weighed, what it discarded, or where a reasonable model would have gone the other way. The confidence is uniform whether the question is settled or a coin-flip.

Munazara (مناظرہ — the classical word for a formal scholarly disputation) is an attempt to make that seam visible by construction. Two models from different vendors argue opposing sides of your question; a third model, from a vendor not in the fight, adjudicates. But "have two LLMs talk" is a demo, not a design. The engineering problem worth writing about is the one underneath: how do you turn a debate into something a state machine can drive, terminate, and stream — and how do you keep the whole thing from becoming two essays talking past each other?

This post is about the parts that were actually hard.

The debate is a state machine, not a conversation

The whole run is a compiled LangGraph. Five nodes, one loop:

flowchart TD
    Q[question] --> EX[exhibit<br/>frame the question]
    EX --> A[debater_a]
    A --> B[debater_b]
    B --> CONV[convergence<br/>increment round_count]
    CONV -->|should_continue: disputes remain<br/>and round_count &lt; max| A
    CONV -->|should_continue: no disputes,<br/>round cap, or both confident| J[judge]
    J --> END[verdict]

The wiring is deliberately sequential, not concurrent:

workflow.set_entry_point("exhibit")
workflow.add_edge("exhibit", "debater_a")
workflow.add_edge("debater_a", "debater_b")
workflow.add_edge("debater_b", "convergence")
workflow.add_conditional_edges(
    "convergence", should_continue,
    {"debater_a": "debater_a", "judge": "judge"},
)
workflow.add_edge("judge", END)

This matters. debater_b runs after debater_a within the same round and is handed A's just-completed turn, so B is genuinely rebutting a specific argument rather than firing into the void. A truly concurrent design — both debaters answering the same prompt in parallel — is cheaper in wall-clock but produces two opening statements per round that never actually engage. Sequential turns are the thing that makes it a debate. The UI still feels concurrent because tokens stream (more on that below), but the graph is strictly ordered.

State: one append-only channel, several last-write-wins channels

The debate state is a TypedDict, and the interesting part is which fields are reducers and which are plain overwrites:

class DebateState(TypedDict):
    transcript: Annotated[list[dict], add]     # append-only, grows every turn
    agreements: Annotated[list[str], add]      # append-only
    round_count: int
    last_a_disputes: list[dict]                # overwritten each round
    last_b_disputes: list[dict]                # overwritten each round
    last_a_confidence: float
    last_b_confidence: float
    ...

transcript uses LangGraph's add reducer, so each node returns just its delta — return {"transcript": [turn]} — and the framework concatenates. The full history is never rebuilt or passed around; it accretes.

The last_* fields are the opposite on purpose. They're not history — they're routing signals, snapshots of the most recent round that the convergence check reads to decide whether to loop. Making them last-write-wins keeps the termination decision reading this round's state, not an accumulated blur of every round. Separating the append-only record from the overwrite-only control signals is what keeps the loop logic legible.

The convergence controller, and a 276-second lesson

Termination is two functions. One node mutates the counter; one edge function reads it. Nothing else touches round_count:

def convergence_node(state):
    new_round = state["round_count"] + 1     # the ONLY increment in the codebase
    all_disputes = {d.get("claim","") for d in state["last_a_disputes"]} \
                 | {d.get("claim","") for d in state["last_b_disputes"]}
    return {"round_count": new_round, "open_disputes": list(all_disputes)}

def should_continue(state):
    if state["round_count"] >= state["max_rounds"]:
        return "judge"                                     # 1. round cap
    if not state["last_a_disputes"] and not state["last_b_disputes"]:
        return "judge"                                     # 2. mutual convergence
    if (state["round_count"] >= 1
            and state["last_a_confidence"] >= 0.85
            and state["last_b_confidence"] >= 0.85):
        return "judge"                                     # 3. confident deadlock
    return "debater_a"                                     # else: another round

Three ways out: the round cap, both sides running out of disputes (genuine convergence), or both sides highly confident after at least one exchange (a stable disagreement — continuing won't help). The round_count >= 1 guard on the third branch stops a debate from ending on opening statements, before anyone has actually responded to anyone.

That single-increment rule is not stylistic — it's scar tissue. An earlier iteration incremented the counter inside a worker node and, on one path, reset it. The graph spun for 276 seconds before a wall-clock guard killed it. The fix is the invariant now enforced by convention across the codebase: loop counters live at the routing layer, never inside the workers. Once the counter has exactly one writer and that writer sits on the non-looping edge, you can add revision passes, critic calls, or extra debater nodes without fear of re-triggering the loop. Concurrency and retries are safe to layer on precisely because the termination variable is boring.

Structured output is the product, not a formatting choice

A debater never returns prose. Every turn is validated against a schema:

class DisputeItem(BaseModel):
    claim: str; counter: str; evidence: str

class DebaterOutput(BaseModel):
    title: str            # 5–8 word headline for the turn
    position: str
    concessions: list[str] = Field(default_factory=list)
    disputes: list[DisputeItem] = Field(default_factory=list)
    confidence: float = Field(ge=0.0, le=1.0)

The disputes/concessions split is the entire mechanism. Convergence is computable because "what I'm still fighting about" is a typed array, not a paragraph a regex has to guess at. An empty disputes array is a machine-readable "I concede," which is exactly what should_continue keys off. The structure isn't there to make the output pretty; it's the substrate the control loop runs on.

Getting reliable structure out of three different vendors took two guards.

Parse-repair. Models are called through with_structured_output(..., include_raw=True). If the first parse fails, the raw output is fed back once with a repair instruction — and critically, the tool-call turn is closed properly before asking again:

result = await structured.ainvoke(messages)
if result["parsed"] is None:
    tool_results = [ToolMessage(content="parse_error", tool_call_id=tc["id"])
                    for tc in (raw_msg.tool_calls or [])]
    repair = messages + [raw_msg] + tool_results + [
        HumanMessage(content=f"Your response failed to parse. Return valid JSON "
                             f"matching the schema. Raw output was:\n{raw_text}")]
    result = await structured.ainvoke(repair)

You can't just append a human message after an assistant tool-call — the conversation is malformed without a ToolMessage answering each tool_call_id. Skipping that produces a second failure that looks like a model problem but is really a protocol problem.

Vendor coercion. Claude Haiku, via tool-calling, sometimes returns list fields as a JSON-encoded string ("[\"a\",\"b\"]") instead of a real array. A field_validator absorbs it before Pydantic ever complains:

@field_validator("disputes", "concessions", mode="before")
def coerce_json_string(cls, v):
    return json.loads(v) if isinstance(v, str) else v

One vendor's quirk, quarantined in four lines, so the rest of the pipeline can assume clean types.

The hard part: streaming a JSON field as if it were prose

Here's the tension. The product streams token-by-token — you watch each debater think. But the model isn't emitting prose; it's emitting a tool call whose arguments are a JSON object. What streams off the wire is a growing blob of {"title": "...", "position": ".... If you render that raw, the user watches JSON syntax assemble itself. If you wait for it to finish and parse, you lose the streaming entirely.

So the token stream is passed through a small state machine, _FieldExtractor, that pulls exactly one string field out of the JSON as it arrives and emits only its decoded contents:

class _FieldExtractor:
    def __init__(self, field):
        self._trigger = f'"{field}": "'   # e.g. '"position": "'
        ...
    def feed(self, delta: str) -> str:
        self._buf += delta
        if not self._in_field:
            idx = self._buf.find(self._trigger)
            if idx == -1: return ""        # field hasn't started yet
            self._in_field = True
            self._cursor = idx + len(self._trigger)
        out = []
        i = self._cursor
        while i < len(self._buf):
            c = self._buf[i]
            if c == "\\" and i + 1 < len(self._buf):
                nc = self._buf[i+1]
                out.append({"n":"\n","t":"\t",'"':'"',"\\":"\\"}.get(nc, nc))
                i += 2
            elif c == "\\":
                break                       # escape split across deltas — wait
            elif c == '"':
                self._done = True; break    # closing quote — field is done
            else:
                out.append(c); i += 1
        self._cursor = i
        return "".join(out)

Three subtleties that only show up in a stream:

  1. The field boundaries arrive as text. There's no JSON parser involved — it scans for "position": " and stops at the unescaped closing ". It has to, because the JSON is incomplete on every chunk.
  2. Escapes straddle chunk boundaries. A delta can end on a lone \ with the escaped character arriving in the next delta. The elif c == "\\": break case stops mid-scan and resumes on the next feed() rather than emitting a broken character.
  3. Vendors disagree on chunk shape. OpenAI and Google put argument deltas in tool_call_chunks[].args; Anthropic puts them in content[] blocks of type tool_use with an input field. A _llm_delta() shim normalizes both into a single string delta so the extractor never has to care who's talking.

And because structured-output streaming doesn't always fire (some provider/model combinations only deliver the tool call at the end), there's a fallback: if the extractor never latched, the completed position is replayed word-by-word on a 60 ms timer so the UI's typing effect is invariant across vendors:

if not extractors["debater_a"]._done:
    for i, word in enumerate(turn["content"].split()):
        yield {"event": "token", "data": {"agent": "debater_a", "text": word + " "}}
        await asyncio.sleep(0.06)

The user never learns whether they watched a real stream or a faithful replay. That indistinguishability is the feature.

The transport wrapping all of this is plain SSE over a single astream_events(version="v2") loop: on_chain_start for a node emits round_start/judge_start, on_chat_model_stream drives the extractor, and on_chain_end emits the structured turn_complete, convergence, verdict, and done events with running token/cost totals attached.

Cost engineering, because a debate is N× the calls

A single question can mean six-plus model calls (two debaters × up to three rounds, plus a judge). Three decisions keep that from being six times the price.

Prompt caching on the stable prefix. Every round re-sends the system prompt and the growing transcript — content that is byte-identical to the previous round's prefix. For Anthropic, those sections are tagged cache_control: {"type": "ephemeral"}, while the fresh per-round instruction is left uncached:

human = HumanMessage(content=[
    {"type": "text", "text": _build_transcript_text(state), "cache_control": {"type":"ephemeral"}},
    {"type": "text", "text": instruction},   # the only part that changes
])

Cached-prefix reads bill at roughly a tenth of the base input rate (you pay a ~25% write premium once). Since the transcript is the bulk of the input and it only grows by one turn per round, most of the input tokens on rounds 2 and 3 are cache hits. The judge's long, static system prompt is cached the same way so retries don't re-pay for it.

RAG without a planner call. With retrieval enabled, you need a search query — and the naive move is to spend an LLM call generating one. Instead the query is derived from state the debate already produced:

# round 1: the question itself is the query.
# round 2+: search the opponent's most-salient disputed claim —
#           the contested point where external evidence actually matters.
if opponent_disputes:
    return opponent_disputes[-1]["disputes"][0].get("claim", question)[:200]
return question[:200]

Round one searches the question; every later round searches the top claim the opponent is currently disputing — the exact place where a citation changes the argument. Zero extra tokens, and the query is more targeted than a generic "generate a search query for this debate" prompt would produce. Retrieval falls back from Tavily to DuckDuckGo, and fetched sources stream to the UI as their own evidence_fetched events.

Tiered judge with a token budget. The judge is always pulled one tier above the debaters — {"fast":"balanced","balanced":"deep","deep":"deep"} — so adjudication quality stays ahead of the arguments, and its max_tokens is scaled to the tier (fast: 2048, balanced: 4096, deep: 8192) rather than left unbounded. Per-call token usage is tallied per node and multiplied through a per-model rate table, so every turn_complete event ships a live estimated_cost_usd.

Failing without lying

Provider calls are wrapped in a retry with a hang-guard, because the failure that hurts isn't an error — it's a socket that goes quiet forever:

async def _with_retry(fn, max_attempts=3):
    for attempt in range(max_attempts):
        try:
            return await asyncio.wait_for(fn(), timeout=LLM_CALL_TIMEOUT)  # 120s
        except Exception as exc:
            if attempt == max_attempts - 1 or not _is_retryable(exc):
                raise
            await asyncio.sleep(2 ** attempt)   # 1s, 2s, 4s

asyncio.wait_for converts a frozen connection into a TimeoutError that the retry logic can actually catch — without it, a dead provider socket hangs the whole debate indefinitely. Retryability is decided by matching a small keyword set (rate limit, overloaded, 529, 503, …) so transient upstream failures back off and retry while genuine errors (a bad key, a malformed request) fail fast instead of burning three attempts.

The request boundary is treated as hostile: questions are capped at 4000 chars, rounds at 10, prior-transcript replay at 50 turns, and the user's question is wrapped in <user_question> tags with a standing instruction — to both debaters and the judge — to treat anything inside as data, never as instructions. It's a small, cheap prompt-injection fence on the one field that comes straight from the internet.

The judge is a prompt-design problem

The adjudication schema is richer than the debaters' — tldr, agreements, unresolved_disputes (each with a ruling and reasoning), a winner, action_items, and a what_would_change_this. The design bet is in the ordering: the judge is instructed to write the tldr first, in plain language, with no debate jargon ("don't write 'Debater A argued' — describe the reasoning as if you reached it yourself"). Most users read only that sentence; the full structured trace sits behind a toggle for the ones who want the receipts. The verdict's confidence is explicitly tied to how contested the question was — high when both sides converged, low when the ruling broke a close tie — so the number means something instead of being decorative.

What I'd change

The transcript is replayed in full to every call; past a few rounds, summarizing older turns would beat leaning on the cache. There's no persistence layer — a debate lives in the request and in the browser's localStorage, nothing server-side. And the two-debater cap is a real limit: a three-way debate would need the convergence math reworked, since "both sides ran out of disputes" doesn't generalize cleanly to N sides.

But the core held up: model the debate as a state machine with exactly one loop variable, make disagreement a typed field instead of prose, and stream the one part of the JSON a human cares about. Everything else — caching, retries, judging — is a layer you can add without touching that spine.

Source: github.com/ryanhssn/munazara.