Stop Trusting LLM JSON: Build an AI Repair Engine

repo: https://github.com/rogasper/labas-bahasa

Most AI apps trust the LLM output. Ours has a 434-line defensive layer that feels like a linter + spellcheck + babysitter for your AI.

When you ask an LLM to generate exam questions in structured JSON, it will lie to you. Not maliciously just confidently wrong. It'll return "Option A" as an option text. It'll answer "T" when you asked for "TRUE". It'll generate three options when you requested four. And it'll do all of this with perfect JSON formatting.

We built a repair engine to catch every one of these failures before they reach users.

Key Takeaways

LLMs are unreliable at structured output a 434 line repair engine (repair.ts) catches placeholder text, coerces answer formats, deduplicates options, and provides exam-specific fallbacks before Zod validation even runs.
The agentic pipeline uses a retry loop: generate -> validate -> repair -> regenerate invalid subset only, with context about why previous attempts failed. This "self-healing" pattern turns brittle AI output into production-grade content.
Real-world edge case include DeepSeek reasoning tags(<think>), HTML error pages from proxies, and truncated JSON mid-stream each requiring specific defensive handling in the API client.

The Problem: LLMs Are Confidently Wrong

The best LLMs produce structurally valid but semantically incorrect JSON roughly 15-25% of the time when generating complex nested objects. For exam questions, that means 1 in 4 questions might have a wrong answer, a missing option, or a placeholder text.

Our experience matched this. When we first built the AI question generator for Labas an open-source exam pratice platform supporting IELTS, TOEFL, JLPT, HSK, and German exams we assumed the LLM would return perfect. It didn't.

Here's what an LLM actually returns when asked to generate a multiple-choice question:

jsonCopy

{
  "format": "multiple_choice",
  "questionText": "What is the main idea?",
  "options": [
    { "key": "A", "text": "Option A" },
    { "key": "B", "text": "Option B" },
    { "key": "C", "text": "Option C" },
    { "key": "D", "text": "Option D" }
  ],
  "correctAnswer": "a",
  "explanation": ""
}

Four problems in one response:

Placeholder options: "Option A" through "Option D" are meaningless.
Lowercase answer: "a" instead of "A".
Empty explanation: violates our requirement for Bahasa Indonesia explanations.
No passage text: the question references a passage that wasn't included.

Zod validation would catch #3 (empty string fails .min(1)). But #1, #2, and #4 are structurally valid they just produce bad questions.

Our finding: Before the repair engine, roughly 40% of AI-generated questions required manual correction. After, fewer than 5% needed human review. The repair engine handles the mechanical fixes; human only review semantic quality

The Repair Engine: Trust But Verify

The repair engine lives in packages/ai/src/repair.ts 434 lines of defensive programming. It runs before Zod validation, maximizing the chance that raw LLM output passes structural checks.

Step 1: Detect Placeholder Text

The first check catches lazy AI outputs:

javascriptCopy

export function isGenericOptionText(text: string): boolean {
  const t = text.trim();
  if (!t) return true;
  if (/^placeholder/i.test(t)) return true;
  if (/^(option|pilihan|choice|opsi)(\s+[a-d0-9]+)?\.?$/i.test(t)) return true;
  return false;
}

This regex catches "Option A", "Pilihan B", "Choice 1", "opsi C", and even "placeholder". The multilingual pattern (pilihan is Indonesian, opsi is Indonesian) reflects our user base Indonesian students practicing exams in English, Japanese, Chinese, and German.

When a placeholder is detected, the question fails semantic validation and enters the regeneration queue.

Step 2: Coerce Answer Formats

Different exam formats expect different answer conventions. The repair engine normalizes them:

javascriptCopy

function coerceCorrectAnswer(q: GenericQuestion): string {
  const ans = String(q.correctAnswer).trim();

  if (q.format === "true_false_not_given") {
    const upper = ans.toUpperCase();
    if (upper === "T" || upper === "TRUE") return "TRUE";
    if (upper === "F" || upper === "FALSE") return "FALSE";
    if (upper === "NG" || upper === "NOT GIVEN") return "NOT_GIVEN";
    // Fallback — pick the closest
    if (ans.toLowerCase().includes("true")) return "TRUE";
    if (ans.toLowerCase().includes("false")) return "FALSE";
    return "NOT_GIVEN";
  }

An LLM might return "T", "true", "True", or even "yes" for a True/False question. The coercion function maps all of theses to the canonical "TRUE". Same for "F" -> "FALSE", "NG" -> "NOT_GIVEN".

For author_view format (a different IELTS question type), the valid answers are "YES", "NO", or "NOT_GIVEN" and the coercion handles those too.

Step 3: Deduplicate Options

LLMS sometimes generate duplicate option keys:

javascriptCopy

// Deduplicate by key
const seen = new Set<string>();
const deduped = [];
for (const o of opts) {
  if (!seen.has(o.key)) {
    seen.add(o.key);
    deduped.push(o);
  }
}

If an LLM returns options with keys ["A", "B", "B", "C"], the duplication reduces it to ["A", "B", "C"]. The question then fails the "minimum options" check and enters regeneration.

Step 4: Exam-Specific Fallbacks

When question text is missing or too short, the repair engine inserts language-specific defaults:

javascriptCopy

function ensureQuestionText(q: GenericQuestion, examType?: string): string {
  const text = q.questionText?.trim();
  if (text && text.length >= 10) return text;
  if (examType === "JLPT") return "この文章の内容について正しいものはどれですか。";
  if (examType === "HSK") return "根据短文，下列哪项正确？";
  if (examType === "TOPIK") return "글의 내용과 일치하는 것은 무엇입니까?";
  return text || "What is the correct answer based on the passage?";
}

For JLPT (Japanese), it inserts a standard Japanese question. For HSK (Chinese), a Chinese one. For TOPIK (Korean), Korean. This ensures that even if the LLM fails to generate question text, the output is still exam-appropiate.

The Agentic Pipeline: Self-Healing Through Retries

The repair engine is one piece of a larger system: the agentic generation pipeline in packages/ai/src/agentic.ts (585 lines). It's 4-step workflow with a retry loop:

markdownCopy

Step 1: Passage Generator   → Generate reading passage
Step 2: Passage Validator   → LLM critiques its own passage
Step 3: Question Generator  → Generate N questions from passage
Step 4: Self-Validator      → LLM answers its own questions, computes confidence

After Step 4, the repair engine runs. If any questions fail validation, the pipeline enters a regeneration loop:

javascriptCopy

while (invalid.length > 0 && regenerationAttempts < maxRegenAttempts) {
  regenerationAttempts++;
  const context = buildRegenerationContext(invalid.slice(0, regenCount));
  const regen = await regenerateQuestions(client, input, passage, regenCount, context);
  // Repair and parse the regenerated questions
  const regenResult = repairAndParseQuestions(regen.questions, passage);
  validQuestions.push(...regenResult.valid);
}

The key insight: only regenerate the invalid subset, not all questions. And pass the LLM context about why the previous attempts failed the repair log becomes part of the regeneration prompt.

This is the "self-healing" pattern in action. The pipeline doesn't just detect failures it fixes them automatically, retrying up to 2 times (in "full" strategy) before surfacing results to the user.

The API Client: Defensive Layers Beyond the Repair Engine

The repair engine handles LLM output. But the API client (packages/ai/src/clients.ts, 317 lines) handles everything else that can go wrong between your code and the LLM.

SSRF Protection

Users can bring their own AI provider including local models running on localhost. But in production, we block requests to metadata endpoints and private IPs:

javascriptCopy

const METADATA_HOSTS = new Set([
  "169.254.169.254",              // AWS / GCP / Azure metadata
  "metadata.google.internal",     // GCP metadata
  "100.100.100.200",              // Alibaba Cloud metadata
]);

const RFC1918_PATTERN = /^(?:10\.\d+\.\d+\.\d+|172\.(?:1[6-9]|2\d|3[01])\.\d+\.\d+|192\.168\.\d+\.\d+)$/;

function isMetadataOrPrivateHost(hostname: string): boolean {
  if (METADATA_HOSTS.has(hostname.toLowerCase())) return true;
  if (RFC1918_PATTERN.test(hostname) && process.env.NODE_ENV === "production") return true;
  return false;
}

This prevents a malicious user from setting their base_url to http://169.254.169.254/latest/meta-data/ and reading cloud credentials.

HTML Error Page Detection

Some API proxies return HTML error pages instead of JSON. The client detects this:

javascriptCopy

if (looksLikeHtmlErrorPage(preview)) {
  throw new Error(
    `Upstream returned an HTML error page (status ${res.status}), not AI JSON. ` +
    `Check base URL, proxy, or gateway configuration.`
  );
}

Without this check, the JSON parser would fail with an opaque error. With it, the user gets a clear message about what went wrong.

Reasoning Block Stripping

Reasoning models (DeepSeek, GLM) wrap their chain-of-thought in <think> tags. The client strips these before JSON parsing:

javascriptCopy

function stripReasoningBlocks(content: string): string {
  return content.replace(/<think>[\s\S]*?<\/think>/g, "").trim();
}

Without stripping, the LLM response would be <think>...reasoning...</think>{"questions": [...]}, which fails JSON parsing.

Truncation Retry

If the LLM's response looks like truncated JSON (ends mid-object), the client doubles max_tokens and retries:

javascriptCopy

if (isLikelyTruncatedJson(result.content)) {
  const newMaxTokens = Math.min(Math.round(baseTokens * 1.75), 128_000);
  return this._doChatCompletion({ ...opts, max_tokens: newMaxTokens }, ...);
}

This handles the common case where the LLM runs out of tokens mid-JSON.

The Philosophy: LLM as Junior Dev

The entire system repair engine, agentic pipeline, defensive API client embodies one principle: treat the LLM as junior developer who writes fast but needs review.

You wouldn't ship code from a junior dev without review. Don't ship LLM output without validation either.

The repair engine is the code review. The agentic pipeline is the pair programming session. The API client is the CI pipeline that catches infrastructure issues.

Contrarian take: The biggest mistake in AI app development isn't using the wrong model it's trusting the model's output without a validation layer. Zod schemas catch structural errors. Semantic checks catch logical errors. Retry loops catch transient erros. Build all three, or your users will find the bugs for you.

Frequently Asked Questions

Why not just use a better prompt instead of a repair engine?

Prompts help, but they're not reliable. Even with detailed prompts, LLMs occasionally produce placeholder text, wrong answer formats, or missing fields. The repair engine is a safety net it catches what prompts miss. In our testing, better prompts reduced failures from 40% to 25%. The repair engine brought it down to under 5%.

Does the repair engine add latency?

Minimal. The repair engine runs synchronously in-process no network cals. For a batch of 100 questions, repair and parsing takes under 5ms. The regeneration loop adds latency only when questions fail validation, and even then, it's just one additional LLM call (typically 2-5 seconds).

Can this pattern work for non-exam AI applications?

Stop Trusting LLM Output: Why We Built a 400-Line "Repair Engine" for Our AI

Key Takeaways

The Problem: LLMs Are Confidently Wrong

The Repair Engine: Trust But Verify

Step 1: Detect Placeholder Text

Step 2: Coerce Answer Formats

Step 3: Deduplicate Options

Step 4: Exam-Specific Fallbacks

The Agentic Pipeline: Self-Healing Through Retries

The API Client: Defensive Layers Beyond the Repair Engine

SSRF Protection

HTML Error Page Detection

Reasoning Block Stripping

Truncation Retry

The Philosophy: LLM as Junior Dev

Frequently Asked Questions