Slop Oracle #

Slop Oracle is a semantic prose linter for reasoning models. It evaluates a document and each prose block against a reusable style definition, then returns calibrated scores, binary findings, uncertainty, and deterministic rewrite guidance.

The linter does not rewrite text. TypeSafe's Jev model makes narrow semantic judgments. The harness owns paragraph selection, score normalization, weighting, thresholds, prioritization, and output. A reasoning model can revise the original text from the report and run the linter again.

The proof of concept uses TypeSafe's JavaScript SDK with Bun. It follows TypeSafe's recommended pattern: decompose a broad judgment into independent questions and combine the results in code.

What it does #

For one input document, Slop Oracle:

1. Parses a JSON style definition.
2. Splits Markdown into blank-line-delimited blocks while preserving line numbers and fenced code blocks.
3. Sends one TypeSafe request for the complete document.
4. Sends one request for each prose block, with bounded concurrency.
5. Fans all applicable style dimensions and checks out as independent questions within each request.
6. Normalizes rubric scores to 0–100 and computes weighted totals in TypeScript.
7. Emits a machine-readable report with raw probabilities, confidence, exact paragraph text, and prioritized rewrite guidance.

Headings stay in the full-document evaluation and are not treated as paragraphs. Fenced code stays in the full-document evaluation but is skipped at paragraph level unless --include-code is set.

Quick start #

Install dependencies:

bun install

Point the CLI at an API-key file. This avoids putting the key in shell history:

export TYPESAFE_API_KEY_FILE=/absolute/path/to/typesafe.key

Evaluate the deliberately sloppy example:

bun run evaluate \
  --style styles/deslop.json \
  --input examples/sloppy.md \
  --output reports/sloppy.json \
  --pretty

Read from stdin and render a human-facing report:

printf '%s\n' 'The service gracefully unlocks seamless innovation.' | \
  bun run evaluate \
    --style styles/plain-technical-prose.json \
    --input=- \
    --format markdown

The key can also come from TYPESAFE_API_KEY or --api-key-file. Slop Oracle never writes the key to a report.

Included styles #

styles/deslop.json adapts the five review dimensions from Stephen Turner's Deslop skill: directness, rhythm, reader trust, specificity/authenticity, and density. It adds semantic checks for common formulaic patterns.

styles/plain-technical-prose.json adapts philpax's plain technical prose skill. It scores sentence focus, actor clarity, terminology, rhetorical restraint, precision, and structural formatting.

styles/technical-readme.json evaluates project documentation as a usable technical artifact. It scores purpose and scope, task enablement, operational precision, information architecture, audience assumptions, boundaries and status, technical specificity, directness, and local coherence. It asks the evaluator to infer the document's apparent task, so a design proposal can be evaluated on whether it supports evaluation and implementation rather than on whether it contains an installation command. That inference can be wrong in important ways; see The score is not the truth.

These are starting rubrics, not universal measures of good writing. Edit them for the audience and document type, then test them against representative positive and negative samples.

Style definition #

A style contains goals and context, weighted score dimensions, binary semantic checks, and decision thresholds:

{
  "schemaVersion": 1,
  "name": "house-style",
  "description": "Direct technical prose for operators.",
  "goals": ["Name the actor and observable behavior."],
  "audience": "An operator diagnosing a production service.",
  "referenceSamples": [
    {
      "label": "good runbook prose",
      "quality": "positive",
      "text": "The scheduler retries the request twice, then records a permanent failure.",
      "notes": "Named actor, bounded behavior, and observable outcome."
    }
  ],
  "thresholds": {
    "pass": 0.75,
    "review": 0.55,
    "minimumConfidence": 0.5
  },
  "dimensions": [
    {
      "id": "precision",
      "label": "Precision",
      "instruction": "How precisely does the text state behavior and constraints?",
      "guidance": "Name the actor, condition, action, and observable result.",
      "weight": 1.2,
      "scope": "both",
      "levels": [
        { "label": "vague", "description": "The text relies on abstractions." },
        { "label": "mixed", "description": "Some behavior is explicit." },
        { "label": "precise", "description": "Behavior and constraints are explicit." }
      ]
    }
  ],
  "checks": [
    {
      "id": "rhetorical_question",
      "label": "Rhetorical question",
      "instruction": "Does the text ask and immediately answer a rhetorical question?",
      "guidance": "Replace the question and answer with one statement.",
      "scope": "both",
      "flagAt": 0.65
    }
  ]
}

Dimension levels are ordered from worst to best. Jev returns a probability distribution and a probability-weighted position across those levels. The harness divides that position by the maximum level index, applies the configured weight, and computes the aggregate. Confidence remains a separate signal and never modifies the quality score.

Checks use TypeSafe's Noul primitive. flagAt is the probability at which the report treats a pattern as present. A check does not alter the aggregate score; it adds a discrete finding.

scope can be document, paragraph, or both. Each style must contain at least one scoring dimension for the full document and one for paragraphs.

Validate a style without making an API call:

bun run src/cli.ts validate-style --style styles/deslop.json

Corpus evaluation #

The corpus command evaluates a JSON manifest while preserving one report per document and continuing past empty or failed inputs:

{
  "name": "project READMEs",
  "files": [
    "/absolute/path/to/project-a/README.md",
    "/absolute/path/to/project-b/README.md"
  ]
}

bun run corpus \
  --style styles/technical-readme.json \
  --manifest artifacts/readme-corpus/manifest.json \
  --output artifacts/readme-corpus/run-01 \
  --document-concurrency 2 \
  --paragraph-concurrency 4

The output directory receives summary.json, summary.md, and a reports directory. Empty files are classified as missing documentation and consume no API requests. artifacts/ is gitignored because corpus inputs and reports can contain local project material.

Report contract #

JSON is the default because another model or tool is the primary consumer. The main fields are:

• summary: document score, verdict, aggregate confidence, and total flagged checks.
• document: every full-document dimension and check, including raw probability distributions.
• paragraphs: the same data with exact text and source line numbers.
• findings: thresholded, sorted items with stable rule IDs and rewrite guidance from the style.
• usage: request and token totals returned by TypeSafe.

Low confidence creates a finding but does not turn an uncertain judgment into a style failure. Actual score and check violations sort ahead of uncertainty notices.

Use --fail-under 75 to make the linter exit with status 2 when the full-document score is below a CI threshold. Runtime and configuration failures use status 1.

Why separate requests? #

TypeSafe evaluates all questions in one request against the same state. A full document and one paragraph are different states, so Slop Oracle sends one request for each target. Dimensions and checks for a target share a request and run independently.

The paragraph request contains the paragraph, style goals, audience, and reference samples. It does not repeat the full document. The separate document request measures cross-paragraph properties such as repetition, terminology, and structure without adding irrelevant context to each local judgment.

This design also respects Jev's documented limitations: questions stay literal and atomic, and arithmetic remains in ordinary code.

The score is not the truth #

Slop Oracle measures agreement with a rubric. It does not measure universal writing quality, factual correctness, or fidelity to the author's intent unless the rubric and supplied context explicitly represent those properties. A precise answer to the wrong question is still wrong.

A rewrite loop adds a second failure mode. The reasoning model must understand what the artifact is and how it works before it can improve the prose. If that model assigns the artifact the wrong ontology, it can optimize efficiently toward a polished misdescription. A rising score can then hide a semantic regression.

The first real-world corpus run exposed this with CogTweak. CogTweak is an in-context steering hack. It presents an LLM with the idea that environment parameters govern generation, uses commands such as export PROFANITY_LEVEL=0.8 to establish a shared fictional state, and asks the model to read that state back. The command does not need to change an operating-system environment variable. The described interface and the model's commitment to it are the mechanism.

The technical README rubric and rewriting agent misunderstood CogTweak as a conventional software project. The rewrite correctly observed that the repository contained no server implementation, then recast the project as an unimplemented API specification. Its score rose from 32.8% to 77.4%, but the rewrite erased the central mechanism. The oracle rewarded stronger conformance to the wrong interpretation.

Use a rewrite loop defensively:

1. Identify the artifact type and mechanism before writing the rubric. Software, specifications, research notes, performative protocols, and fictional interfaces need different definitions of useful documentation.
2. Record facts and properties that the rewrite must preserve. Include the original text, source material, and representative examples when they are needed to judge fidelity.
3. Score prose quality and source fidelity separately. A style score cannot establish that a rewrite still describes the same thing.
4. Inspect large score improvements. A capable model can satisfy a rubric by removing the unusual property that made the source valuable.
5. Treat the report as evidence for revision, not as approval. The current harness does not prove factual or mechanistic fidelity.

Slop Oracle is a semantic prose linter, not a semantic truth oracle. When the rubric is wrong or the rewriting model misunderstands the task, the result may be polished, hilarious, confusing, or all three.

Limits of the proof of concept #

• The 100,000-character and 80-paragraph defaults are deliberate guardrails, not TypeSafe API limits.
• Markdown parsing is intentionally small. It preserves fenced blocks and source lines but does not build a full syntax tree.
• A score is a repeatable semantic judgment, not an explanation or proof. The rubric's guidance supplies the rewrite instruction.
• The harness does not currently compare a rewrite with source evidence or enforce preservation constraints.
• Thresholds need calibration on real examples. The default values are hypotheses.
• Jev's primary training language is English. Test other languages before relying on the scores.

Development #

Run type checking and the mocked unit suite:

bun run check

The API adapter implements a small OracleClient interface. Tests use a fake oracle, so the suite spends no API credit. Only explicit CLI evaluations call TypeSafe.