Legal RAG Bench Retrieval Benchmark

Executive Summary

Benchmark Dataset

Queries ask legal questions with one relevant passage ID. The retriever must surface the source passage that supports the answer.

TypeGraph processed the full 4,876-passage corpus. The indexed document count is lower because content deduplication collapsed 218 exact duplicate passages; all gold passages used by the 100 eval questions remained represented.

Methodology

1. Loaded the Isaacus Legal RAG Bench corpus and QA relevance pairs from the public dataset.
2. Seeded a TypeGraph Cloud benchmark bucket with graph extraction disabled and chunking set to 2048 tokens with 256 overlap.
3. Used bucket-level content deduplication, which processed all 4,876 source passages and indexed 4,658 unique-content document chunks.
4. Ran documents-only semantic retrieval. BM25, graph, and recency weights were disabled for this run.
5. Requested 12 retrieval candidates per query and scored the deduplicated top 10 against the source relevance pairs.
6. Calculated cost from TypeGraph metered events: ingest embedding tokens, search embedding tokens, and compute duration at the public metered rates.

Detailed Metrics Overview

Legal RAG Bench Leaderboard Comparison

Official rows come from Isaacus MLEB results.jsonl. The highlighted TypeGraph row is inserted for direct comparison using this page's documents-only semantic run.

Metered Cost

Storage is excluded. Costs use public TypeGraph metered rates: ingest embeddings at $0.12/M tokens, search embeddings at $0.04/M tokens, and compute at $0.52/CPU-hour.

Relevant Code

import { readFile } from 'node:fs/promises'
import { typegraphInit } from '@typegraph-ai/sdk'

type CorpusRow = {
  id: string
  title?: string
  text: string
}

const typegraph = await typegraphInit({
  apiKey: process.env.TYPEGRAPH_API_KEY!,
  tenantId: process.env.TYPEGRAPH_TENANT_ID!,
})

const corpus = JSON.parse(
  await readFile('./data/legal-rag-bench/corpus.json', 'utf8'),
) as CorpusRow[]

const bucket = await typegraph.bucket.create({
  name: 'legal-rag-bench',
  indexDefaults: {
    chunkSize: 2048,
    chunkOverlap: 256,
    graphExtraction: false,
    deduplicateBy: ['content'],
  },
})

await typegraph.document.ingest(
  corpus.map((row) => ({
    id: row.id,
    name: row.title ?? row.id,
    content: row.text,
    metadata: { corpusId: row.id },
  })),
  { bucketId: bucket.id },
)

const response = await typegraph.search(query.question, {
  buckets: [process.env.TYPEGRAPH_BUCKET_ID!],
  resources: ['documents'],
  weights: { semantic: 1, bm25: false, graph: false, recency: false },
  limit: 12,
})

const retrievedCorpusIds = response.results.chunks.map((chunk) =>
  String(chunk.metadata?.corpusId ?? chunk.document.name),
)

const ingestCost =
  (embedIngestTokens / 1_000_000) * 0.12 +
  (ingestComputeMs / 3_600_000) * 0.52

const queryCost =
  (embedQueryTokens / 1_000_000) * 0.04 +
  (queryComputeMs / 3_600_000) * 0.52

References

Related TypeGraph Reading