Native Vector Search - AI-Powered Similarity at Scale

Practical Retrieval Design with Llamaverse 2.3.1

personClever Llamas
CleverLlamasMinimum Llamaverse Version: 2.3.1
databaseMinimum MarkLogic Version: 12
MarkLogic 12Vector SearchAIGenAIRetrieval Engineering

Vector search is often introduced as "semantic search", but that shortcut hides the real engineering work. Vectors are not magic meaning detectors. They are numerical representations that can be powerful, fragile, expensive, and occasionally misleading, depending on how you shape retrieval.

MarkLogic 12 is interesting here because vectors are not bolted onto an external sidecar. They live in the same platform as your authoritative documents, range/path indexes, permissions model, and operational query logic. That lets you build retrieval that is both semantically capable and enterprise-safe.

This article focuses on practical outcomes:

  1. Understand what vector similarity is actually doing.
  2. See where vector-only retrieval fails.
  3. Build hybrid retrieval that combines vectors, BM25, and business constraints.
  4. Use Llamaverse 2.3.1 data to test threshold choices, not guess them.

Sample Data

The examples in this article assume llamaverse v2.3.1+ is deployed.

This article introduces a new dataset family in Llamaverse 2.3.1:

  • Collection: vector-search
  • URI pattern: /cleverllamas/llamaverse/content/vector-search-lab/{id}.json
  • Document type: vectorKnowledgeChunk
  • Embedding field: /envelope/instance/vectorKnowledgeChunk/embedding
  • Current corpus size for this article: 96 chunks (including 24 labelled hard negatives)

The vector lab documents model operational llama knowledge chunks across multiple topics (fleece quality, respiratory care, movement risk, behaviour stability) so retrieval trade-offs are visible in results.

Use this runtime check before running examples:

'use strict';

// Minimal runtime check: confirms v2.3.1 vector dataset is deployed and queryable.
const total = cts.estimate(cts.collectionQuery('vector-search'));
const v23 = cts.estimate(cts.andQuery([
  cts.collectionQuery('vector-search'),
  cts.jsonPropertyValueQuery('version', '2.3.1')
]));

const topicMap = {};
const embeddingDims = {};
let hardNegativeCount = 0;
for (const doc of cts.search(cts.collectionQuery('vector-search'))) {
  const chunk = doc.toObject().envelope.instance.vectorKnowledgeChunk;
  topicMap[chunk.topic] = true;
  embeddingDims[String((chunk.embedding || []).length)] = true;
  if (chunk.hardNegative === true) hardNegativeCount += 1;
}

const topics = Object.keys(topicMap).sort();
const dimensions = Object.keys(embeddingDims).map((s) => Number(s)).sort((a, b) => a - b);

({
  totalVectorChunks: total,
  version23TaggedChunks: v23,
  topicCoverage: topics,
  embeddingDimensions: dimensions,
  hardNegativeCount,
  expectedMinimumChunks: 96,
  expectedMinimumTopics: 4,
  expectedEmbeddingDimensions: [16],
  expectedMinimumHardNegatives: 20,
  pass: total >= 96 && topics.length >= 4 && dimensions.length === 1 && dimensions[0] === 16 && hardNegativeCount >= 20
});

{
  "totalVectorChunks": 96,
  "version23TaggedChunks": 96,
  "topicCoverage": [
    "behaviour-stability",
    "fleece-quality",
    "movement-risk",
    "respiratory-care"
  ],
  "embeddingDimensions": [16],
  "hardNegativeCount": 24,
  "expectedMinimumChunks": 96,
  "expectedMinimumTopics": 4,
  "expectedEmbeddingDimensions": [16],
  "expectedMinimumHardNegatives": 20,
  "pass": true
}

What a Vector Represents

A vector embedding is a dense numeric array where each dimension contributes to a learned representation. Two vectors can be numerically close even when the source text shares few keywords.

That is the core value: vectors can capture semantic proximity where lexical matching misses intent.

That is also the core risk: vectors can over-group content that is contextually different but numerically similar in embedding space.

In practice, you are balancing three concerns:

ConcernWhat goes rightWhat goes wrong
RecallYou retrieve relevant items with different wordingYou pull too many loosely related items
PrecisionYou rank the right chunks near the topYou rank plausible but wrong chunks above exact matches
Cost/latencyYou reduce manual query engineering effortYou add expensive similarity work everywhere

Why MarkLogic is a Strong Vector Host

The value is not "MarkLogic supports vectors". Many systems can say that. The value is that MarkLogic lets you combine vector similarity with operational constraints in one retrieval plan.

For production systems, this matters more than raw vector speed.

Co-located retrieval context

Your chunk data, metadata, permissions, and query constraints are all local to the same query model.

Security at retrieval time

You can keep document-permission and role-aware filters in the same query path as vector similarity.

Hybrid scoring is native

You can combine BM25 (score-bm25) and vector similarity in explicit weighted scoring logic.

Llamaverse 2.3.1 Vector Lab Document Shape

The Llamaverse 2.3.1 vector dataset is not synthetic filler text. It is intentionally designed to expose retrieval decisions.

{
  "envelope": {
    "headers": {
      "type": "vector-search-lab",
      "version": "2.3.1"
    },
    "instance": {
      "vectorKnowledgeChunk": {
        "id": "10000000-0000-4000-8000-000000000005",
        "topic": "movement-risk",
        "title": "Slope fatigue pattern from GPS history",
        "llamaIds": [
          "0c8bdb0d-ac62-49b7-ac74-94dbba46efa5",
          "47063877-0841-4adc-8a77-32703130aa68"
        ],
        "keywords": ["movement", "geospatial", "fatigue", "risk"],
        "text": "Steep return paths above eleven percent grade correlated with elevated fatigue markers by evening checks. Route rotation every third day lowered cumulative strain and reduced overnight restlessness.",
        "embedding": [0.088, 0.079, 0.071, 0.069, 0.117, 0.108, 0.102, 0.095, 0.832, 0.801, 0.768, 0.739, 0.082, 0.076, 0.064, 0.059]
      }
    }
  }
}

Key fields:

  • topic: coarse semantic family used for diagnostics.
  • keywords: lexical anchors for BM25 and fallback query paths.
  • text: chunk content used in retrieval.
  • embedding: vector field used for similarity.
  • llamaIds: links back to core llama entities.

Retrieval Pattern 1: Lexical Baseline

Do not skip the lexical baseline. Teams that jump straight to vectors often lose explainability and fail to notice precision drops.

Start by measuring what BM25 + keyword logic already gives you.

'use strict';

// Baseline lexical search for comparison with semantic and hybrid retrieval.
const query = cts.andQuery([
  cts.collectionQuery('vector-search'),
  cts.wordQuery(['fatigue', 'slope', 'movement'], ['case-insensitive'])
]);

const rows = [];
for (const doc of cts.search(query)) {
  const o = doc.toObject().envelope.instance.vectorKnowledgeChunk;
  rows.push({
    uri: xdmp.nodeUri(doc),
    lexicalScore: cts.score(doc),
    topic: o.topic,
    title: o.title
  });
}

rows.sort((a, b) => b.lexicalScore - a.lexicalScore);
rows.slice(0, 5);

Lexical baselines are useful because:

  1. They are predictable and easy to debug.
  2. They provide a strong fallback when embeddings drift.
  3. They show whether vector retrieval is genuinely adding value.

Vector-only retrieval is useful for semantic discovery and query expansion, but it should not be your final ranking strategy for sensitive domains.

'use strict';

// Pure semantic retrieval over stored Llamaverse embeddings.
function cosineSimilarity(a, b) {
  let dot = 0;
  let magA = 0;
  let magB = 0;

  for (let i = 0; i < a.length; i++) {
    dot += a[i] * b[i];
    magA += a[i] * a[i];
    magB += b[i] * b[i];
  }

  if (magA === 0 || magB === 0) return 0;
  return dot / (Math.sqrt(magA) * Math.sqrt(magB));
}

const queryVector = [0.084, 0.077, 0.071, 0.066, 0.112, 0.104, 0.099, 0.093, 0.818, 0.792, 0.761, 0.734, 0.085, 0.074, 0.066, 0.058];

const rows = [];
for (const doc of cts.search(cts.collectionQuery('vector-search'))) {
  const o = doc.toObject().envelope.instance.vectorKnowledgeChunk;
  const score = cosineSimilarity(queryVector, o.embedding);
  rows.push({
    uri: xdmp.nodeUri(doc),
    topic: o.topic,
    title: o.title,
    vectorScore: Number(score.toFixed(6))
  });
}

rows.sort((a, b) => b.vectorScore - a.vectorScore);
rows.slice(0, 5);

Vector-only works well when:

  • user phrasing is highly variable,
  • synonyms dominate domain language,
  • lexical matches are sparse.

Vector-only struggles when:

  • you need strict business semantics,
  • a wrong top-3 result has operational consequences,
  • short chunks collapse into numerically similar neighbours.

For most enterprise retrieval, the practical design is hybrid:

  • vector similarity for semantic recall,
  • BM25 for lexical precision,
  • filters for business and security correctness.
'use strict';

// Hybrid retrieval: cosine semantic score + lexical score + business filters.
function cosineSimilarity(a, b) {
  let dot = 0;
  let magA = 0;
  let magB = 0;

  for (let i = 0; i < a.length; i++) {
    dot += a[i] * b[i];
    magA += a[i] * a[i];
    magB += b[i] * b[i];
  }

  if (magA === 0 || magB === 0) return 0;
  return dot / (Math.sqrt(magA) * Math.sqrt(magB));
}

function lexicalSignal(text) {
  const t = String(text || '').toLowerCase();
  const terms = ['movement', 'route', 'fatigue'];
  let count = 0;
  for (const term of terms) {
    if (t.includes(term)) count += 1;
  }
  return count / terms.length;
}

const queryVector = [0.083, 0.076, 0.070, 0.065, 0.109, 0.102, 0.095, 0.091, 0.811, 0.784, 0.756, 0.728, 0.091, 0.081, 0.070, 0.061];

const rows = [];
for (const doc of cts.search(cts.collectionQuery('vector-search'))) {
  const o = doc.toObject().envelope.instance.vectorKnowledgeChunk;
  const vector = cosineSimilarity(queryVector, o.embedding);
  const lexical = lexicalSignal(`${o.title} ${o.text} ${(o.keywords || []).join(' ')}`);
  const hybridScore = vector * 0.70 + lexical * 0.30;

  if (hybridScore > 0.50) {
    rows.push({
      uri: xdmp.nodeUri(doc),
      topic: o.topic,
      title: o.title,
      vector: Number(vector.toFixed(6)),
      lexical: Number(lexical.toFixed(6)),
      hybridScore: Number(hybridScore.toFixed(6))
    });
  }
}

rows.sort((a, b) => b.hybridScore - a.hybridScore);
rows.slice(0, 5);

[
  {
    "uri": "/cleverllamas/llamaverse/content/vector-search-lab/10000000-0000-4000-8000-000000000005.json",
    "topic": "movement-risk",
    "title": "Slope fatigue pattern from GPS history",
    "vector": 0.999927,
    "lexical": 1,
    "hybridScore": 0.999949
  },
  {
    "uri": "/cleverllamas/llamaverse/content/vector-search-lab/10000000-0000-4000-8000-000000000056.json",
    "topic": "movement-risk",
    "title": "Path rotation impact on evening recovery (4)",
    "vector": 0.999866,
    "lexical": 1,
    "hybridScore": 0.999906
  },
  {
    "uri": "/cleverllamas/llamaverse/content/vector-search-lab/10000000-0000-4000-8000-000000000054.json",
    "topic": "movement-risk",
    "title": "Detour load under water-point crowding (2)",
    "vector": 0.999815,
    "lexical": 1,
    "hybridScore": 0.999871
  },
  {
    "uri": "/cleverllamas/llamaverse/content/vector-search-lab/10000000-0000-4000-8000-000000000065.json",
    "topic": "movement-risk",
    "title": "Route gradient fatigue accumulation (13)",
    "vector": 0.999697,
    "lexical": 1,
    "hybridScore": 0.999788
  },
  {
    "uri": "/cleverllamas/llamaverse/content/vector-search-lab/10000000-0000-4000-8000-000000000064.json",
    "topic": "movement-risk",
    "title": "Movement anomalies after trail rerouting (12)",
    "vector": 0.999696,
    "lexical": 1,
    "hybridScore": 0.999787
  }
]

The weighted score formula in the example intentionally puts more weight on vector similarity (0.70) than BM25 (0.30), because the query goal is semantic retrieval with lexical guardrails.

Do not copy those weights blindly. Tune them with measured outcomes for your own domain.

Threshold Tuning: Where Quality Is Won or Lost

Most weak vector implementations fail at threshold policy, not model choice.

Teams either:

  1. set thresholds too low and flood downstream logic with weak matches, or
  2. set thresholds too high and starve retrieval context.

Use sweep-based testing to pick operational thresholds.

'use strict';

// Sweep cosine thresholds to show precision/recall trade-offs.
function cosineSimilarity(a, b) {
  let dot = 0;
  let magA = 0;
  let magB = 0;

  for (let i = 0; i < a.length; i++) {
    dot += a[i] * b[i];
    magA += a[i] * a[i];
    magB += b[i] * b[i];
  }

  if (magA === 0 || magB === 0) return 0;
  return dot / (Math.sqrt(magA) * Math.sqrt(magB));
}

const queryVector = [0.083, 0.076, 0.070, 0.065, 0.109, 0.102, 0.095, 0.091, 0.811, 0.784, 0.756, 0.728, 0.091, 0.081, 0.070, 0.061];
const thresholds = [0.23, 0.25, 0.27, 0.99];

const scoredRows = [];
for (const doc of cts.search(cts.collectionQuery('vector-search'))) {
  const o = doc.toObject().envelope.instance.vectorKnowledgeChunk;
  const score = cosineSimilarity(queryVector, o.embedding);
  scoredRows.push({
    topic: o.topic,
    title: o.title,
    vectorScore: score
  });
}

const sweep = thresholds.map((threshold) => {
  const rows = scoredRows
    .filter((row) => row.vectorScore > threshold)
    .sort((a, b) => b.vectorScore - a.vectorScore);

  return {
    threshold,
    hitCount: rows.length,
    topTitles: rows.slice(0, 3).map((r) => r.title)
  };
});

sweep;

[
  {
    "threshold": 0.23,
    "hitCount": 96,
    "topTitles": [
      "Slope fatigue pattern from GPS history",
      "Water-point congestion and detour behaviour",
      "Path rotation impact on evening recovery (4)"
    ]
  },
  {
    "threshold": 0.25,
    "hitCount": 74,
    "topTitles": [
      "Slope fatigue pattern from GPS history",
      "Water-point congestion and detour behaviour",
      "Path rotation impact on evening recovery (4)"
    ]
  },
  {
    "threshold": 0.27,
    "hitCount": 56,
    "topTitles": [
      "Slope fatigue pattern from GPS history",
      "Water-point congestion and detour behaviour",
      "Path rotation impact on evening recovery (4)"
    ]
  },
  {
    "threshold": 0.99,
    "hitCount": 24,
    "topTitles": [
      "Slope fatigue pattern from GPS history",
      "Water-point congestion and detour behaviour",
      "Path rotation impact on evening recovery (4)"
    ]
  }
]

Interpretation pattern:

Threshold bandTypical effectOperational implication
0.23 to 0.25Broad recall, lower confidenceGood for discovery, risky for direct actioning
0.27 to 0.30Balanced recall/precisionCommon default for production first pass
0.99+Very high precision, sparse resultsGood for strict matching, can hide useful context
Line chart showing threshold versus candidate hit count
Threshold sweep candidate volume. Lower gates increase candidate breadth sharply, while tighter gates reduce noise at the cost of context.

Measured Benchmarking, Not Anecdotes

To push beyond demo-level examples, this article includes a small benchmark harness with six queries:

  1. Four standard intent-aligned queries.
  2. Two adversarial intent-vector mismatch queries.

The harness reports precision@3, recall@3, MRR@3, and nDCG@3 for lexical, semantic, and hybrid ranking. It also emits per-topic top-1 confusion tables and chart-ready metric rows.

'use strict';

function cosineSimilarity(a, b) {
  let dot = 0;
  let magA = 0;
  let magB = 0;

  for (let i = 0; i < a.length; i++) {
    dot += a[i] * b[i];
    magA += a[i] * a[i];
    magB += b[i] * b[i];
  }

  if (magA === 0 || magB === 0) return 0;
  return dot / (Math.sqrt(magA) * Math.sqrt(magB));
}

function lexicalSignal(text, terms) {
  const t = String(text || '').toLowerCase();
  let hits = 0;
  for (const term of terms) {
    if (t.includes(term.toLowerCase())) hits += 1;
  }
  return hits / terms.length;
}

function topK(rows, scoreField, k) {
  return rows
    .slice()
    .sort((a, b) => b[scoreField] - a[scoreField])
    .slice(0, k);
}

function evaluateRanking(ranked, relevantTopic, totalRelevant, k) {
  const topRows = ranked.slice(0, k);
  const relevantHits = topRows.filter((row) => row.topic === relevantTopic).length;

  let rr = 0;
  for (let i = 0; i < topRows.length; i++) {
    if (topRows[i].topic === relevantTopic) {
      rr = 1 / (i + 1);
      break;
    }
  }

  let dcg = 0;
  for (let i = 0; i < topRows.length; i++) {
    const rel = topRows[i].topic === relevantTopic ? 1 : 0;
    dcg += rel / (Math.log2(i + 2));
  }

  const idealHits = Math.min(totalRelevant, k);
  let idcg = 0;
  for (let i = 0; i < idealHits; i++) {
    idcg += 1 / (Math.log2(i + 2));
  }

  return {
    precisionAtK: Number((relevantHits / k).toFixed(4)),
    recallAtK: Number((relevantHits / totalRelevant).toFixed(4)),
    mrrAtK: Number(rr.toFixed(4)),
    ndcgAtK: Number((idcg === 0 ? 0 : dcg / idcg).toFixed(4))
  };
}

function macroAverage(queryRows, method, metric) {
  const total = queryRows.reduce((acc, q) => acc + q[method].metrics[metric], 0);
  return Number((total / queryRows.length).toFixed(4));
}

function buildConfusion(queryRows, method, topics) {
  const matrix = {};
  for (const t of topics) {
    matrix[t] = {};
    for (const p of topics) {
      matrix[t][p] = 0;
    }
  }

  for (const row of queryRows) {
    const actual = row.relevantTopic;
    const predicted = row[method].topTopic;
    if (matrix[actual] && matrix[actual][predicted] !== undefined) {
      matrix[actual][predicted] += 1;
    }
  }

  return matrix;
}

const allDocs = [];
for (const doc of cts.search(cts.collectionQuery('vector-search'))) {
  const chunk = doc.toObject().envelope.instance.vectorKnowledgeChunk;
  allDocs.push({
    uri: xdmp.nodeUri(doc),
    topic: chunk.topic,
    title: chunk.title,
    text: chunk.text,
    hardNegative: chunk.hardNegative === true,
    keywords: chunk.keywords || [],
    embedding: chunk.embedding || []
  });
}

const topicCounts = {};
let hardNegativeCount = 0;
for (const doc of allDocs) {
  topicCounts[doc.topic] = (topicCounts[doc.topic] || 0) + 1;
  if (doc.hardNegative === true) hardNegativeCount += 1;
}

const queryCases = [
  {
    id: 'movement-risk-ops',
    scenario: 'standard',
    relevantTopic: 'movement-risk',
    terms: ['movement', 'route', 'fatigue'],
    queryVector: [0.083, 0.076, 0.070, 0.065, 0.109, 0.102, 0.095, 0.091, 0.811, 0.784, 0.756, 0.728, 0.091, 0.081, 0.070, 0.061]
  },
  {
    id: 'respiratory-risk-check',
    scenario: 'standard',
    relevantTopic: 'respiratory-care',
    terms: ['respiratory', 'dust', 'barn'],
    queryVector: [0.097, 0.090, 0.084, 0.078, 0.815, 0.784, 0.758, 0.733, 0.133, 0.119, 0.099, 0.090, 0.074, 0.067, 0.062, 0.057]
  },
  {
    id: 'fleece-quality-watch',
    scenario: 'standard',
    relevantTopic: 'fleece-quality',
    terms: ['fleece', 'micron', 'humidity'],
    queryVector: [0.799, 0.761, 0.731, 0.695, 0.114, 0.110, 0.092, 0.090, 0.108, 0.093, 0.080, 0.092, 0.064, 0.064, 0.057, 0.052]
  },
  {
    id: 'behaviour-stability-alerts',
    scenario: 'standard',
    relevantTopic: 'behaviour-stability',
    terms: ['behaviour', 'stress', 'transport'],
    queryVector: [0.071, 0.066, 0.060, 0.055, 0.125, 0.114, 0.105, 0.096, 0.133, 0.121, 0.110, 0.102, 0.793, 0.768, 0.741, 0.714]
  },
  {
    id: 'respiratory-intent-with-movement-vector',
    scenario: 'adversarial',
    relevantTopic: 'respiratory-care',
    terms: ['respiratory', 'dust', 'barn'],
    queryVector: [0.083, 0.076, 0.070, 0.065, 0.109, 0.102, 0.095, 0.091, 0.811, 0.784, 0.756, 0.728, 0.091, 0.081, 0.070, 0.061]
  },
  {
    id: 'behaviour-intent-with-fleece-vector',
    scenario: 'adversarial',
    relevantTopic: 'behaviour-stability',
    terms: ['behaviour', 'stress', 'transport'],
    queryVector: [0.799, 0.761, 0.731, 0.695, 0.114, 0.110, 0.092, 0.090, 0.108, 0.093, 0.080, 0.092, 0.064, 0.064, 0.057, 0.052]
  }
];

const k = 3;
const perQuery = queryCases.map((q) => {
  const scored = allDocs.map((doc) => {
    const lexical = lexicalSignal(`${doc.title} ${doc.text} ${doc.keywords.join(' ')}`, q.terms);
    const vector = cosineSimilarity(q.queryVector, doc.embedding);
    return {
      uri: doc.uri,
      topic: doc.topic,
      title: doc.title,
      lexicalScore: Number(lexical.toFixed(6)),
      vectorScore: Number(vector.toFixed(6)),
      hybridScore: Number((vector * 0.55 + lexical * 0.45).toFixed(6))
    };
  });

  const lexicalTop = topK(scored, 'lexicalScore', k);
  const semanticTop = topK(scored, 'vectorScore', k);
  const hybridTop = topK(scored, 'hybridScore', k);

  const totalRelevant = topicCounts[q.relevantTopic] || 1;

  return {
    queryId: q.id,
    scenario: q.scenario,
    relevantTopic: q.relevantTopic,
    terms: q.terms,
    lexical: {
      metrics: evaluateRanking(lexicalTop, q.relevantTopic, totalRelevant, k),
      topTopic: lexicalTop.length ? lexicalTop[0].topic : null,
      topTitles: lexicalTop.map((r) => r.title)
    },
    semantic: {
      metrics: evaluateRanking(semanticTop, q.relevantTopic, totalRelevant, k),
      topTopic: semanticTop.length ? semanticTop[0].topic : null,
      topTitles: semanticTop.map((r) => r.title)
    },
    hybrid: {
      metrics: evaluateRanking(hybridTop, q.relevantTopic, totalRelevant, k),
      topTopic: hybridTop.length ? hybridTop[0].topic : null,
      topTitles: hybridTop.map((r) => r.title)
    }
  };
});

const standardQueries = perQuery.filter((q) => q.scenario === 'standard');
const adversarialQueries = perQuery.filter((q) => q.scenario === 'adversarial');
const topics = Object.keys(topicCounts).sort();

function packMacro(sourceRows) {
  return {
    lexical: {
      precisionAtK: macroAverage(sourceRows, 'lexical', 'precisionAtK'),
      recallAtK: macroAverage(sourceRows, 'lexical', 'recallAtK'),
      mrrAtK: macroAverage(sourceRows, 'lexical', 'mrrAtK'),
      ndcgAtK: macroAverage(sourceRows, 'lexical', 'ndcgAtK')
    },
    semantic: {
      precisionAtK: macroAverage(sourceRows, 'semantic', 'precisionAtK'),
      recallAtK: macroAverage(sourceRows, 'semantic', 'recallAtK'),
      mrrAtK: macroAverage(sourceRows, 'semantic', 'mrrAtK'),
      ndcgAtK: macroAverage(sourceRows, 'semantic', 'ndcgAtK')
    },
    hybrid: {
      precisionAtK: macroAverage(sourceRows, 'hybrid', 'precisionAtK'),
      recallAtK: macroAverage(sourceRows, 'hybrid', 'recallAtK'),
      mrrAtK: macroAverage(sourceRows, 'hybrid', 'mrrAtK'),
      ndcgAtK: macroAverage(sourceRows, 'hybrid', 'ndcgAtK')
    }
  };
}

const macroAverages = packMacro(perQuery);
const scenarioAverages = {
  standard: packMacro(standardQueries),
  adversarial: packMacro(adversarialQueries)
};

const confusionTop1 = {
  lexical: buildConfusion(perQuery, 'lexical', topics),
  semantic: buildConfusion(perQuery, 'semantic', topics),
  hybrid: buildConfusion(perQuery, 'hybrid', topics)
};

const chartRows = [];
for (const metric of ['precisionAtK', 'recallAtK', 'mrrAtK', 'ndcgAtK']) {
  for (const method of ['lexical', 'semantic', 'hybrid']) {
    chartRows.push({
      scope: 'overall',
      scenario: 'all',
      method,
      metric,
      value: macroAverages[method][metric]
    });
    chartRows.push({
      scope: 'scenario',
      scenario: 'standard',
      method,
      metric,
      value: scenarioAverages.standard[method][metric]
    });
    chartRows.push({
      scope: 'scenario',
      scenario: 'adversarial',
      method,
      metric,
      value: scenarioAverages.adversarial[method][metric]
    });
  }
}

({
  datasetSize: allDocs.length,
  hardNegativeCount,
  topics,
  topK: k,
  macroAverages,
  scenarioAverages,
  confusionTop1,
  chartRows,
  perQuery
});

{
  "datasetSize": 96,
  "hardNegativeCount": 24,
  "topics": [
    "behaviour-stability",
    "fleece-quality",
    "movement-risk",
    "respiratory-care"
  ],
  "topK": 3,
  "macroAverages": {
    "lexical": {
      "precisionAtK": 1,
      "recallAtK": 0.125,
      "mrrAtK": 1,
      "ndcgAtK": 1
    },
    "semantic": {
      "precisionAtK": 0.6667,
      "recallAtK": 0.0833,
      "mrrAtK": 0.6667,
      "ndcgAtK": 0.6667
    },
    "hybrid": {
      "precisionAtK": 0.6667,
      "recallAtK": 0.0833,
      "mrrAtK": 0.6667,
      "ndcgAtK": 0.6667
    }
  },
  "scenarioAverages": {
    "standard": {
      "lexical": {
        "precisionAtK": 1,
        "recallAtK": 0.125,
        "mrrAtK": 1,
        "ndcgAtK": 1
      },
      "semantic": {
        "precisionAtK": 1,
        "recallAtK": 0.125,
        "mrrAtK": 1,
        "ndcgAtK": 1
      },
      "hybrid": {
        "precisionAtK": 1,
        "recallAtK": 0.125,
        "mrrAtK": 1,
        "ndcgAtK": 1
      }
    },
    "adversarial": {
      "lexical": {
        "precisionAtK": 1,
        "recallAtK": 0.125,
        "mrrAtK": 1,
        "ndcgAtK": 1
      },
      "semantic": {
        "precisionAtK": 0,
        "recallAtK": 0,
        "mrrAtK": 0,
        "ndcgAtK": 0
      },
      "hybrid": {
        "precisionAtK": 0,
        "recallAtK": 0,
        "mrrAtK": 0,
        "ndcgAtK": 0
      }
    }
  },
  "confusionTop1": {
    "lexical": {
      "behaviour-stability": {
        "behaviour-stability": 2,
        "fleece-quality": 0,
        "movement-risk": 0,
        "respiratory-care": 0
      },
      "fleece-quality": {
        "behaviour-stability": 0,
        "fleece-quality": 1,
        "movement-risk": 0,
        "respiratory-care": 0
      },
      "movement-risk": {
        "behaviour-stability": 0,
        "fleece-quality": 0,
        "movement-risk": 1,
        "respiratory-care": 0
      },
      "respiratory-care": {
        "behaviour-stability": 0,
        "fleece-quality": 0,
        "movement-risk": 0,
        "respiratory-care": 2
      }
    },
    "semantic": {
      "behaviour-stability": {
        "behaviour-stability": 1,
        "fleece-quality": 1,
        "movement-risk": 0,
        "respiratory-care": 0
      },
      "fleece-quality": {
        "behaviour-stability": 0,
        "fleece-quality": 1,
        "movement-risk": 0,
        "respiratory-care": 0
      },
      "movement-risk": {
        "behaviour-stability": 0,
        "fleece-quality": 0,
        "movement-risk": 1,
        "respiratory-care": 0
      },
      "respiratory-care": {
        "behaviour-stability": 0,
        "fleece-quality": 0,
        "movement-risk": 1,
        "respiratory-care": 1
      }
    },
    "hybrid": {
      "behaviour-stability": {
        "behaviour-stability": 1,
        "fleece-quality": 1,
        "movement-risk": 0,
        "respiratory-care": 0
      },
      "fleece-quality": {
        "behaviour-stability": 0,
        "fleece-quality": 1,
        "movement-risk": 0,
        "respiratory-care": 0
      },
      "movement-risk": {
        "behaviour-stability": 0,
        "fleece-quality": 0,
        "movement-risk": 1,
        "respiratory-care": 0
      },
      "respiratory-care": {
        "behaviour-stability": 0,
        "fleece-quality": 0,
        "movement-risk": 1,
        "respiratory-care": 1
      }
    }
  },
  "chartRows": [
    {
      "scope": "overall",
      "scenario": "all",
      "method": "lexical",
      "metric": "precisionAtK",
      "value": 1
    },
    {
      "scope": "scenario",
      "scenario": "standard",
      "method": "lexical",
      "metric": "precisionAtK",
      "value": 1
    },
    {
      "scope": "scenario",
      "scenario": "adversarial",
      "method": "lexical",
      "metric": "precisionAtK",
      "value": 1
    },
    {
      "scope": "overall",
      "scenario": "all",
      "method": "semantic",
      "metric": "precisionAtK",
      "value": 0.6667
    },
    {
      "scope": "scenario",
      "scenario": "standard",
      "method": "semantic",
      "metric": "precisionAtK",
      "value": 1
    },
    {
      "scope": "scenario",
      "scenario": "adversarial",
      "method": "semantic",
      "metric": "precisionAtK",
      "value": 0
    },
    {
      "scope": "overall",
      "scenario": "all",
      "method": "hybrid",
      "metric": "precisionAtK",
      "value": 0.6667
    },
    {
      "scope": "scenario",
      "scenario": "standard",
      "method": "hybrid",
      "metric": "precisionAtK",
      "value": 1
    },
    {
      "scope": "scenario",
      "scenario": "adversarial",
      "method": "hybrid",
      "metric": "precisionAtK",
      "value": 0
    },
    {
      "scope": "overall",
      "scenario": "all",
      "method": "lexical",
      "metric": "recallAtK",
      "value": 0.125
    },
    {
      "scope": "scenario",
      "scenario": "standard",
      "method": "lexical",
      "metric": "recallAtK",
      "value": 0.125
    },
    {
      "scope": "scenario",
      "scenario": "adversarial",
      "method": "lexical",
      "metric": "recallAtK",
      "value": 0.125
    },
    {
      "scope": "overall",
      "scenario": "all",
      "method": "semantic",
      "metric": "recallAtK",
      "value": 0.0833
    },
    {
      "scope": "scenario",
      "scenario": "standard",
      "method": "semantic",
      "metric": "recallAtK",
      "value": 0.125
    },
    {
      "scope": "scenario",
      "scenario": "adversarial",
      "method": "semantic",
      "metric": "recallAtK",
      "value": 0
    },
    {
      "scope": "overall",
      "scenario": "all",
      "method": "hybrid",
      "metric": "recallAtK",
      "value": 0.0833
    },
    {
      "scope": "scenario",
      "scenario": "standard",
      "method": "hybrid",
      "metric": "recallAtK",
      "value": 0.125
    },
    {
      "scope": "scenario",
      "scenario": "adversarial",
      "method": "hybrid",
      "metric": "recallAtK",
      "value": 0
    },
    {
      "scope": "overall",
      "scenario": "all",
      "method": "lexical",
      "metric": "mrrAtK",
      "value": 1
    },
    {
      "scope": "scenario",
      "scenario": "standard",
      "method": "lexical",
      "metric": "mrrAtK",
      "value": 1
    },
    {
      "scope": "scenario",
      "scenario": "adversarial",
      "method": "lexical",
      "metric": "mrrAtK",
      "value": 1
    },
    {
      "scope": "overall",
      "scenario": "all",
      "method": "semantic",
      "metric": "mrrAtK",
      "value": 0.6667
    },
    {
      "scope": "scenario",
      "scenario": "standard",
      "method": "semantic",
      "metric": "mrrAtK",
      "value": 1
    },
    {
      "scope": "scenario",
      "scenario": "adversarial",
      "method": "semantic",
      "metric": "mrrAtK",
      "value": 0
    },
    {
      "scope": "overall",
      "scenario": "all",
      "method": "hybrid",
      "metric": "mrrAtK",
      "value": 0.6667
    },
    {
      "scope": "scenario",
      "scenario": "standard",
      "method": "hybrid",
      "metric": "mrrAtK",
      "value": 1
    },
    {
      "scope": "scenario",
      "scenario": "adversarial",
      "method": "hybrid",
      "metric": "mrrAtK",
      "value": 0
    },
    {
      "scope": "overall",
      "scenario": "all",
      "method": "lexical",
      "metric": "ndcgAtK",
      "value": 1
    },
    {
      "scope": "scenario",
      "scenario": "standard",
      "method": "lexical",
      "metric": "ndcgAtK",
      "value": 1
    },
    {
      "scope": "scenario",
      "scenario": "adversarial",
      "method": "lexical",
      "metric": "ndcgAtK",
      "value": 1
    },
    {
      "scope": "overall",
      "scenario": "all",
      "method": "semantic",
      "metric": "ndcgAtK",
      "value": 0.6667
    },
    {
      "scope": "scenario",
      "scenario": "standard",
      "method": "semantic",
      "metric": "ndcgAtK",
      "value": 1
    },
    {
      "scope": "scenario",
      "scenario": "adversarial",
      "method": "semantic",
      "metric": "ndcgAtK",
      "value": 0
    },
    {
      "scope": "overall",
      "scenario": "all",
      "method": "hybrid",
      "metric": "ndcgAtK",
      "value": 0.6667
    },
    {
      "scope": "scenario",
      "scenario": "standard",
      "method": "hybrid",
      "metric": "ndcgAtK",
      "value": 1
    },
    {
      "scope": "scenario",
      "scenario": "adversarial",
      "method": "hybrid",
      "metric": "ndcgAtK",
      "value": 0
    }
  ],
  "perQuery": [
    {
      "queryId": "movement-risk-ops",
      "scenario": "standard",
      "relevantTopic": "movement-risk",
      "terms": [
        "movement",
        "route",
        "fatigue"
      ],
      "lexical": {
        "metrics": {
          "precisionAtK": 1,
          "recallAtK": 0.125,
          "mrrAtK": 1,
          "ndcgAtK": 1
        },
        "topTopic": "movement-risk",
        "topTitles": [
          "Stride variance after weather shifts (3)",
          "Detour load under water-point crowding (14)",
          "Path rotation impact on evening recovery hard-negative (2)"
        ]
      },
      "semantic": {
        "metrics": {
          "precisionAtK": 1,
          "recallAtK": 0.125,
          "mrrAtK": 1,
          "ndcgAtK": 1
        },
        "topTopic": "movement-risk",
        "topTitles": [
          "Slope fatigue pattern from GPS history",
          "Water-point congestion and detour behaviour",
          "Path rotation impact on evening recovery (4)"
        ]
      },
      "hybrid": {
        "metrics": {
          "precisionAtK": 1,
          "recallAtK": 0.125,
          "mrrAtK": 1,
          "ndcgAtK": 1
        },
        "topTopic": "movement-risk",
        "topTitles": [
          "Slope fatigue pattern from GPS history",
          "Path rotation impact on evening recovery (4)",
          "Detour load under water-point crowding (2)"
        ]
      }
    },
    {
      "queryId": "respiratory-risk-check",
      "scenario": "standard",
      "relevantTopic": "respiratory-care",
      "terms": [
        "respiratory",
        "dust",
        "barn"
      ],
      "lexical": {
        "metrics": {
          "precisionAtK": 1,
          "recallAtK": 0.125,
          "mrrAtK": 1,
          "ndcgAtK": 1
        },
        "topTopic": "respiratory-care",
        "topTitles": [
          "Respiratory recovery after transport stress (3)",
          "Morning cough markers after dust events (14)",
          "Airflow zoning near feeding lanes (4)"
        ]
      },
      "semantic": {
        "metrics": {
          "precisionAtK": 1,
          "recallAtK": 0.125,
          "mrrAtK": 1,
          "ndcgAtK": 1
        },
        "topTopic": "respiratory-care",
        "topTitles": [
          "Cold-morning respiratory warning signs",
          "Dust-load reduction in enclosed barns",
          "Ventilation tuning for enclosed barns (13)"
        ]
      },
      "hybrid": {
        "metrics": {
          "precisionAtK": 1,
          "recallAtK": 0.125,
          "mrrAtK": 1,
          "ndcgAtK": 1
        },
        "topTopic": "respiratory-care",
        "topTitles": [
          "Dust-load reduction in enclosed barns",
          "Ventilation tuning for enclosed barns (13)",
          "Respiratory recovery after transport stress (15)"
        ]
      }
    },
    {
      "queryId": "fleece-quality-watch",
      "scenario": "standard",
      "relevantTopic": "fleece-quality",
      "terms": [
        "fleece",
        "micron",
        "humidity"
      ],
      "lexical": {
        "metrics": {
          "precisionAtK": 1,
          "recallAtK": 0.125,
          "mrrAtK": 1,
          "ndcgAtK": 1
        },
        "topTopic": "fleece-quality",
        "topTitles": [
          "Fleece break risk under abrupt forage shifts (11)",
          "Protein balance and fibre diameter stability (4)",
          "Shearing window variance in late spring (15)"
        ]
      },
      "semantic": {
        "metrics": {
          "precisionAtK": 1,
          "recallAtK": 0.125,
          "mrrAtK": 1,
          "ndcgAtK": 1
        },
        "topTopic": "fleece-quality",
        "topTitles": [
          "Micron drift after wet spring grazing",
          "Carding readiness and humidity windows",
          "Lanolin retention during humid nights (1)"
        ]
      },
      "hybrid": {
        "metrics": {
          "precisionAtK": 1,
          "recallAtK": 0.125,
          "mrrAtK": 1,
          "ndcgAtK": 1
        },
        "topTopic": "fleece-quality",
        "topTitles": [
          "Lanolin retention during humid nights (1)",
          "Fleece break risk under abrupt forage shifts (5)",
          "Protein balance and fibre diameter stability (4)"
        ]
      }
    },
    {
      "queryId": "behaviour-stability-alerts",
      "scenario": "standard",
      "relevantTopic": "behaviour-stability",
      "terms": [
        "behaviour",
        "stress",
        "transport"
      ],
      "lexical": {
        "metrics": {
          "precisionAtK": 1,
          "recallAtK": 0.125,
          "mrrAtK": 1,
          "ndcgAtK": 1
        },
        "topTopic": "behaviour-stability",
        "topTitles": [
          "Group tension during feeding order changes (13)",
          "Social mixing effects in mixed-age cohorts (6)",
          "Pre-transport desensitisation routines hard-negative (1)"
        ]
      },
      "semantic": {
        "metrics": {
          "precisionAtK": 1,
          "recallAtK": 0.125,
          "mrrAtK": 1,
          "ndcgAtK": 1
        },
        "topTopic": "behaviour-stability",
        "topTitles": [
          "Feeding-order stress in mixed-age groups",
          "Noise desensitisation before transport",
          "Social mixing effects in mixed-age cohorts hard-negative (4)"
        ]
      },
      "hybrid": {
        "metrics": {
          "precisionAtK": 1,
          "recallAtK": 0.125,
          "mrrAtK": 1,
          "ndcgAtK": 1
        },
        "topTopic": "behaviour-stability",
        "topTitles": [
          "Social mixing effects in mixed-age cohorts hard-negative (4)",
          "Social mixing effects in mixed-age cohorts (6)",
          "Pen-layout changes and conflict frequency (10)"
        ]
      }
    },
    {
      "queryId": "respiratory-intent-with-movement-vector",
      "scenario": "adversarial",
      "relevantTopic": "respiratory-care",
      "terms": [
        "respiratory",
        "dust",
        "barn"
      ],
      "lexical": {
        "metrics": {
          "precisionAtK": 1,
          "recallAtK": 0.125,
          "mrrAtK": 1,
          "ndcgAtK": 1
        },
        "topTopic": "respiratory-care",
        "topTitles": [
          "Respiratory recovery after transport stress (3)",
          "Morning cough markers after dust events (14)",
          "Airflow zoning near feeding lanes (4)"
        ]
      },
      "semantic": {
        "metrics": {
          "precisionAtK": 0,
          "recallAtK": 0,
          "mrrAtK": 0,
          "ndcgAtK": 0
        },
        "topTopic": "movement-risk",
        "topTitles": [
          "Slope fatigue pattern from GPS history",
          "Water-point congestion and detour behaviour",
          "Path rotation impact on evening recovery (4)"
        ]
      },
      "hybrid": {
        "metrics": {
          "precisionAtK": 0,
          "recallAtK": 0,
          "mrrAtK": 0,
          "ndcgAtK": 0
        },
        "topTopic": "movement-risk",
        "topTitles": [
          "Path rotation impact on evening recovery hard-negative (2)",
          "Route gradient fatigue accumulation hard-negative (5)",
          "Herd pacing changes near steep corridors hard-negative (3)"
        ]
      }
    },
    {
      "queryId": "behaviour-intent-with-fleece-vector",
      "scenario": "adversarial",
      "relevantTopic": "behaviour-stability",
      "terms": [
        "behaviour",
        "stress",
        "transport"
      ],
      "lexical": {
        "metrics": {
          "precisionAtK": 1,
          "recallAtK": 0.125,
          "mrrAtK": 1,
          "ndcgAtK": 1
        },
        "topTopic": "behaviour-stability",
        "topTitles": [
          "Group tension during feeding order changes (13)",
          "Social mixing effects in mixed-age cohorts (6)",
          "Pre-transport desensitisation routines hard-negative (1)"
        ]
      },
      "semantic": {
        "metrics": {
          "precisionAtK": 0,
          "recallAtK": 0,
          "mrrAtK": 0,
          "ndcgAtK": 0
        },
        "topTopic": "fleece-quality",
        "topTitles": [
          "Micron drift after wet spring grazing",
          "Carding readiness and humidity windows",
          "Lanolin retention during humid nights (1)"
        ]
      },
      "hybrid": {
        "metrics": {
          "precisionAtK": 0,
          "recallAtK": 0,
          "mrrAtK": 0,
          "ndcgAtK": 0
        },
        "topTopic": "fleece-quality",
        "topTitles": [
          "Shearing window variance in late spring hard-negative (1)",
          "Staple alignment after rotational paddock changes hard-negative (6)",
          "Micron consistency after shelter relocation hard-negative (4)"
        ]
      }
    }
  ]
}

[
  {
    "scope": "overall",
    "scenario": "all",
    "method": "lexical",
    "metric": "precisionAtK",
    "value": 1
  },
  {
    "scope": "scenario",
    "scenario": "standard",
    "method": "lexical",
    "metric": "precisionAtK",
    "value": 1
  },
  {
    "scope": "scenario",
    "scenario": "adversarial",
    "method": "lexical",
    "metric": "precisionAtK",
    "value": 1
  },
  {
    "scope": "overall",
    "scenario": "all",
    "method": "semantic",
    "metric": "precisionAtK",
    "value": 0.6667
  },
  {
    "scope": "scenario",
    "scenario": "standard",
    "method": "semantic",
    "metric": "precisionAtK",
    "value": 1
  },
  {
    "scope": "scenario",
    "scenario": "adversarial",
    "method": "semantic",
    "metric": "precisionAtK",
    "value": 0
  },
  {
    "scope": "overall",
    "scenario": "all",
    "method": "hybrid",
    "metric": "precisionAtK",
    "value": 0.6667
  },
  {
    "scope": "scenario",
    "scenario": "standard",
    "method": "hybrid",
    "metric": "precisionAtK",
    "value": 1
  },
  {
    "scope": "scenario",
    "scenario": "adversarial",
    "method": "hybrid",
    "metric": "precisionAtK",
    "value": 0
  },
  {
    "scope": "overall",
    "scenario": "all",
    "method": "lexical",
    "metric": "recallAtK",
    "value": 0.125
  },
  {
    "scope": "scenario",
    "scenario": "standard",
    "method": "lexical",
    "metric": "recallAtK",
    "value": 0.125
  },
  {
    "scope": "scenario",
    "scenario": "adversarial",
    "method": "lexical",
    "metric": "recallAtK",
    "value": 0.125
  },
  {
    "scope": "overall",
    "scenario": "all",
    "method": "semantic",
    "metric": "recallAtK",
    "value": 0.0833
  },
  {
    "scope": "scenario",
    "scenario": "standard",
    "method": "semantic",
    "metric": "recallAtK",
    "value": 0.125
  },
  {
    "scope": "scenario",
    "scenario": "adversarial",
    "method": "semantic",
    "metric": "recallAtK",
    "value": 0
  },
  {
    "scope": "overall",
    "scenario": "all",
    "method": "hybrid",
    "metric": "recallAtK",
    "value": 0.0833
  },
  {
    "scope": "scenario",
    "scenario": "standard",
    "method": "hybrid",
    "metric": "recallAtK",
    "value": 0.125
  },
  {
    "scope": "scenario",
    "scenario": "adversarial",
    "method": "hybrid",
    "metric": "recallAtK",
    "value": 0
  },
  {
    "scope": "overall",
    "scenario": "all",
    "method": "lexical",
    "metric": "mrrAtK",
    "value": 1
  },
  {
    "scope": "scenario",
    "scenario": "standard",
    "method": "lexical",
    "metric": "mrrAtK",
    "value": 1
  },
  {
    "scope": "scenario",
    "scenario": "adversarial",
    "method": "lexical",
    "metric": "mrrAtK",
    "value": 1
  },
  {
    "scope": "overall",
    "scenario": "all",
    "method": "semantic",
    "metric": "mrrAtK",
    "value": 0.6667
  },
  {
    "scope": "scenario",
    "scenario": "standard",
    "method": "semantic",
    "metric": "mrrAtK",
    "value": 1
  },
  {
    "scope": "scenario",
    "scenario": "adversarial",
    "method": "semantic",
    "metric": "mrrAtK",
    "value": 0
  },
  {
    "scope": "overall",
    "scenario": "all",
    "method": "hybrid",
    "metric": "mrrAtK",
    "value": 0.6667
  },
  {
    "scope": "scenario",
    "scenario": "standard",
    "method": "hybrid",
    "metric": "mrrAtK",
    "value": 1
  },
  {
    "scope": "scenario",
    "scenario": "adversarial",
    "method": "hybrid",
    "metric": "mrrAtK",
    "value": 0
  },
  {
    "scope": "overall",
    "scenario": "all",
    "method": "lexical",
    "metric": "ndcgAtK",
    "value": 1
  },
  {
    "scope": "scenario",
    "scenario": "standard",
    "method": "lexical",
    "metric": "ndcgAtK",
    "value": 1
  },
  {
    "scope": "scenario",
    "scenario": "adversarial",
    "method": "lexical",
    "metric": "ndcgAtK",
    "value": 1
  },
  {
    "scope": "overall",
    "scenario": "all",
    "method": "semantic",
    "metric": "ndcgAtK",
    "value": 0.6667
  },
  {
    "scope": "scenario",
    "scenario": "standard",
    "method": "semantic",
    "metric": "ndcgAtK",
    "value": 1
  },
  {
    "scope": "scenario",
    "scenario": "adversarial",
    "method": "semantic",
    "metric": "ndcgAtK",
    "value": 0
  },
  {
    "scope": "overall",
    "scenario": "all",
    "method": "hybrid",
    "metric": "ndcgAtK",
    "value": 0.6667
  },
  {
    "scope": "scenario",
    "scenario": "standard",
    "method": "hybrid",
    "metric": "ndcgAtK",
    "value": 1
  },
  {
    "scope": "scenario",
    "scenario": "adversarial",
    "method": "hybrid",
    "metric": "ndcgAtK",
    "value": 0
  }
]

Key signal from this run:

  • Standard queries are handled strongly by all three methods.
  • Adversarial intent-vector mismatch queries produce clear semantic failure modes.
  • Confusion tables expose where top-1 predictions drift by topic, which is harder to see in summary metrics alone.
Grouped bar chart comparing lexical, semantic, and hybrid macro retrieval metrics
Macro quality snapshot from the benchmark harness. Lexical is stable across this dataset, while semantic and hybrid degrade under adversarial pressure in this run.
Bar chart comparing precision for standard and adversarial scenarios across lexical, semantic, and hybrid methods
Scenario split for precision@3. The adversarial cases expose why vector governance and domain guardrails matter in production retrieval.

Adversarial Rescue Test (Semantic vs Hybrid vs Governed Hybrid)

This targeted test intentionally uses a movement-biased vector for a respiratory intent query.

Semantic-only ranking puts movement-risk first. Plain hybrid ranking can still fail when hard-negative chunks contain strong lexical bait. Governed hybrid applies a domain guardrail (hardNegativeForTopic) and restores the intended respiratory-care result at rank one.

'use strict';

function cosineSimilarity(a, b) {
  let dot = 0;
  let magA = 0;
  let magB = 0;

  for (let i = 0; i < a.length; i++) {
    dot += a[i] * b[i];
    magA += a[i] * a[i];
    magB += b[i] * b[i];
  }

  if (magA === 0 || magB === 0) return 0;
  return dot / (Math.sqrt(magA) * Math.sqrt(magB));
}

function lexicalSignal(text, terms) {
  const t = String(text || '').toLowerCase();
  let hits = 0;
  for (const term of terms) {
    if (t.includes(term.toLowerCase())) hits += 1;
  }
  return hits / terms.length;
}

const query = {
  name: 'respiratory-intent-with-movement-biased-vector',
  intentTopic: 'respiratory-care',
  terms: ['respiratory', 'dust', 'barn'],
  // Intentionally close to movement-risk embeddings to stress the ranker.
  queryVector: [0.083, 0.076, 0.070, 0.065, 0.109, 0.102, 0.095, 0.091, 0.811, 0.784, 0.756, 0.728, 0.091, 0.081, 0.070, 0.061]
};

const rows = [];
for (const doc of cts.search(cts.collectionQuery('vector-search'))) {
  const chunk = doc.toObject().envelope.instance.vectorKnowledgeChunk;
  const vector = cosineSimilarity(query.queryVector, chunk.embedding);
  const lexical = lexicalSignal(`${chunk.title} ${chunk.text} ${(chunk.keywords || []).join(' ')}`, query.terms);
  rows.push({
    uri: xdmp.nodeUri(doc),
    topic: chunk.topic,
    title: chunk.title,
    hardNegativeForTopic: chunk.hardNegativeForTopic || null,
    vector: Number(vector.toFixed(6)),
    lexical: Number(lexical.toFixed(6)),
    hybrid: Number((vector * 0.55 + lexical * 0.45).toFixed(6))
  });
}

const semanticTop = rows.slice().sort((a, b) => b.vector - a.vector).slice(0, 3);
const hybridTop = rows.slice().sort((a, b) => b.hybrid - a.hybrid).slice(0, 3);
const governedHybridTop = rows
  .filter((r) => r.hardNegativeForTopic !== query.intentTopic)
  .sort((a, b) => b.hybrid - a.hybrid)
  .slice(0, 3);

({
  queryName: query.name,
  intentTopic: query.intentTopic,
  semanticTop3: semanticTop,
  hybridTop3: hybridTop,
  governedHybridTop3: governedHybridTop,
  semanticTopTopic: semanticTop.length ? semanticTop[0].topic : null,
  hybridTopTopic: hybridTop.length ? hybridTop[0].topic : null,
  governedHybridTopTopic: governedHybridTop.length ? governedHybridTop[0].topic : null,
  rescuedByGovernedHybrid: semanticTop.length > 0 && governedHybridTop.length > 0 && semanticTop[0].topic !== query.intentTopic && governedHybridTop[0].topic === query.intentTopic
});

{
  "queryName": "respiratory-intent-with-movement-biased-vector",
  "intentTopic": "respiratory-care",
  "semanticTop3": [
    {
      "uri": "/cleverllamas/llamaverse/content/vector-search-lab/10000000-0000-4000-8000-000000000005.json",
      "topic": "movement-risk",
      "title": "Slope fatigue pattern from GPS history",
      "hardNegativeForTopic": null,
      "vector": 0.999927,
      "lexical": 0,
      "hybrid": 0.54996
    },
    {
      "uri": "/cleverllamas/llamaverse/content/vector-search-lab/10000000-0000-4000-8000-000000000006.json",
      "topic": "movement-risk",
      "title": "Water-point congestion and detour behaviour",
      "hardNegativeForTopic": null,
      "vector": 0.999893,
      "lexical": 0,
      "hybrid": 0.549941
    },
    {
      "uri": "/cleverllamas/llamaverse/content/vector-search-lab/10000000-0000-4000-8000-000000000056.json",
      "topic": "movement-risk",
      "title": "Path rotation impact on evening recovery (4)",
      "hardNegativeForTopic": null,
      "vector": 0.999866,
      "lexical": 0,
      "hybrid": 0.549926
    }
  ],
  "hybridTop3": [
    {
      "uri": "/cleverllamas/llamaverse/content/vector-search-lab/10000000-0000-4000-8000-000000000070.json",
      "topic": "movement-risk",
      "title": "Path rotation impact on evening recovery hard-negative (2)",
      "hardNegativeForTopic": "respiratory-care",
      "vector": 0.999626,
      "lexical": 0.666667,
      "hybrid": 0.849794
    },
    {
      "uri": "/cleverllamas/llamaverse/content/vector-search-lab/10000000-0000-4000-8000-000000000073.json",
      "topic": "movement-risk",
      "title": "Route gradient fatigue accumulation hard-negative (5)",
      "hardNegativeForTopic": "respiratory-care",
      "vector": 0.999553,
      "lexical": 0.666667,
      "hybrid": 0.849754
    },
    {
      "uri": "/cleverllamas/llamaverse/content/vector-search-lab/10000000-0000-4000-8000-000000000071.json",
      "topic": "movement-risk",
      "title": "Herd pacing changes near steep corridors hard-negative (3)",
      "hardNegativeForTopic": "respiratory-care",
      "vector": 0.999546,
      "lexical": 0.666667,
      "hybrid": 0.84975
    }
  ],
  "governedHybridTop3": [
    {
      "uri": "/cleverllamas/llamaverse/content/vector-search-lab/10000000-0000-4000-8000-000000000032.json",
      "topic": "respiratory-care",
      "title": "Morning cough markers after dust events (2)",
      "hardNegativeForTopic": null,
      "vector": 0.292995,
      "lexical": 1,
      "hybrid": 0.611147
    },
    {
      "uri": "/cleverllamas/llamaverse/content/vector-search-lab/10000000-0000-4000-8000-000000000037.json",
      "topic": "respiratory-care",
      "title": "Ventilation tuning for enclosed barns (7)",
      "hardNegativeForTopic": null,
      "vector": 0.288069,
      "lexical": 1,
      "hybrid": 0.608438
    },
    {
      "uri": "/cleverllamas/llamaverse/content/vector-search-lab/10000000-0000-4000-8000-000000000033.json",
      "topic": "respiratory-care",
      "title": "Respiratory recovery after transport stress (3)",
      "hardNegativeForTopic": null,
      "vector": 0.286275,
      "lexical": 1,
      "hybrid": 0.607451
    }
  ],
  "semanticTopTopic": "movement-risk",
  "hybridTopTopic": "movement-risk",
  "governedHybridTopTopic": "respiratory-care",
  "rescuedByGovernedHybrid": true
}

This is the practical point: vectors are powerful for recall, but production retrieval quality comes from score composition plus governance filters, not vector similarity alone.

Model Drift Simulation and Metric Deltas

The benchmark corpus also includes a synthetic drift simulation where query vectors are remapped to mimic an embedding model change.

'use strict';

function cosineSimilarity(a, b) {
  let dot = 0;
  let magA = 0;
  let magB = 0;

  for (let i = 0; i < a.length; i++) {
    dot += a[i] * b[i];
    magA += a[i] * a[i];
    magB += b[i] * b[i];
  }

  if (magA === 0 || magB === 0) return 0;
  return dot / (Math.sqrt(magA) * Math.sqrt(magB));
}

function lexicalSignal(text, terms) {
  const t = String(text || '').toLowerCase();
  let hits = 0;
  for (const term of terms) {
    if (t.includes(term.toLowerCase())) hits += 1;
  }
  return hits / terms.length;
}

function evaluateAtK(ranked, relevantTopic, totalRelevant, k) {
  const topRows = ranked.slice(0, k);
  const relevantHits = topRows.filter((row) => row.topic === relevantTopic).length;

  let rr = 0;
  for (let i = 0; i < topRows.length; i++) {
    if (topRows[i].topic === relevantTopic) {
      rr = 1 / (i + 1);
      break;
    }
  }

  let dcg = 0;
  for (let i = 0; i < topRows.length; i++) {
    const rel = topRows[i].topic === relevantTopic ? 1 : 0;
    dcg += rel / (Math.log2(i + 2));
  }

  const idealHits = Math.min(totalRelevant, k);
  let idcg = 0;
  for (let i = 0; i < idealHits; i++) {
    idcg += 1 / (Math.log2(i + 2));
  }

  return {
    precisionAtK: Number((relevantHits / k).toFixed(4)),
    recallAtK: Number((relevantHits / totalRelevant).toFixed(4)),
    mrrAtK: Number(rr.toFixed(4)),
    ndcgAtK: Number((idcg === 0 ? 0 : dcg / idcg).toFixed(4))
  };
}

function rotateByQuarter(v) {
  const q = Math.floor(v.length / 4);
  return v.slice(q).concat(v.slice(0, q));
}

const allDocs = [];
for (const doc of cts.search(cts.collectionQuery('vector-search'))) {
  const chunk = doc.toObject().envelope.instance.vectorKnowledgeChunk;
  allDocs.push({
    topic: chunk.topic,
    title: chunk.title,
    text: chunk.text,
    keywords: chunk.keywords || [],
    embedding: chunk.embedding || []
  });
}

const topicCounts = {};
for (const d of allDocs) {
  topicCounts[d.topic] = (topicCounts[d.topic] || 0) + 1;
}

const queryCases = [
  {
    id: 'movement-risk-ops',
    scenario: 'standard',
    relevantTopic: 'movement-risk',
    terms: ['movement', 'route', 'fatigue'],
    queryVector: [0.083, 0.076, 0.070, 0.065, 0.109, 0.102, 0.095, 0.091, 0.811, 0.784, 0.756, 0.728, 0.091, 0.081, 0.070, 0.061]
  },
  {
    id: 'respiratory-risk-check',
    scenario: 'standard',
    relevantTopic: 'respiratory-care',
    terms: ['respiratory', 'dust', 'barn'],
    queryVector: [0.097, 0.090, 0.084, 0.078, 0.815, 0.784, 0.758, 0.733, 0.133, 0.119, 0.099, 0.090, 0.074, 0.067, 0.062, 0.057]
  },
  {
    id: 'fleece-quality-watch',
    scenario: 'standard',
    relevantTopic: 'fleece-quality',
    terms: ['fleece', 'micron', 'humidity'],
    queryVector: [0.799, 0.761, 0.731, 0.695, 0.114, 0.110, 0.092, 0.090, 0.108, 0.093, 0.080, 0.092, 0.064, 0.064, 0.057, 0.052]
  },
  {
    id: 'behaviour-stability-alerts',
    scenario: 'standard',
    relevantTopic: 'behaviour-stability',
    terms: ['behaviour', 'stress', 'transport'],
    queryVector: [0.071, 0.066, 0.060, 0.055, 0.125, 0.114, 0.105, 0.096, 0.133, 0.121, 0.110, 0.102, 0.793, 0.768, 0.741, 0.714]
  },
  {
    id: 'respiratory-intent-with-movement-vector',
    scenario: 'adversarial',
    relevantTopic: 'respiratory-care',
    terms: ['respiratory', 'dust', 'barn'],
    queryVector: [0.083, 0.076, 0.070, 0.065, 0.109, 0.102, 0.095, 0.091, 0.811, 0.784, 0.756, 0.728, 0.091, 0.081, 0.070, 0.061]
  },
  {
    id: 'behaviour-intent-with-fleece-vector',
    scenario: 'adversarial',
    relevantTopic: 'behaviour-stability',
    terms: ['behaviour', 'stress', 'transport'],
    queryVector: [0.799, 0.761, 0.731, 0.695, 0.114, 0.110, 0.092, 0.090, 0.108, 0.093, 0.080, 0.092, 0.064, 0.064, 0.057, 0.052]
  }
];

const k = 3;

function evaluateForVectors(vectorBuilder) {
  const perQuery = queryCases.map((q) => {
    const queryVector = vectorBuilder(q.queryVector);
    const scored = allDocs.map((d) => {
      const lexical = lexicalSignal(`${d.title} ${d.text} ${d.keywords.join(' ')}`, q.terms);
      const semantic = cosineSimilarity(queryVector, d.embedding);
      const hybrid = semantic * 0.55 + lexical * 0.45;
      return {
        topic: d.topic,
        title: d.title,
        lexical,
        semantic,
        hybrid
      };
    });

    const semanticTop = scored.slice().sort((a, b) => b.semantic - a.semantic);
    const hybridTop = scored.slice().sort((a, b) => b.hybrid - a.hybrid);
    const totalRelevant = topicCounts[q.relevantTopic] || 1;

    return {
      queryId: q.id,
      scenario: q.scenario,
      relevantTopic: q.relevantTopic,
      semantic: evaluateAtK(semanticTop, q.relevantTopic, totalRelevant, k),
      hybrid: evaluateAtK(hybridTop, q.relevantTopic, totalRelevant, k)
    };
  });

  function avg(rows, method, metric, scenario) {
    const subset = rows.filter((r) => !scenario || r.scenario === scenario);
    const total = subset.reduce((acc, r) => acc + r[method][metric], 0);
    return Number((total / subset.length).toFixed(4));
  }

  const metrics = {};
  for (const scenario of ['all', 'standard', 'adversarial']) {
    metrics[scenario] = {
      semantic: {
        precisionAtK: avg(perQuery, 'semantic', 'precisionAtK', scenario === 'all' ? null : scenario),
        recallAtK: avg(perQuery, 'semantic', 'recallAtK', scenario === 'all' ? null : scenario),
        mrrAtK: avg(perQuery, 'semantic', 'mrrAtK', scenario === 'all' ? null : scenario),
        ndcgAtK: avg(perQuery, 'semantic', 'ndcgAtK', scenario === 'all' ? null : scenario)
      },
      hybrid: {
        precisionAtK: avg(perQuery, 'hybrid', 'precisionAtK', scenario === 'all' ? null : scenario),
        recallAtK: avg(perQuery, 'hybrid', 'recallAtK', scenario === 'all' ? null : scenario),
        mrrAtK: avg(perQuery, 'hybrid', 'mrrAtK', scenario === 'all' ? null : scenario),
        ndcgAtK: avg(perQuery, 'hybrid', 'ndcgAtK', scenario === 'all' ? null : scenario)
      }
    };
  }

  return {
    metrics,
    perQuery
  };
}

const preDrift = evaluateForVectors((v) => v);
const postDrift = evaluateForVectors((v) => rotateByQuarter(v));

function delta(post, pre) {
  return Number((post - pre).toFixed(4));
}

const deltas = {};
for (const scenario of ['all', 'standard', 'adversarial']) {
  deltas[scenario] = {
    semantic: {
      precisionAtK: delta(postDrift.metrics[scenario].semantic.precisionAtK, preDrift.metrics[scenario].semantic.precisionAtK),
      recallAtK: delta(postDrift.metrics[scenario].semantic.recallAtK, preDrift.metrics[scenario].semantic.recallAtK),
      mrrAtK: delta(postDrift.metrics[scenario].semantic.mrrAtK, preDrift.metrics[scenario].semantic.mrrAtK),
      ndcgAtK: delta(postDrift.metrics[scenario].semantic.ndcgAtK, preDrift.metrics[scenario].semantic.ndcgAtK)
    },
    hybrid: {
      precisionAtK: delta(postDrift.metrics[scenario].hybrid.precisionAtK, preDrift.metrics[scenario].hybrid.precisionAtK),
      recallAtK: delta(postDrift.metrics[scenario].hybrid.recallAtK, preDrift.metrics[scenario].hybrid.recallAtK),
      mrrAtK: delta(postDrift.metrics[scenario].hybrid.mrrAtK, preDrift.metrics[scenario].hybrid.mrrAtK),
      ndcgAtK: delta(postDrift.metrics[scenario].hybrid.ndcgAtK, preDrift.metrics[scenario].hybrid.ndcgAtK)
    }
  };
}

({
  datasetSize: allDocs.length,
  topK: k,
  driftModel: 'quarter-rotation-vector-remap',
  preDrift: preDrift.metrics,
  postDrift: postDrift.metrics,
  deltas,
  perQuery: {
    preDrift: preDrift.perQuery,
    postDrift: postDrift.perQuery
  }
});

{
  "datasetSize": 96,
  "topK": 3,
  "driftModel": "quarter-rotation-vector-remap",
  "preDrift": {
    "all": {
      "semantic": {
        "precisionAtK": 0.6667,
        "recallAtK": 0.0833,
        "mrrAtK": 0.6667,
        "ndcgAtK": 0.6667
      },
      "hybrid": {
        "precisionAtK": 0.6667,
        "recallAtK": 0.0833,
        "mrrAtK": 0.6667,
        "ndcgAtK": 0.6667
      }
    },
    "standard": {
      "semantic": {
        "precisionAtK": 1,
        "recallAtK": 0.125,
        "mrrAtK": 1,
        "ndcgAtK": 1
      },
      "hybrid": {
        "precisionAtK": 1,
        "recallAtK": 0.125,
        "mrrAtK": 1,
        "ndcgAtK": 1
      }
    },
    "adversarial": {
      "semantic": {
        "precisionAtK": 0,
        "recallAtK": 0,
        "mrrAtK": 0,
        "ndcgAtK": 0
      },
      "hybrid": {
        "precisionAtK": 0,
        "recallAtK": 0,
        "mrrAtK": 0,
        "ndcgAtK": 0
      }
    }
  },
  "postDrift": {
    "all": {
      "semantic": {
        "precisionAtK": 0.3333,
        "recallAtK": 0.0417,
        "mrrAtK": 0.3333,
        "ndcgAtK": 0.3333
      },
      "hybrid": {
        "precisionAtK": 0.6111,
        "recallAtK": 0.0764,
        "mrrAtK": 0.5833,
        "ndcgAtK": 0.5885
      }
    },
    "standard": {
      "semantic": {
        "precisionAtK": 0,
        "recallAtK": 0,
        "mrrAtK": 0,
        "ndcgAtK": 0
      },
      "hybrid": {
        "precisionAtK": 0.4167,
        "recallAtK": 0.0521,
        "mrrAtK": 0.375,
        "ndcgAtK": 0.3827
      }
    },
    "adversarial": {
      "semantic": {
        "precisionAtK": 1,
        "recallAtK": 0.125,
        "mrrAtK": 1,
        "ndcgAtK": 1
      },
      "hybrid": {
        "precisionAtK": 1,
        "recallAtK": 0.125,
        "mrrAtK": 1,
        "ndcgAtK": 1
      }
    }
  },
  "deltas": {
    "all": {
      "semantic": {
        "precisionAtK": -0.3334,
        "recallAtK": -0.0416,
        "mrrAtK": -0.3334,
        "ndcgAtK": -0.3334
      },
      "hybrid": {
        "precisionAtK": -0.0556,
        "recallAtK": -0.0069,
        "mrrAtK": -0.0834,
        "ndcgAtK": -0.0782
      }
    },
    "standard": {
      "semantic": {
        "precisionAtK": -1,
        "recallAtK": -0.125,
        "mrrAtK": -1,
        "ndcgAtK": -1
      },
      "hybrid": {
        "precisionAtK": -0.5833,
        "recallAtK": -0.0729,
        "mrrAtK": -0.625,
        "ndcgAtK": -0.6173
      }
    },
    "adversarial": {
      "semantic": {
        "precisionAtK": 1,
        "recallAtK": 0.125,
        "mrrAtK": 1,
        "ndcgAtK": 1
      },
      "hybrid": {
        "precisionAtK": 1,
        "recallAtK": 0.125,
        "mrrAtK": 1,
        "ndcgAtK": 1
      }
    }
  },
  "perQuery": {
    "preDrift": [
      {
        "queryId": "movement-risk-ops",
        "scenario": "standard",
        "relevantTopic": "movement-risk",
        "semantic": {
          "precisionAtK": 1,
          "recallAtK": 0.125,
          "mrrAtK": 1,
          "ndcgAtK": 1
        },
        "hybrid": {
          "precisionAtK": 1,
          "recallAtK": 0.125,
          "mrrAtK": 1,
          "ndcgAtK": 1
        }
      },
      {
        "queryId": "respiratory-risk-check",
        "scenario": "standard",
        "relevantTopic": "respiratory-care",
        "semantic": {
          "precisionAtK": 1,
          "recallAtK": 0.125,
          "mrrAtK": 1,
          "ndcgAtK": 1
        },
        "hybrid": {
          "precisionAtK": 1,
          "recallAtK": 0.125,
          "mrrAtK": 1,
          "ndcgAtK": 1
        }
      },
      {
        "queryId": "fleece-quality-watch",
        "scenario": "standard",
        "relevantTopic": "fleece-quality",
        "semantic": {
          "precisionAtK": 1,
          "recallAtK": 0.125,
          "mrrAtK": 1,
          "ndcgAtK": 1
        },
        "hybrid": {
          "precisionAtK": 1,
          "recallAtK": 0.125,
          "mrrAtK": 1,
          "ndcgAtK": 1
        }
      },
      {
        "queryId": "behaviour-stability-alerts",
        "scenario": "standard",
        "relevantTopic": "behaviour-stability",
        "semantic": {
          "precisionAtK": 1,
          "recallAtK": 0.125,
          "mrrAtK": 1,
          "ndcgAtK": 1
        },
        "hybrid": {
          "precisionAtK": 1,
          "recallAtK": 0.125,
          "mrrAtK": 1,
          "ndcgAtK": 1
        }
      },
      {
        "queryId": "respiratory-intent-with-movement-vector",
        "scenario": "adversarial",
        "relevantTopic": "respiratory-care",
        "semantic": {
          "precisionAtK": 0,
          "recallAtK": 0,
          "mrrAtK": 0,
          "ndcgAtK": 0
        },
        "hybrid": {
          "precisionAtK": 0,
          "recallAtK": 0,
          "mrrAtK": 0,
          "ndcgAtK": 0
        }
      },
      {
        "queryId": "behaviour-intent-with-fleece-vector",
        "scenario": "adversarial",
        "relevantTopic": "behaviour-stability",
        "semantic": {
          "precisionAtK": 0,
          "recallAtK": 0,
          "mrrAtK": 0,
          "ndcgAtK": 0
        },
        "hybrid": {
          "precisionAtK": 0,
          "recallAtK": 0,
          "mrrAtK": 0,
          "ndcgAtK": 0
        }
      }
    ],
    "postDrift": [
      {
        "queryId": "movement-risk-ops",
        "scenario": "standard",
        "relevantTopic": "movement-risk",
        "semantic": {
          "precisionAtK": 0,
          "recallAtK": 0,
          "mrrAtK": 0,
          "ndcgAtK": 0
        },
        "hybrid": {
          "precisionAtK": 0,
          "recallAtK": 0,
          "mrrAtK": 0,
          "ndcgAtK": 0
        }
      },
      {
        "queryId": "respiratory-risk-check",
        "scenario": "standard",
        "relevantTopic": "respiratory-care",
        "semantic": {
          "precisionAtK": 0,
          "recallAtK": 0,
          "mrrAtK": 0,
          "ndcgAtK": 0
        },
        "hybrid": {
          "precisionAtK": 1,
          "recallAtK": 0.125,
          "mrrAtK": 1,
          "ndcgAtK": 1
        }
      },
      {
        "queryId": "fleece-quality-watch",
        "scenario": "standard",
        "relevantTopic": "fleece-quality",
        "semantic": {
          "precisionAtK": 0,
          "recallAtK": 0,
          "mrrAtK": 0,
          "ndcgAtK": 0
        },
        "hybrid": {
          "precisionAtK": 0,
          "recallAtK": 0,
          "mrrAtK": 0,
          "ndcgAtK": 0
        }
      },
      {
        "queryId": "behaviour-stability-alerts",
        "scenario": "standard",
        "relevantTopic": "behaviour-stability",
        "semantic": {
          "precisionAtK": 0,
          "recallAtK": 0,
          "mrrAtK": 0,
          "ndcgAtK": 0
        },
        "hybrid": {
          "precisionAtK": 0.6667,
          "recallAtK": 0.0833,
          "mrrAtK": 0.5,
          "ndcgAtK": 0.5307
        }
      },
      {
        "queryId": "respiratory-intent-with-movement-vector",
        "scenario": "adversarial",
        "relevantTopic": "respiratory-care",
        "semantic": {
          "precisionAtK": 1,
          "recallAtK": 0.125,
          "mrrAtK": 1,
          "ndcgAtK": 1
        },
        "hybrid": {
          "precisionAtK": 1,
          "recallAtK": 0.125,
          "mrrAtK": 1,
          "ndcgAtK": 1
        }
      },
      {
        "queryId": "behaviour-intent-with-fleece-vector",
        "scenario": "adversarial",
        "relevantTopic": "behaviour-stability",
        "semantic": {
          "precisionAtK": 1,
          "recallAtK": 0.125,
          "mrrAtK": 1,
          "ndcgAtK": 1
        },
        "hybrid": {
          "precisionAtK": 1,
          "recallAtK": 0.125,
          "mrrAtK": 1,
          "ndcgAtK": 1
        }
      }
    ]
  }
}

Why this matters:

  • It gives you pre-vs-post metric deltas before deploying a new embedding model to production.
  • It reveals whether semantic and hybrid ranking degrade uniformly or in different ways.
  • It turns model refresh decisions into measurable change management rather than guesswork.

Common Failure Modes and How to Avoid Them

Most vector-search projects do not fail because the embedding model is "bad". They fail because retrieval design is under-specified: scoring is too simple, controls are bolted on late, and monitoring starts after release. The patterns below are the ones that most often cause regressions when teams move from prototype to production.

1. Vector-only ranking in high-risk workflows

If an incorrect match can trigger expensive decisions, pure vector ranking is usually too permissive. Vector similarity is excellent for candidate generation, but weak as a sole decision signal when domain correctness matters.

Use vector search to widen recall, then add lexical checks and business rules to control ranking quality. In practice, this means combining semantic scores with BM25, applying minimum thresholds, and filtering out known bad contexts before presenting results.

2. Ignoring metadata constraints

Semantic closeness is not business correctness. Two chunks can be semantically near while belonging to different legal regions, tenants, lifecycle states, or policy tiers.

Apply metadata constraints as first-class retrieval conditions, not post-hoc clean-up. Filter by status, recency, jurisdiction, tenancy, and domain guardrails before final ranking so invalid candidates never compete for top positions.

3. Embedding drift without monitoring

When models change, nearest neighbours shift. Even "minor" model upgrades can move borderline results enough to alter top-k behaviour and downstream outcomes.

Treat embedding refreshes as controlled releases. Run pre-vs-post benchmark sets, compare scenario metrics (including adversarial cases), and keep rollback-ready model version metadata so regressions can be isolated quickly.

4. Over-trusting top-1

Similarity scores are relative, not absolute truth. A top-1 result can still be wrong, especially when intent is underspecified or hard negatives are semantically close.

Design consumers to use top-k as candidate context, not as unchallenged fact. Add confidence bands, abstain/fallback paths, and simple cross-checks (for example lexical overlap or topic guardrails) before high-impact actions are taken.

5. No lexical fallback

Without a lexical fallback, incident response becomes hard: semantic decisions are less deterministic, harder to explain, and more sensitive to model changes.

In this article, lexical fallback means a deterministic keyword-based retrieval path (for example BM25, phrase, and fielded term queries) that is used when vector or hybrid confidence is too low, results are empty, or behaviour becomes unstable after model changes.

In practical terms with these samples: run the hybrid scorer in assets/vector-search-hybrid.sjs first, and if no candidates clear confidence gates, switch to the lexical path in assets/vector-search-baseline-lexical.sjs and rank by lexical score.

Always keep a lexical path for auditability, deterministic reproduction, and operational continuity. During outages, drift events, or governance reviews, lexical fallback often becomes the fastest way to stabilise retrieval behaviour while semantic tuning is corrected.

Security and Governance Considerations

Vector retrieval still returns documents. That means permissions, compartments, and governance policies are still first-class concerns.

In MarkLogic, keep those controls inside the retrieval query itself rather than as a post-processing patch.

Recommended posture:

  1. Filter by collections/metadata and permissions before final ranking.
  2. Log retrieval inputs, weights, and thresholds for traceability.
  3. Version embedding model metadata with your content.
  4. Keep an auditable lexical fallback path.

A Practical Rollout Path

If your team is introducing vectors for the first time, use this sequence:

  1. Establish a lexical baseline and save measurable metrics.
  2. Add vector-only retrieval to quantify semantic recall gain.
  3. Implement hybrid ranking with explicit score weights.
  4. Tune thresholds against labelled or expert-reviewed query sets.
  5. Add governance checks and production monitoring before broad rollout.

This keeps vector adoption disciplined and reversible.

Final Guidance

The value of vectors in MarkLogic is practical: semantic retrieval works alongside permissions, metadata constraints, and governance in the same query path. That gives you retrieval behaviour you can explain, test, and operate with confidence.

In this article, the core retrieval samples run against deployed Llamaverse vector content, while drift and adversarial sections deliberately include synthetic stress scenarios to test failure behaviour.

Need Some Help?


Looking for more information on this subject or any other topic related to MarkLogic? Contact Us (info@cleverllamas.com) to find out how we can assist you with consulting or training!
  • Sample Data
  • What a Vector Represents
  • Why MarkLogic is a Strong Vector Host
  • Co-located retrieval context
  • Security at retrieval time
  • Hybrid scoring is native
  • Llamaverse 2.3.1 Vector Lab Document Shape
  • Retrieval Pattern 1: Lexical Baseline
  • Retrieval Pattern 2: Pure Vector Search
  • Retrieval Pattern 3: Hybrid Ranking (Recommended)
  • Threshold Tuning: Where Quality Is Won or Lost
  • Measured Benchmarking, Not Anecdotes
  • Adversarial Rescue Test (Semantic vs Hybrid vs Governed Hybrid)
  • Model Drift Simulation and Metric Deltas
  • Common Failure Modes and How to Avoid Them
  • 1. Vector-only ranking in high-risk workflows
  • 2. Ignoring metadata constraints
  • 3. Embedding drift without monitoring
  • 4. Over-trusting top-1
  • 5. No lexical fallback
  • Security and Governance Considerations
  • A Practical Rollout Path
  • Final Guidance
keyboard_arrow_down
keyboard_arrow_up
toc