All About Lexicons

MarkLogic lexicons are often split into separate mental buckets — "the URI index", "the collection index", "range index values" — and treated as unrelated features. They are not. URI lookup, collection enumeration, scalar value retrieval, co-occurrence analysis, and tuple reads all share the same underlying index infrastructure and the same reference-based query model.

The practical consequence is straightforward. Teams that understand the lexicon family build concise, fast, composable queries. Teams that treat each lexicon type as a separate one-off mechanism end up with fragile workarounds, expensive document scans for questions indexes already answer, and query code that becomes painful to maintain as models evolve.

This article covers the full lexicon family: what each type is for, when to use it, and the design and operational patterns that hold up at scale.

The examples in this article assume the llamaverse (v2.3.1+) is deployed. The llamaverse sample data is freely available from github.com/cleverllamas/llamaverse — see the llamaverse article for full setup instructions.

# The Lexicon Family

MarkLogic exposes these lexicon surfaces:

All lexicon surfaces are queried through references (cts:uri-reference(), cts:collection-reference(), cts:element-reference(), cts:path-reference(), cts:field-reference("..."), and so on). Once you have a reference, the same family of functions applies: cts:values(), cts:range-query(reference, ...), cts:tuples(), op.fromLexicons(), and index-based ordering in search flows via order by cts:index-order(reference, ...).

# The Shared Mental Model

Scope note: this article is about lexicons and lexicon references. TDE template trade-offs and template-led query design are out of scope for this article.

Use this pattern consistently across all lexicon types:

1. Define the reference — cts:collection-reference(), cts:element-reference(xs:QName("breed")), cts:path-reference("/heightCm"), etc.
2. Decide what you need — distinct values (cts:values), filtered document sets (cts:range-query), co-occurring combinations (cts:tuples), or fragment-attached rows (op.fromLexicons).
3. Add scope — a cts:query argument narrows retrieval to a meaningful subset.

# Index Prerequisites

Lexicon queries depend on the correct index flags being enabled on the database.

{
  "collection-lexicon": true,
  "uri-lexicon": true
}

If a query shape is correct but behaves slowly, returns unexpectedly empty results, or throws an index-not-present error, verify index settings before rewriting query logic. The Admin UI path for each index type is:

# URI Lexicon

The URI lexicon is easy to describe and easy to underuse. At face value it is an index of document URIs. In practice it becomes a fast control surface for operations, incremental processing, existence checks, and selective maintenance — without loading whole documents.

URI-centric workflows let you ask identity questions cheaply:

1. Which documents match this URI pattern?
2. Which URI is next after the one I last processed?
3. Does this exact URI exist?
4. Which URIs in this partition also match a content constraint?

# cts:uris() Basics

cts:uris(start, options, query, quality-weight, forest-ids) is the workhorse. The argument order matters and is a frequent source of subtle bugs when values are accidentally shifted.

xquery version "1.0-ml";

(: Basic ordered URI retrieval scoped to wild-llamas. :)

cts:uris(
  (),
  ("document", "item-order", "ascending"),
  cts:collection-query("wild-llamas")
)[1 to 3]

/cleverllamas/llamaverse/raw/wild-llamas/llamas/00048384-cb13-4557-805e-a6b4e57f7eab.json
/cleverllamas/llamaverse/raw/wild-llamas/llamas/0005e261-4302-4ae8-9574-732a54040423.json
/cleverllamas/llamaverse/raw/wild-llamas/llamas/000eca10-d166-469f-b17a-3c3b35ee0883.json

Two practical notes:

1. Keep parameter order explicit in shared utility modules to avoid accidental argument shifting.
2. Treat URI retrieval as a first pass for identity and partitioning, then load content only for the subset you actually need.

# cts:uri-match() for Wildcard Patterns

Use cts:uri-match() when your primary filter is URI shape rather than a CTS query constraint.

xquery version "1.0-ml";

(: Wildcard URI match from lexicon. :)

cts:uri-match(
  "/cleverllamas/llamaverse/raw/wild-llamas/llamas/*.json",
  ("document", "ascending")
)[1 to 3]

/cleverllamas/llamaverse/raw/wild-llamas/llamas/00048384-cb13-4557-805e-a6b4e57f7eab.json
/cleverllamas/llamaverse/raw/wild-llamas/llamas/0005e261-4302-4ae8-9574-732a54040423.json
/cleverllamas/llamaverse/raw/wild-llamas/llamas/000eca10-d166-469f-b17a-3c3b35ee0883.json

If you also need content constraints, combine URI-first retrieval with a CTS query in cts:uris() rather than running separate scans.

# Fast Existence Checks

For URI-space workflows, lexicon existence checks are clean and cheap.

xquery version "1.0-ml";

let $uri := "/cleverllamas/llamaverse/raw/wild-llamas/llamas/00048384-cb13-4557-805e-a6b4e57f7eab.json"
(: "document" ensures this checks for a document fragment specifically.      :)
(: A URI can also appear as properties or locks, depending on options.        :)
return fn:exists(cts:uri-match($uri, ("document")))

true

The ("document") option is deliberate: this check asks whether a document fragment exists at that URI, not whether the URI appears in the lexicon under any fragment type.

This distinction matters. Per cts:uri-match / cts:uris option semantics, URI lexicon retrieval can target any, document, properties, or locks. In other words, a URI can be present for properties/metadata or lock state even when no document fragment is present, so use the fragment option that matches the lifecycle decision you are making.

Directories are relevant here as well. Directory creation uses xdmp:directory-create. A created directory has a URI and is represented as a document fragment, so the "document" option includes directory URIs alongside regular document URIs.

This pattern is useful for idempotent import and replay pipelines where URI existence determines whether to insert, skip, or repair.

# Lexical Pagination

Lexicon pagination is most robust when you resume from the last URI value rather than relying on large skip offsets.

xquery version "1.0-ml";

(: Lexical pagination using start URI and truncate option.             :)
(: Resume from the last URI returned rather than a skip offset.        :)

let $page1 := cts:uris(
  (),
  ("document", "item-order", "ascending", "truncate=3"),
  cts:collection-query("wild-llamas")
)
let $last := $page1[fn:last()]
let $page2 := cts:uris(
  $last,
  ("document", "item-order", "ascending", "truncate=4"),
  cts:collection-query("wild-llamas")
)[fn:position() gt 1]
return ($page1, "---", $page2)

This pattern gives deterministic traversal and restartability for long-running batch jobs.

Use it for:

1. Nightly maintenance over millions of documents.
2. Resumable reindex and backfill workflows.
3. Retry-safe integration exports.

# Combining URI and Content Constraints

A common misconception is that URI lexicon workflows are URI-only. cts:uris() accepts a CTS query constraint, so you can keep URI ordering while limiting to semantically relevant documents.

Pattern:

1. Provide a URI start value for resumability.
2. Apply a CTS query for business filtering.
3. Process returned URIs in deterministic windows.

# URI Option Reference

# Forest-Aware Traversal

For very large datasets, the optional forest-ids parameter in cts:uris() scopes traversal to specific forests. This is useful for staged maintenance, targeted recovery, and controlled parallel execution. Only use forest-scoped traversal if your runbook documents the partitioning assumptions clearly.

# URI Lexicon Operational Patterns

# Collection Lexicon

The collection lexicon provides instant enumeration of all collection names and supports lexicon-joined scope queries across those names.

# List All Collections

xquery version "1.0-ml";

(: List every collection in the database.                              :)
(: With the collection lexicon enabled, this reads directly from the   :)
(: index and returns instantly, regardless of database size.           :)
cts:collections()

"llamaverse"
"content"
"raw"
"raw*:wild-llamas"
"wild-llamas"

With the collection lexicon enabled this reads directly from the index and returns instantly, regardless of database size.

# Count Documents per Collection

xquery version "1.0-ml";

(: Count documents in every collection, ordered highest-first.         :)
(: cts:estimate() reads the collection lexicon — no document scan.     :)
for $c in cts:collections()
let $count := cts:estimate(cts:collection-query($c))
order by $count descending
return fn:concat($c, ": ", fn:string($count))

"raw: 4648"
"raw*:wild-llamas: 4560"
"wild-llamas: 4560"
"llamaverse: 136"
"content: 48"

cts:estimate() reads the collection lexicon without scanning document content, making this a fast audit tool even on large databases.

# Documents NOT in a Collection

A clean pattern using the collection reference to express exclusion:

xquery version "1.0-ml";

(: Documents NOT in a specific collection.                             :)
(: Uses cts:collection-reference() to express exclusion cleanly.       :)

cts:search(
  fn:collection(),
  cts:not-query(
    cts:range-query(
      cts:collection-reference(),
      "=",
      "wild-llamas"
    )
  )
)[1 to 10] ! xdmp:node-uri(.)

Using cts:range-query(cts:collection-reference(), "=", ...) inside a cts:not-query() is the absolute textbook way to do this. Any other approach is usually going to cost more time, memory, and CPU. Avoid workarounds that load all documents and then filter.

# Co-occurring Collections

Finding which collections always appear together with a given collection is a fast lexicon join:

xquery version "1.0-ml";

(: Find all collections that co-occur with documents in "wild-llamas". :)
(: The third argument to cts:collections() scopes the walk to only     :)
(: documents that also match the supplied query — a lexicon join.      :)
cts:collections(
  (),                                   (: start value: begin from the top :)
  (),                                   (: options: none                   :)
  cts:collection-query("wild-llamas")   (: scope: only wild-llamas docs    :)
)

"raw"
"raw*:wild-llamas"
"wild-llamas"

Co-occurrence analysis is a great way to validate collection assignment strategy in mature content models, and to detect accidental membership drift after ingestion changes.

# Scalar Value Lexicons

Scalar value lexicons are backed by range indexes on elements, attributes, or paths. Once an index is present, cts:values() returns distinct values from it without any document scan.

# Distinct Values with cts:values()

cts:values(reference) returns one item per unique value — not one item per matching document.

xquery version "1.0-ml";

(: Distinct breed values from a range-reference-backed lexicon.        :)
(: Scoped to wild-llamas for relevance.                                :)

cts:values(
  cts:element-reference(xs:QName("breed")),
  (),
  (),
  cts:collection-query("wild-llamas")
)

"Huacaya"
"Suri"

This is ideal for faceting, filter menus, and metadata introspection. The result set is small and comes entirely from the index.

# Range Queries over References

cts:range-query(reference, operator, value) applies structured predicates using the same reference model as cts:values().

xquery version "1.0-ml";

(: URI-first range query variant.                                           :)
(: Returns URIs in lexicographic URI order.                                 :)

let $breed-ref := cts:element-reference(xs:QName("breed"))
let $query := cts:and-query((
  cts:collection-query("wild-llamas"),
  cts:range-query($breed-ref, "=", "Suri")
))
for $uri in cts:uris("", ("item-order", "ascending", "document"), $query)[1 to 5]
return $uri

/cleverllamas/llamaverse/raw/wild-llamas/llamas/0005e261-4302-4ae8-9574-732a54040423.json
/cleverllamas/llamaverse/raw/wild-llamas/llamas/002c7133-e985-4bd6-9ea8-f057fa0c6ee7.json
/cleverllamas/llamaverse/raw/wild-llamas/llamas/006c5ffa-6c5e-4169-8a08-eec49f48148c.json
/cleverllamas/llamaverse/raw/wild-llamas/llamas/007229e2-ee25-47c1-b18a-2e0c0ec0dbcd.json
/cleverllamas/llamaverse/raw/wild-llamas/llamas/00863414-306e-49cb-acd0-6579bea5421e.json

xquery version "1.0-ml";

(: Document-node range query variant.                                       :)
(: Ordered by a second indexed dimension (heightCm), not URI.               :)

let $breed-ref := cts:element-reference(xs:QName("breed"))
let $height-ref := cts:path-reference("/heightCm")
let $breed-query := cts:range-query($breed-ref, "=", "Suri")
for $doc in cts:search(
  fn:collection("wild-llamas"),
  $breed-query,
  ("unfiltered", cts:index-order($height-ref, "ascending"))
)[1 to 5]
let $height := fn:string(($doc//heightCm, $doc//height-cm)[1])
return fn:concat(xdmp:node-uri($doc), " | heightCm=", $height)

/cleverllamas/llamaverse/raw/wild-llamas/llamas/1707fa8c-1d33-45bd-8e45-d4007a6f93fa.json | heightCm=100
/cleverllamas/llamaverse/raw/wild-llamas/llamas/0ceea126-5068-4524-9bc6-df1e39bb8b8d.json | heightCm=100
/cleverllamas/llamaverse/raw/wild-llamas/llamas/1fce7c61-9d43-466e-a608-d01f35c3265e.json | heightCm=100
/cleverllamas/llamaverse/raw/wild-llamas/llamas/d2067696-2fab-4bc9-8b13-e5e618aea78d.json | heightCm=100
/cleverllamas/llamaverse/raw/wild-llamas/llamas/07eabcd8-537a-41ff-a5b1-b26a30d87dcc.json | heightCm=100

Both patterns use the same range predicate, but they serve slightly different purposes:

• cts:uris(...) is ideal when you only need URI values and want lexical URI ordering.
• cts:search(...) returns document nodes, which is useful when you need to order by another indexed value from the matched documents (for example heightCm via cts:index-order(...)).

For structured dimensions (breed, height, weight, date), cts:range-query with an explicit reference is usually cleaner and more predictable than word or value query families.

# Field Lexicons

Fields are the missing layer in many lexicon designs.

If you need one logical lexicon over multiple physical sources, fields are the right tool. A field can unify values that come from different element/path locations while giving you one stable reference in query code (cts:field-reference("field-name")).

This matters most in two scenarios:

1. Data models that evolved over time and now store the same concept in different structures.
2. Values stored in fragment metadata, where fields provide the practical route to queryable lexicon access across that metadata.

# Why Fields Improve Lexicon Design

Without fields, query code often accumulates multiple references and custom OR logic to represent a single business dimension. With fields:

1. Index config maps many sources to one named field.
2. Query code uses one reference (cts:field-reference).
3. Application semantics stay stable even if source structure changes.

That decoupling is exactly what keeps lexicon-heavy systems maintainable.

# Unified Values from a Field Reference

The example below assumes a field (for example, llama-species-unified) has been configured to collect species values from multiple source locations. A field is a general modelling and query abstraction, not a lexicon-only feature. Adding a field range index is one common use when you want value lexicon and range-query behaviour over that unified field surface.

{
  "field-name": "llama-species-unified",
  "included-paths": [
    "/species",
    "/traits/species",
    "/envelope/instance/species",
    "/metadata/species"
  ],
  "notes": "Illustrative field definition excerpt used by the examples in this section."
}

{
  "field-name": "llama-species-unified",
  "scalar-type": "string",
  "range-value-positions": false,
  "collation": "http://marklogic.com/collation/",
  "invalid-values": "ignore"
}

xquery version "1.0-ml";

for $species in
  cts:values(
    cts:field-reference("llama-species-unified"),
    (),
    (),
    cts:collection-query("llamaverse")
  )
return $species

Alpaca
Llama
Vicuna

# Filter Documents with a Field-Based Range Query

Once the field is in place, filtering is the same reference model used elsewhere in this article:

xquery version "1.0-ml";

let $species-ref := cts:field-reference("llama-species-unified")
let $query :=
  cts:and-query((
    cts:collection-query("llamaverse"),
    cts:range-query($species-ref, "=", "Suri")
  ))
for $uri in cts:uris("", ("item-order", "ascending", "document"), $query)
return $uri

/cleverllamas/llamaverse/content/field-lexicon/field-source-breed.json

# Prove It Live: Field + Metadata Coverage

# What We Are Proving

Before deploy, we run one query that checks two things in one pass:

1. Distinct values resolve through the field reference.
2. A metadata-carried value can still be matched through the same field reference.

Think of it as a quick confidence check: one query, two checks to set our mind at ease.

# The Verification Query

xquery version "1.0-ml";

let $species-ref := cts:field-reference("llama-species-unified")
let $target-value := "Guanaco"
let $scope := cts:collection-query("llamaverse")
let $metadata-check-query :=
  cts:and-query((
    $scope,
    cts:range-query($species-ref, "=", $target-value)
  ))

return (
  "Distinct values from the unified field:",
  cts:values($species-ref, (), (), $scope),
  "",
  fn:concat("URIs matching ", $target-value, " via the unified field:"),
  for $uri in cts:uris("", ("item-order", "ascending", "document"), $metadata-check-query)
  return $uri
)

Distinct values from the unified field:
Acanthaster planci
Achatina fulica
Adineta vaga
Agalychnis callidryas
Ailurus fulgens
...

URIs matching Guanaco via the unified field:
/cleverllamas/llamaverse/content/field-lexicon/field-source-species.json

# How to Read the Results

You should see both distinct values and URI matches for the target value from one field reference. If distinct values appear but metadata-backed URI matches do not, the issue is usually field include paths or reindex state rather than query logic.

# Field + Metadata Pattern

When teams need lexicon behaviour over values carried in fragment metadata, field-based design is typically the cleanest approach:

1. Define a field for the business dimension.
2. Include all relevant content paths plus metadata-bearing sources in that field definition.
3. Query with cts:field-reference so content and metadata-backed values are addressed uniformly.

Operationally, this gives one canonical query surface for a dimension, regardless of where values are found in the document, properties or metadata.

# Co-occurrences and Tuples

When you need paired or multi-dimensional value combinations, move from single-value lexicons to tuple retrieval.

xquery version "1.0-ml";

(: Multiple range-backed co-occurrence patterns.                       :)
(: Returns distinct breed values, paired with corresponding heights.   :)
(: Uses cts:values() with multiple references to explore correlation.  :)

for $breed in cts:values(
  cts:element-reference(xs:QName("breed")),
  (),
  (),
  cts:collection-query("wild-llamas")
)[1 to 20]
let $heights := cts:values(
  cts:element-reference(xs:QName("heightCm")),
  (),
  (),
  cts:and-query((
    cts:element-value-query(xs:QName("breed"), $breed),
    cts:collection-query("wild-llamas")
  ))
)
return element breed-height-pair {
  element breed { $breed },
  element count-with-this-breed { count($heights) },
  element avg-height { 
    if (count($heights) > 0) 
    then avg($heights[. castable as xs:double] ! xs:double(.))
    else ()
  }
}

Huacaya | 100
Huacaya | 101
Huacaya | 102
Huacaya | 103
Huacaya | 104
Huacaya | 105
Huacaya | 106
Huacaya | 107
Huacaya | 108
Huacaya | 109

cts:tuples() returns combinations of referenced values that co-occur in the same fragment scope. This is the right primitive for analytics-style grouping and dimension exploration where you need more than one dimension at a time.

# cts:values vs op.fromLexicons

These two functions often confuse teams because they look similar but serve different purposes.

`cts:values(reference)`
- returns distinct values only
- one output item per unique value
- no direct fragment identity in the output

`op.fromLexicons(indexDef, ..., "fragmentId")`
- returns rows sourced from lexicons
- rows remain attached to fragment identity
- can join/filter/project in Optic while keeping fragment linkage

The key distinction is fragment attachment:

• cts:values() collapses to distinct values. Fragment identity is gone.
• op.fromLexicons() keeps rows attached to the fragment they came from.

Optic example with explicit fragment attachment:

'use strict';

const op = require('/MarkLogic/optic');

// fromLexicons returns lexicon values as rows that remain fragment-attached.
// Include fragmentId to make that attachment explicit.

op.fromLexicons(
  {
    breed: cts.elementReference(xs.QName('breed')),
    heightCm: cts.pathReference('/heightCm', ['type=int'])
  },
  null,
  'fragmentId'
)
.where(op.eq(op.col('breed'), 'Suri'))
.select(['fragmentId', 'breed', 'heightCm'])
.limit(10)
.result();

{"fragmentId":"http://marklogic.com/fragment/0000037B96E1D862","breed":"Suri","heightCm":102}
{"fragmentId":"http://marklogic.com/fragment/000003CB96E1D862","breed":"Suri","heightCm":137}
{"fragmentId":"http://marklogic.com/fragment/000003DB96E1D862","breed":"Suri","heightCm":151}
{"fragmentId":"http://marklogic.com/fragment/000003EB96E1D862","breed":"Suri","heightCm":144}
{"fragmentId":"http://marklogic.com/fragment/0000044B96E1D862","breed":"Suri","heightCm":199}

Optic example returning distinct values (similar to cts:values()):

'use strict';

const op = require('/MarkLogic/optic');

// Collapse lexicon rows down to unique breed values, similar to cts:values().
op.fromLexicons({
  breed: cts.elementReference(xs.QName('breed'))
})
.groupBy('breed', [])
.orderBy('breed')
.result();

{"breed":"Huacaya"}
{"breed":"Hybrid"}
{"breed":"Suri"}

Optic example returning only the breed value sequence (same end shape as cts:values()):

'use strict';

const op = require('/MarkLogic/optic');

// Convert distinct Optic rows into a plain value sequence shape.
const rows = op.fromLexicons({
  breed: cts.elementReference(xs.QName('breed'))
})
.groupBy('breed', [])
.orderBy('breed')
.result();

rows.toArray().map(row => row.breed);

["Huacaya", "Hybrid", "Suri"]

Use op.fromLexicons() when you need to join lexicon-driven rows with other Optic views while preserving provenance. Use cts:values() when you only need the unique values themselves.

# Design Decision Matrix

# Throughput and Safety Guidelines

1. Batch URI traversal by windows (truncate=N) rather than unbounded scans.
2. Persist the last successful URI as checkpoint state for restartable jobs.
3. Keep worker actions idempotent for replay safety.
4. Separate discovery (lexicon reads) from expensive mutation where possible.
5. Use item-order for deterministic ordering in all long-running jobs.
6. Centralise reference definitions in shared utility modules — do not re-declare them per query.
7. Validate tuple and co-occurrence assumptions against real fragment boundaries before deployment.

# Common Mistakes

If you keep one consistent reference-first model across URI, collection, value, field, and tuple queries, lexicon-heavy systems stay fast, predictable, and far easier to maintain.