Triple Index and TDE Interaction

One of the most important TDE facts in MarkLogic is also one of the easiest to miss: TDE rows and semantic triples share the same underlying triple index. That sounds like an implementation detail. It is not. It explains security behaviour, reindexing behaviour, and why cts:column-range-query() can produce a confusing mismatch between cts:uris() and cts:search(). It also explains why some common assumptions about column-level visibility are simply wrong.

The examples in this article use the llamaverse (v2.0+): JSON documents for the residents of a llama sanctuary, deployed with three TDE templates that project llamaverse.llamas, llamaverse.secretPowers, and llamaverse.wildLlamas views. The llamaverse sample data is freely available from github.com/cleverllamas/llamaverse — see the llamaverse article for full setup instructions.

# The Core Architecture

# TDE and Triples Share the Triple Index

MarkLogic does not maintain one storage engine for semantic triples and another for TDE rows. Both features are built on the same triple index. Semantic triples are explicit RDF-style subject-predicate-object statements. TDE rows are relational projections extracted from documents. Under the hood, both are represented in structures optimised for subject-predicate-object style access — which is why you can query the same underlying store through three completely different interfaces: Optic, SQL, and SPARQL.

This is not trivial. It is the explanation for several production behaviours that otherwise look arbitrary.

# Conceptual Mapping

This is a conceptual model — the internal implementation is more complex — but it is close enough to explain the important behaviours.

# Why the Triple Index Must Be Enabled

If the triple index is disabled, TDE views do not work. If the triple index is being reindexed, TDE availability and freshness are affected. This is the first operational consequence of the shared foundation.

The cache is a performance helper. The index is the feature. Do not confuse them.

# TDE Templates in Practice

# The Llamaverse Template Structure

The llamaverse ships with three TDE templates stored in the schemas database. The llamas.tde template is the primary one — it extracts a row for each llama profile from the /envelope/instance/llamas context. The full template and a sample source document are shown below.

{
  "template": {
    "context": "/envelope/instance/llamas",
    "rows": [
      {
        "schemaName": "llamaverse",
        "viewName": "llamas",
        "columns": [
          { "name": "id",               "scalarType": "string", "val": "id" },
          { "name": "name",             "scalarType": "string", "val": "name" },
          { "name": "heightCm",         "scalarType": "int",    "val": "heightCm" },
          { "name": "weightKg",         "scalarType": "int",    "val": "weightKg" },
          { "name": "eyeColor",         "scalarType": "string", "val": "eyeColor" },
          { "name": "hairColor",        "scalarType": "string", "val": "hairColor" },
          { "name": "breed",            "scalarType": "string", "val": "breed" },
          { "name": "placeOfBirth",     "scalarType": "string", "val": "placeOfBirth" },
          { "name": "medicalCondition", "scalarType": "string", "val": "(medicalCondition/name, 'none')[last()]" },
          { "name": "secretPowerId",    "scalarType": "string", "val": "secretPower/id" },
          { "name": "description",      "scalarType": "string", "val": "description" }
        ]
      }
    ]
  }
}

{
  "envelope": {
    "headers": { "type": "llamas" },
    "instance": {
      "llamas": {
        "id": "0c8bdb0d-ac62-49b7-ac74-94dbba46efa5",
        "name": "Aaron",
        "heightCm": 180,
        "weightKg": 138,
        "eyeColor": "Amber",
        "hairColor": "Blonde",
        "breed": "Huacaya",
        "placeOfBirth": "Cusco, Peru",
        "interests": ["playing chess", "birdwatching", "writing poetry"],
        "medicalCondition": null,
        "relatedTo": {
          "id": "a6c69bb3-fa75-4327-bed6-1f0623ae2c6c",
          "relationship": "father"
        },
        "description": "Aaron is an amber-eyed llama with blonde hair, standing 180 cm tall. Originally from Cusco, Peru, Aaron enjoys playing chess, birdwatching, and writing poetry.",
        "secretPower": {
          "name": "Snake Charmer",
          "id": "f6236908-9c0e-416e-a591-a1bd3986dc05"
        }
      }
    }
  }
}

Several things are worth noting in this template:

• The context is /envelope/instance/llamas — extraction begins inside the envelope wrapper, not at document root.
• medicalCondition uses a value expression (medicalCondition/name, 'none')[last()] — if there is no medical condition, the string "none" is substituted. This is a common pattern for optional fields.
• secretPowerId extracts from a nested object: secretPower/id. This becomes the foreign key for the join to secretPowers.

# Querying the View with Optic

The Optic API is the cleanest way to work with TDE views. op:from-view() begins from the extracted rows rather than from documents, so MarkLogic can reason over the row structure efficiently without opening source documents.

xquery version "1.0-ml";

(: Query the llamaverse.llamas TDE view using the Optic API.             :)
(: Selects profile columns for all llamas and limits to 5 rows.          :)
import module namespace op = "http://marklogic.com/optic"
  at "/MarkLogic/optic.xqy";

op:from-view("llamaverse", "llamas")
=> op:select(("name", "breed", "heightCm"))
=> op:order-by("name")
=> op:limit(5)
=> op:result()

{"llamaverse.llamas.name":"Aaron",   "llamaverse.llamas.breed":"Huacaya", "llamaverse.llamas.heightCm":180}
{"llamaverse.llamas.name":"Angela",  "llamaverse.llamas.breed":"Huacaya", "llamaverse.llamas.heightCm":167}
{"llamaverse.llamas.name":"Anthony", "llamaverse.llamas.breed":"Huacaya", "llamaverse.llamas.heightCm":171}
{"llamaverse.llamas.name":"Ashley",  "llamaverse.llamas.breed":"Huacaya", "llamaverse.llamas.heightCm":167}
{"llamaverse.llamas.name":"Bradley", "llamaverse.llamas.breed":"Huacaya", "llamaverse.llamas.heightCm":172}

# Aggregation

Because TDE rows are indexed facts, aggregations over selective queries are efficient.

xquery version "1.0-ml";

(: Count llamas grouped by breed using the Optic API.                    :)
(: Demonstrates aggregation over a TDE-backed view.                      :)
import module namespace op = "http://marklogic.com/optic"
  at "/MarkLogic/optic.xqy";

op:from-view("llamaverse", "llamas")
=> op:group-by("breed", op:count("count", "id"))
=> op:order-by(op:desc("count"))
=> op:result()

{"breed":"Huacaya", "count":20}

# Joining Two Views

The secretPowerId column in llamaverse.llamas is a foreign key that maps to id in llamaverse.secretPowers. Optic can join these two TDE-backed views without touching the source documents.

xquery version "1.0-ml";

(: Join llamaverse.llamas to llamaverse.secretPowers via the secretPowerId:)
(: foreign key. Demonstrates a TDE join entirely within the Optic API.   :)
import module namespace op = "http://marklogic.com/optic"
  at "/MarkLogic/optic.xqy";

let $llamas := op:from-view("llamaverse", "llamas")
let $powers := op:from-view("llamaverse", "secretPowers")

return
  $llamas
  => op:join-inner($powers,
       op:on(
         op:schema-col("llamaverse", "llamas",        "secretPowerId"),
         op:schema-col("llamaverse", "secretPowers",  "id")
       ))
  => op:select((
       op:schema-col("llamaverse", "llamas",       "name"),
       op:schema-col("llamaverse", "llamas",       "breed"),
       op:schema-col("llamaverse", "secretPowers", "name")
     ))
  => op:order-by(op:schema-col("llamaverse", "llamas", "name"))
  => op:limit(5)
  => op:result()

{"llamaverse.llamas.name":"Aaron",   "llamaverse.llamas.breed":"Huacaya", "llamaverse.secretPowers.name":"Snake Charmer"}
{"llamaverse.llamas.name":"Angela",  "llamaverse.llamas.breed":"Huacaya", "llamaverse.secretPowers.name":"Leaf Dancer"}
{"llamaverse.llamas.name":"Anthony", "llamaverse.llamas.breed":"Huacaya", "llamaverse.secretPowers.name":"Seer of Stars"}
{"llamaverse.llamas.name":"Ashley",  "llamaverse.llamas.breed":"Huacaya", "llamaverse.secretPowers.name":"Druid's Sight"}
{"llamaverse.llamas.name":"Bradley", "llamaverse.llamas.breed":"Huacaya", "llamaverse.secretPowers.name":"Time Whisperer"}

# Security Behaviour

# Security Starts With the Source Document

The most important TDE security rule is document-level inheritance: if a user cannot see the source document, they cannot see rows extracted from that document. There is no detached TDE visibility model that escapes this. Rows do not float free from their origin.

# Element Level Security and Whole-Row Suppression

If any column value in a TDE row comes from an ELS-protected path, the entire row is hidden from users who cannot see that protected content. Not that column — the whole row.

People often expect TDE to behave like a relational result set with nullable or masked columns. That is not how this layer works. The triple index enforces row visibility in relation to the underlying protected content. MarkLogic does not provide column-level security slicing within a single extracted row. If one value is protected by ELS, the row is suppressed.

# Consequence: No Per-Column Masking Within a Row

There is no per-column security view within a single TDE row. You cannot model "column A visible to everybody, column B only visible to auditors" in the same row and expect TDE to partially reveal it.

The practical workaround is to separate extraction by security domain:

For the llamaverse, a template that extracts name, breed, and heightCm is fine for general users. A template that also extracts medicalCondition should be scoped to the role that has access to that data — and if that data is ELS-protected, the whole row will be hidden from users who cannot see it.

# The cts:column-range-query() Gotcha

# Filtered vs Unfiltered With TDE Columns

cts:column-range-query() lets you query a TDE column from CTS — a clean bridge between TDE and document retrieval. But there is a well-known gotcha: cts:search() with cts:column-range-query() can return zero results in filtered mode even when cts:uris() resolves matching fragments.

xquery version "1.0-ml";

(: cts:column-range-query() filtered vs unfiltered gotcha.               :)
(:                                                                        :)
(: cts:uris() is unfiltered by default — it resolves from index data     :)
(: and correctly returns all 20 matching URIs.                           :)
(:                                                                        :)
(: cts:search() is filtered by default. The filter pass discards every   :)
(: candidate, returning an empty sequence — even though the URIs exist.  :)
(:                                                                        :)
(: The fix is to pass "unfiltered" explicitly to cts:search().           :)
let $q := cts:column-range-query("llamaverse", "llamas", "breed", "Huacaya")

return (
  "cts:uris()   (unfiltered default): " || fn:string(fn:count(
    cts:uris((), (), $q)
  )),
  "cts:search() (filtered  default): " || fn:string(fn:count(
    cts:search(fn:collection(), $q)
  )),
  "cts:search() (unfiltered explicit): " || fn:string(fn:count(
    cts:search(fn:collection(), $q, "unfiltered")
  ))
)

cts:uris()   (unfiltered default):  20
cts:search() (filtered  default):    0
cts:search() (unfiltered explicit): 20

All 20 llamaverse llama documents match — cts:uris() correctly returns 20. cts:search() in its default filtered mode returns nothing. Passing "unfiltered" explicitly to cts:search() restores the correct count.

# Why This Happens

cts:uris() is unfiltered by default and resolves URIs from index data. cts:search() is filtered by default — it resolves candidates from the index, then opens each document to validate the match. With cts:column-range-query(), the observed behaviour is that filtered cts:search() discards all candidates while unfiltered search returns the expected matches. The exact filter-step mechanism is not documented in detail publicly, so treat this as tested behaviour and pass "unfiltered" explicitly when using cts:column-range-query() inside cts:search(). This behaviour is discussed in depth in the Filtered vs Unfiltered Searches article.

# Reindexing Considerations

# Template Changes Are Indexing Events

Whenever TDE-relevant structures change, reindexing enters the picture. Triggers include:

• enabling the triple index on a previously disabled database
• inserting, updating, or deleting TDE templates
• changing extraction definitions in ways that require rebuilt rows
• modifying ELS-protected paths that contribute to extracted rows

In large databases this can be substantial — real IO, real time, real operational planning. Treat template deployment as an indexing event, not a metadata-only change.

# Query Behaviour During Reindexing

During reindexing, some results may not fully reflect the final extracted state until it catches up. Schedule and monitor reindexing accordingly.

# Quick Reference

# Deployment Checklist

Before deploying or changing TDE in production, verify:

• Is the triple index enabled on the target database?
• What reindexing will this change trigger, and how long will it take?
• Are template contexts selective and stable?
• Do any columns touch ELS-protected paths?
• Should sensitive data move to a separate template?
• Do downstream queries use Optic, SQL, SPARQL, or cts:column-range-query()?
• Are you relying on filtered or unfiltered document retrieval anywhere?

Treat this checklist as release hygiene, not optional paperwork. It saves real incidents later.