Optic Data Analysis Starter Pack

The row set, the aggregation logic, and the shaped result stay together so you can see exactly what each grouping operation does to the input.

This pack covers the analysis surface of Optic: summarising a row set, keeping grouped values together, and then expanding grouped structures again when the next step needs rows instead of arrays.

If reporting logic feels brittle, the row shape is usually the first thing to challenge.

The examples in this article assume the llamaverse (v2.0+) is deployed. The llamaverse sample data is freely available from github.com/cleverllamas/llamaverse - see the llamaverse article for full setup instructions.

# API Context for This Pack

# A Note on "Ungrouping"

There is no standalone op:ungroup() function in the Optic API.

In practice, "ungrouping" means one of two things:

1. Expanding grouped arrays back into rows with op:unnest-inner() or op:unnest-left-outer().
2. Post-processing op:group-to-arrays() output when the grouped object form is the right transport shape but not the final reporting shape.

That distinction matters, because Optic is explicit about whether you still have rows or already have grouped array values.

# op:group-by()

op:group-by() is the workhorse summariser. Use it when one row per original item is too noisy and the real question is about totals, counts, or grouped metrics.

{
  "groupingQuestion": "How many llamas do we have per breed?"
}

xquery version "1.0-ml";
import module namespace op = "http://marklogic.com/optic" at "/MarkLogic/optic.xqy";

op:from-view("llamaverse", "llamas")
  => op:group-by(
       "breed",
       op:count("llamaCount", "id")
     )
  => op:order-by(op:desc("llamaCount"))
  => op:result()

'use strict';

const op = require('/MarkLogic/optic');

const results = op.fromView('llamaverse', 'llamas')
  .groupBy('breed', op.count('count'))
  .orderBy(op.desc('count'))
  .result();

({
  sample: 'optic/data-analysis-starter-pack/assets/op-group-by-breed-count.sjs',
  kind: Array.isArray(results) ? (results.every((item) => typeof item === 'object' && 'subject' in item && 'predicate' in item && 'object' in item) ? 'triples' : 'rows') : ((results !== null && typeof results === 'object') ? 'object' : 'scalar'),
  count: Array.isArray(results) ? results.length : 0,
  data: results
});

{"breed": "Huacaya", "llamaCount": 20}

What to notice: once you group, every column that survives must either be a key or an aggregate.

# op:array-aggregate()

op:array-aggregate() is what you reach for when the summary still needs detail attached to it. Instead of reducing a group to only counts or sums, you keep a collected array of values for each group.

{
  "groupingQuestion": "Which llama names are attached to each birthplace?"
}

xquery version "1.0-ml";
import module namespace op = "http://marklogic.com/optic" at "/MarkLogic/optic.xqy";

op:from-view("llamaverse", "llamas")
  => op:order-by(("placeOfBirth", "name"))
  => op:group-by(
       "placeOfBirth",
       op:array-aggregate("llamaNames", "name")
     )
  => op:order-by("placeOfBirth")
  => op:limit(3)
  => op:result()

'use strict';

const op = require('/MarkLogic/optic');

const results = op.fromView('llamaverse', 'llamas')
  .groupBy('birthplaceCountry', op.arrayAggregate('llamaNames', 'name'))
  .orderBy('birthplaceCountry')
  .limit(5)
  .result();

({
  sample: 'optic/data-analysis-starter-pack/assets/op-array-aggregate-birthplace-names.sjs',
  kind: Array.isArray(results) ? (results.every((item) => typeof item === 'object' && 'subject' in item && 'predicate' in item && 'object' in item) ? 'triples' : 'rows') : ((results !== null && typeof results === 'object') ? 'object' : 'scalar'),
  count: Array.isArray(results) ? results.length : 0,
  data: results
});

{"placeOfBirth": "Arequipa, Peru", "llamaNames": ["Ashley", "Bradley", "Debbie", "Jaime", "Katrina"]}
{"placeOfBirth": "Cusco, Peru", "llamaNames": ["Aaron", "Angela", "Glen"]}
{"placeOfBirth": "La Paz, Bolivia", "llamaNames": ["Jeanette", "Kimberly"]}

What to notice: this is still grouped output, but the grouped detail now travels with the summary instead of disappearing.

# op:group-to-arrays()

op:group-to-arrays() is the higher-level grouped reporting shape. Use it when you want multiple grouped summaries returned as separate named arrays in a single result object.

{
  "reportGoal": "Return grouped summaries by breed, by birthplace, and overall"
}

xquery version "1.0-ml";
import module namespace op = "http://marklogic.com/optic" at "/MarkLogic/optic.xqy";

op:from-view("llamaverse", "llamas")
  => op:group-to-arrays(
       (
         op:named-group("byBreed", "breed"),
         op:named-group("byBirthplace", "placeOfBirth"),
         op:named-group("overall")
       ),
       op:count("llamaCount", "id")
     )
  => op:result()

'use strict';

const op = require('/MarkLogic/optic');

const results = op.fromView('llamaverse', 'llamas')
  .groupToArrays(op.namedGroup('breedSummary', ['breed']), op.count('count'))
  .result();

({
  sample: 'optic/data-analysis-starter-pack/assets/op-group-to-arrays-summary.sjs',
  kind: Array.isArray(results) ? (results.every((item) => typeof item === 'object' && 'subject' in item && 'predicate' in item && 'object' in item) ? 'triples' : 'rows') : ((results !== null && typeof results === 'object') ? 'object' : 'scalar'),
  count: Array.isArray(results) ? results.length : 0,
  data: results
});

{"byBreed": [{"breed": "Huacaya", "llamaCount": 20}], "byBirthplace": [{"placeOfBirth": "Arequipa, Peru", "llamaCount": 5}, {"placeOfBirth": "Cusco, Peru", "llamaCount": 3}, {"placeOfBirth": "La Paz, Bolivia", "llamaCount": 2}, {"placeOfBirth": "Quito, Ecuador", "llamaCount": 5}, {"placeOfBirth": "Santiago, Chile", "llamaCount": 5}], "overall": [{"llamaCount": 20}]}

What to notice: this is the compact reporting shape you want when a caller expects grouped objects, not a flat union of grouped rows.

# group-by-union vs grouped arrays

op:group-by-union() is the lower-level alternative when you want multiple grouping sets in one row stream instead of a single object containing named arrays.

{
  "name": "Aaron",
  "breed": "Huacaya",
  "placeOfBirth": "Cusco, Peru",
  "secretPowerId": "d8839ba6-2b77-4bcc-9927-b86cdfecb9fb"
}

xquery version "1.0-ml";
import module namespace op = "http://marklogic.com/optic" at "/MarkLogic/optic.xqy";

op:from-view("llamaverse", "llamas")
  => op:group-by-union(
       (
         op:group("breed"),
         op:group("placeOfBirth"),
         op:group()
       ),
       op:count("llamaCount", "id")
     )
  => op:result()

'use strict';

const op = require('/MarkLogic/optic');

const results = op.fromView('llamaverse', 'llamas')
  .groupByUnion(
    [
      op.groupBy('breed', op.count('countByBreed')),
      op.groupBy('birthplaceCountry', op.count('countByCountry'))
    ],
    op.group('summary')
  )
  .result();

({
  sample: 'optic/data-analysis-starter-pack/assets/op-group-by-union-summary.sjs',
  kind: Array.isArray(results) ? (results.every((item) => typeof item === 'object' && 'subject' in item && 'predicate' in item && 'object' in item) ? 'triples' : 'rows') : ((results !== null && typeof results === 'object') ? 'object' : 'scalar'),
  count: Array.isArray(results) ? results.length : 0,
  data: results
});

{"breed": "Huacaya", "placeOfBirth": null, "llamaCount": 20}
{"breed": null, "placeOfBirth": "Arequipa, Peru", "llamaCount": 5}
{"breed": null, "placeOfBirth": null, "llamaCount": 20}
{"breed": null, "placeOfBirth": "Cusco, Peru", "llamaCount": 3}
{"breed": null, "placeOfBirth": "La Paz, Bolivia", "llamaCount": 2}
{"breed": null, "placeOfBirth": "Quito, Ecuador", "llamaCount": 5}
{"breed": null, "placeOfBirth": "Santiago, Chile", "llamaCount": 5}

What to notice: grouped arrays are easier for reporting payloads. group-by-union() is better when you still want one row stream that downstream Optic logic can keep processing.

# Ungrouping in Practice

If grouped arrays are the right transport form but the next step needs rows again, ungrouping usually means one of these patterns:

1. Use op:unnest-inner() or op:unnest-left-outer() when the grouped value is still part of an Optic row set. The concrete op:xpath() sequence-expansion example lives in Optic Data Manipulation Starter Pack under the op:xpath() sequence expansion with op:unnest-inner() section.
2. Use plain XQuery post-processing when the result has already been shaped into named arrays with op:group-to-arrays().

That is why this pack treats grouping and ungrouping as a pair of reporting choices rather than as opposite names for one function.

# Ready to Ask Harder Questions?

Decision rule: choose row shape deliberately before grouping. Most reporting pain starts with the wrong source shape, not the wrong aggregate.

Grouping and aggregation are powerful, but they work even better when paired with the right data sources:

• Optic Data Source Starter Pack — Choose the right row source before you aggregate. Different sources unlock different analysis patterns.
• Optic Joins Starter Pack — Combine data from multiple sources, then analyse the merged result set.