caav docs

caav attaches attention to 3D objects, records it separately for each person, and shows it back to them. It is the toolkit from the IEEE VIS 2026 paper HeedVision: Attention Awareness in Collaborative Immersive Analytics Environments. CAAV stands for collaborative attention-aware visualization, the term the paper introduces.

The library has three layers. caav is the core and has no dependencies. caav/three connects it to three.js. caav/react adds hooks for React Three Fiber.

1. Install
2. Quickstart
3. How it works
4. Attaching to objects
5. People and sources
6. Recording
7. Accumulators
8. Presentations
9. Giving each person their own view
10. Triggers
11. Shared colour
12. Events and spatial tooltips
13. React Three Fiber
14. API reference
15. Limits
16. Where this differs from the paper

Install

The library is not on npm yet, and the source is on GitHub under the MIT License. You can load it directly from this site using an import map, with no build step:

<script type="importmap">
  {
    "imports": {
      "three": "https://cdn.jsdelivr.net/npm/three@0.180.0/build/three.module.js",
      "three-mesh-bvh": "https://cdn.jsdelivr.net/npm/three-mesh-bvh@0.9.18/build/index.module.js",
      "caav": "https://heedvision.arvindchee.nu/toolkit/lib/core/index.js",
      "caav/three": "https://heedvision.arvindchee.nu/toolkit/lib/three/index.js"
    }
  }
</script>

The one-file example is a complete page built this way. Save it, change the paths to the absolute ones above, and open it through any local server.

The React hooks are at lib/react/index.js. They also need react and @react-three/fiber, which is easiest with a bundler.

It has been tested with three 0.180, three-mesh-bvh 0.9.18 and @react-three/fiber 9.8. Type declarations sit next to each file under lib/.

Quickstart

This sets up two people looking at one object, each rendering their own view. object, scene, the two cameras and the two renderers are your own.

import { alwaysOn, createSession, interpolatedColours } from "caav";
import { attachAttention, createAttentionView, headRaySource } from "caav/three";

const session = createSession({ captureRate: 10, influenceRadius: 1.5 });
const attached = attachAttention(session, object, { voxelSize: 0.05 });
const colour = interpolatedColours();

session.registerViewer({ id: "a", colour: "#3d8bff", source: headRaySource(cameraA) });
session.registerViewer({ id: "b", colour: "#f08a3c", source: headRaySource(cameraB) });

const viewA = createAttentionView({
  session, attached, viewer: "a", camera: cameraA, scene, layer: 1,
  presentation: "world-embedded", trigger: alwaysOn(), colour,
});
const viewB = createAttentionView({
  session, attached, viewer: "b", camera: cameraB, scene, layer: 2,
  presentation: "world-separated", trigger: alwaysOn(), colour,
});

let last = performance.now();
function frame(now: number) {
  const dt = (now - last) / 1000;
  last = now;
  session.tick(dt);
  viewA.update(dt);
  viewB.update(dt);
  rendererA.render(scene, cameraA);
  rendererB.render(scene, cameraB);
  requestAnimationFrame(frame);
}
requestAnimationFrame(frame);

The playground on the home page writes this code for whatever settings you choose, so it is often the quickest way to get a setup you can copy.

How it works

The paper splits attention awareness into three parts, and so does the library.

Capture finds what each person is looking at. Ten times a second by default, a ray goes out from each person's head and finds the nearest voxel it hits.

Recording adds to that voxel's value for that person. Each person has their own value at every voxel, between 0 and 1, and those values fade over time. Keeping them separate is what makes this collaborative: you can always ask who attended what, and combine the values however you like.

Revisualizing shows it. Each person has a view that decides how to combine the values, what colour they become, where they are drawn and when.

Capture and recording never stop, whatever a view is showing. A trigger only controls what a person sees.

Attaching to objects

attachAttention(session, object, options) works on any object with at least one mesh under it, including groups and instanced meshes. It merges the meshes into the object's local space and voxelizes the result with a bounding volume hierarchy, keeping only cells that touch the surface.

import { attachAttention } from "caav/three";

const attached = attachAttention(session, object, {
  id: "terrain",   // defaults to the object's uuid
  voxelSize: 0.05, // in the object's local units, defaults to 1/32 of its longest side
});

Attention is stored in the object's local space, so it stays on the object when the object moves, rotates or scales.

Voxel resolution

Smaller voxels show finer detail, but each view does more work and each voxel takes longer to fill, since a gaze spends less time in any one of them. Sparse objects need bigger voxels: a head ray mostly passes between the points of a scatterplot, and a voxel only exists where it touches geometry. The playground uses 0.08 for its surfaces and 0.1 for its scatterplot of 100 points, spheres of radius 0.05 in a box 2 wide. Change the size at any time:

attached.setVoxelSize(0.1);

Views rebuild themselves on their next update. Recorded attention for that object starts again from zero, because the old values belong to voxels that no longer exist. A miniature that someone moved stays where they put it.

Several objects

Attach as many as you like. Each person's attention goes only to the nearest voxel along their gaze, so one object can block the view of another. Each object needs its own view for each person.

People and sources

session.registerViewer adds a person. The library calls them viewers.

import { headRaySource } from "caav/three";

session.registerViewer({
  id: "a",
  colour: "#3d8bff",
  label: "Arvind",                  // optional display name
  source: headRaySource(camera),
});

Nothing limits the number of people. The paper notes that telling people apart by colour works well for 2 to 4, and gets harder beyond that.

Change a person's colour by assigning session.viewers.get("a").colour. Calling session.removeViewer(id) forgets that person and everything they recorded.

Other sources of attention

A source is anything with a sample() method that returns a world-space ray, or null when there is none, for instance when tracking is lost. This is how you would connect an eye tracker:

session.registerViewer({
  id: "eyes",
  colour: "#17a67e",
  source: { sample: () => eyeTracker.gazeRay() },
});

Recording

These options go to createSession, and any of them can change later through session.configure. Changing them keeps recorded attention.

import { halfLifeDecay, noDecay } from "caav";

session.configure({ influenceRadius: 2, decay: halfLifeDecay(60) });
session.configure({ decay: noDecay });
session.reset(); // clears recorded attention and keeps everyone registered

You can read the values directly. Each one is a Float32Array with one number per voxel:

const values = attached.target.field.channel("a");
const gaze = session.gaze("a"); // { target, voxel, distance, point }, or null

Accumulators

An accumulator combines everyone's values at a voxel into one number, which sets how opaque that voxel is. Each view chooses its own.

threshold on a view sets the point below which a voxel stays transparent, and defaults to 0.05.

Presentations

Each person's view draws attention in one of four ways, following the paper's design space. Switch with view.setPresentation(kind).

World-space presentations stay put in the shared space, so people can point at them. Camera-space ones stay readable from wherever a person stands, at the cost of that shared reference.

The miniature starts to the person's right and stays fixed in the world until moved:

view.setPresentation("world-separated");
view.miniature?.moveTo(1.5, 1, 0);

To let people drag it, raycast against view.miniature.object. The playground's source does this.

camera-separated draws into an element you provide. Pass it as overlay, and give that element its own positioning context, for example position: relative. The panel has the class caav-panel, so you can restyle it, and a text description for screen readers.

import { createAttentionView } from "caav/three";

const view = createAttentionView({
  session, attached, viewer: "a", camera, scene,
  overlay: stage,
  presentation: "camera-separated",
  style: { surface: "#10131a", ink: "#8a93a6" }, // the neutrals drawn around attention
});

Giving each person their own view

Everything about a view belongs to that person: presentation, trigger, accumulator, colour policy and threshold. This reflects the paper's point that the presentation can be left to each collaborator.

If every person has their own scene, or their own headset, no setup is needed. If several views render one shared scene, as the playground does, give each view a layer. That puts its objects on a three.js layer that only its camera can see, so Person A's voxels do not appear in Person B's view. Keep your shared content on layer 0, which cameras see by default.

import { alwaysOn, explicitTrigger } from "caav";
import { createAttentionView } from "caav/three";

const viewA = createAttentionView({
  session, attached, viewer: "a", camera: cameraA, scene, layer: 1,
  presentation: "world-embedded",
  trigger: explicitTrigger(),
  accumulator: "sum",
});

const viewB = createAttentionView({
  session, attached, viewer: "b", camera: cameraB, scene, layer: 2, overlay: stageB,
  presentation: "camera-separated",
  trigger: alwaysOn(),
  accumulator: "difference",
});

Changing a setting affects one view only:

viewB.setPresentation("world-embedded");
viewB.setAccumulator("max");
viewB.setThreshold(0.2);

Triggers

A trigger decides when a person sees attention. Each view has its own, and view.setTrigger swaps it.

Always on shows attention all the time. The paper warns that this risks the Midas touch, where people's own gaze keeps setting things off, and reinforcing loops.

import { alwaysOn } from "caav";

view.setTrigger(alwaysOn());

Explicit shows it when the person asks. HeedVision uses this, with a controller button. Connect request to whatever input you have:

import { explicitTrigger } from "caav";

view.setTrigger(explicitTrigger());
button.onclick = () => view.trigger.request();     // toggles
holdStart.onclick = () => view.trigger.request(true);
holdEnd.onclick = () => view.trigger.request(false);

Implicit lets the system decide. By default attention appears once a person's gaze has rested for dwell seconds or someone else looks at the same region, and stays for hold seconds after that ends, so it does not flicker.

import { implicitTrigger } from "caav";

view.setTrigger(implicitTrigger({ dwell: 1.5, hold: 2 }));

Pass when to write your own rule. It receives how long the gaze has rested and how many others share the region:

import { implicitTrigger } from "caav";

// Only when at least two other people are looking at the same spot.
view.setTrigger(implicitTrigger({ when: (context) => context.joint >= 2 }));

Triggering from events

An explicit trigger can also be driven by the events below. This shows attention to Person A for as long as they are sharing a region with someone:

import { explicitTrigger } from "caav";

view.setTrigger(explicitTrigger());
session.on("jointstart", (event) => {
  if (event.viewer === "a") view.trigger.request(true);
});
session.on("jointend", (event) => {
  if (event.viewer === "a") view.trigger.request(false);
});

Shared colour

Each person has a colour. A voxel only one person attended takes that person's exact colour. When two or more overlap, the view's colour policy decides.

Interpolated is the default. The colours blend, weighted by how much attention each person gave, so the result shifts as attention shifts. Blue and orange at equal attention make the pink #db6cc2.

import { interpolatedColours } from "caav";

view.setColourPolicy(interpolatedColours());

The blend happens in OKLab, a colour space built around how colours look. Lightness and chroma are averaged, and hue takes the circular mean, which goes the short way round the colour wheel. Averaging RGB directly would turn blue and orange into grey. Three or four people use the same formula, and the order of people makes no difference. If their hues cancel out, the result fades to a neutral grey instead of jumping to an arbitrary hue. A blend does not tell you how many people overlapped, so use the count accumulator if that matters.

Choosing a colour is optional. Any overlap, however many people, takes the colour you give:

import { explicitColours } from "caav";

view.setColourPolicy(explicitColours({ shared: "#ec4f9c" }));

Both policies count a person as attending a voxel when their value is at least 0.05. Pass threshold to change that.

You can write your own policy: an object with a kind and a resolve(viewers) method that returns an sRGB colour or null.

Events and spatial tooltips

The session reports what people do as events, so your code can react to attention in space. session.on returns a function that stops listening.

Every event also has viewer, target, voxel, point in world space, and localPoint, the voxel's centre in the object's own space. Joint events fire once for each person involved, so each person can react separately.

Spatial tooltips

createSpatialTooltip pins a text label to a point in space. It follows the point as the camera moves, and hides when the point is off screen. Together with an event, that makes a tooltip triggered by attention.

import { createSpatialTooltip } from "caav/three";

const tooltip = createSpatialTooltip({ camera, overlay: stage });

session.on("dwell", (event) => {
  if (event.viewer === "a") tooltip.show(event.point, `Rested here for ${event.seconds.toFixed(1)} s`, { seconds: 3 });
});
session.on("gazeleave", (event) => {
  if (event.viewer === "a") tooltip.hide();
});

// In your frame loop:
tooltip.update(dt);

It is a DOM element with the class caav-tooltip, added to the overlay you give it, so restyle it with CSS. It has role="status", so screen readers announce it.

Tooltips for specific places

To react only to certain places on your data, check localPoint. Because it is in the object's own space, this keeps working when the object moves.

const peak = [0.1, 0.9, -0.2];

session.on("dwell", (event) => {
  const [x, y, z] = event.localPoint;
  if (Math.hypot(x - peak[0], y - peak[1], z - peak[2]) < 0.3) {
    tooltip.show(event.point, "The highest peak: 2,400 m", { seconds: 4 });
  }
});

React Three Fiber

caav/react wraps the same calls in hooks. It needs React 19 and @react-three/fiber 9. There is a working example.

import { useRef } from "react";
import type { Mesh } from "three";
import { useAttachAttention, useAttentionSession, useAttentionView, useHeadTrackedViewer } from "caav/react";

function Terrain() {
  const mesh = useRef<Mesh>(null);
  const session = useAttentionSession({ captureRate: 10, influenceRadius: 1.5 });
  const attached = useAttachAttention(session, mesh, { voxelSize: 0.05 });
  useHeadTrackedViewer(session, { id: "you", colour: "#3d8bff" });
  useAttentionView(session, attached, "you", { presentation: "world-embedded" });

  return (
    <mesh ref={mesh}>
      <torusKnotGeometry />
      <meshStandardMaterial color="#6c7078" />
    </mesh>
  );
}

Pass new values to useAttentionView and the view updates in place. Keep trigger objects stable between renders, for example by creating them outside the component or with useState, because a new one resets its state.

API reference

caav

The core has no rendering dependency, so you can use it with another engine by building a VoxelGrid yourself and passing a toLocal function that moves rays into the grid's space.

caav/three

createAttentionView takes session, attached, viewer, camera and scene, and optionally presentation, trigger, accumulator, colour, threshold, layer, overlay and style. With nothing else set, you get HeedVision's own configuration: world-space, embedded, and explicitly triggered.

caav/react

Limits

• No network sync. All people share one session in one page. To connect headsets, send gaze rays or recorded values over your own connection.
• Not yet run on a headset. Head rays come from a three.js camera and an XR camera should fit, but WebXR has not been tested.
• No eye tracking built in. The source interface is ready for it.
• Fixed geometry. Voxels are computed from the geometry when you attach. For an object that changes shape, call setVoxelSize again, which also clears its attention.
• One attention region per sample. The paper's data-agnostic capture, which would record every voxel a ray passes through, is not implemented.

Where this differs from the paper

• The paper blends colours additively. Here the default blend is the OKLab one described above, and you can choose a fixed colour instead.
• The paper's region of influence is a single voxel, which is the default here too. influenceRadius lets you widen it.
• The paper's system uses Spatialstrates for syncing and was evaluated in world-space, embedded, explicit mode. The other three quadrants were proofs of concept there, and they are unevaluated here too.