Data Visualization Design Pattern Leveraging 'extras' Attribute

glTF
1

Background: I’m opening up a discussion that will hopefully influence glTF publishers/exporters to consider data visualization requirements with 3D assets. Before opening up the specific issue detailed below with the Blender developers, I wanted to document the issue and get feedback on possible other considerations.

Use Case: 3D visual assets can be ‘data driven’ by data assets made available in at least two primary ways by a rendering engine.

  1. The data assets are statically associated with the 3D asset and incorporated into the glTF file.
  2. The data assets are dynamically associated (by referencing/lookup scheme) with the 3D asset.

This design pattern is used in a prototype application:

Capture
Capture.PNG1931×935 102 KB

In this prototype, individual meshes with individual materials (not shared) are dynamically ‘driven’ with data that corresponds to the 3D images’ sub-components. The 3D images were developed in Blender and then exported with the glTF export tool.

To identify the visual sub-components associated with the data, the ‘extras’ attribute for the meshes were associated with a ‘visualize’ property. That along with the ‘name’ property was sufficient to accurately identify the sub-component and its data association. An good solution for a gen 1 prototype.

Issues/Discussion:

Experimenting with static bound data revealed some issues; at least with Blender. While it supports Custom Properties, it exports a JSON object like the following; in a format that is not immediately parse-able into a JSON object:

        "extras": {
            "visualDatum": "{cost: 5, price:8}"
        }

visualDatum’s value is a string which is not parse-able with modern browsers’ JSON method because the property names are not within double-quotes.

Recommendation(s):

  • gltF exporters should export objects that don’t need to be parsed into JSON objects.

  • Suggested property names to use for a generalized visualization design pattern:

  • “visualKey”: a value (string or number) that the rendering logic uniquely identify a 3D visual’s sub-component to be dynamically accessed.

  • “visualDatum”: an object that represents datum associated with the 3D visual’s sub-component.

2

I think the issue here is that Blender’s Custom Properties are simply key/value pairs – it has no concept of nested data, except if you’re adding the properties through its Python API: https://easyblend.org/html/data_system/custom_properties.html#python-access

The glTF format does support nested .extras data for sure, it’s just that Blender is giving the exporter a string here.

3

Thanks.

A Python script would be a good workaround and practical as well. I don’t see a developer manually going 1 by 1 updating the custom properties for a 3D Visualization.

I’ll try developing a script that will dynamically populate Custom Properties from a .csv file with matching ‘name’ content.

I’m going to give it shot and post back here when I’m done…assuming I have link privileges by then.

Update: Script can be found by searching github for:

  • mariodelgadosr/AddBlenderCustomPropertiesFromCSV

Related FYI:

I’ve detailed a design/programming pattern (search github for ‘mariodelgadosr/datavisual’) that utilizes the

  • Blender custom properties
  • glTF extras properties;
  • Three.js userData

(they’re all referencing value-added information) for dynamic data visualization.

FYI #2 that may affect some developers:

Three.js sanitizes name property values. So a Blender name ‘Object.001’ is sanitized to ‘Object001’. The github/documentation goes into more details about this.

1 Like
4

The general idea of augmenting the pure visualization data with “external” data has several use-cases. The most obvious one being the visualization of CAD data, where certain manufacturing information should be visualized. For example, indicating a property like a failureRate of a certain part using a green-to-red-color-coding. But one can imagine more sophisticated usages as well, like the options to filter or query the elements that are visualized, based on these properties.

We had some discussion internally via mail about this topic. First, as an overview of the most relevant links for everybody who is interested:

  1. The main page of an implementation of these ideas is here:

    • http://mydatavisualizer.com/ .

    It also links to a demo page where augmented demo data sets can be selected or uploaded

  2. The general concepts of the approach are explained in this notebook:

    • https://observablehq.com/@mariodelgadosr/datavisual-data-visual-design-pattern-for-webgl-3d-assets

    It includes a live demo of an example use case, and a more detailed description of the query/join operations that are supported

  3. The main GitHub repo is here:

    • https://github.com/MarioDelgadoSr/DataVisual

(Not having the permission to add links here is a bit annoying, but understandable as a measure of spam prevention…)

I already asked some questions during our discussion via mail. These questions may seem naive or overly critical, but hopefully bring up some discussion points. The main question had been about the exact relationship between the underlying data (e.g. basically the data from CSV files), and glTF itself. I imagined two options for establishing this connection: 1. Via an ID in the extras property, or 2. By integrating the actual data in the extras property

The first one could boil down to something as simple as this:

"meshes" : {{
   "extras": {
     "partId": "part1234"
   },
   "primitives": ...
}}

and whenever the “external data” is required by the viewer, it is queries, maybe via some REST/database lookup using the partId.

The other option would be to integrate the actual data in the extras object. I can see that there are technical hurdles and possible drawbacks for relying on the ID only. But putting the data into the extras object may have drawbacks as well:

  1. The extras data is unstructured. There is no schema specifying what type the data has.
  2. The data is highly vendor specific
  3. The data may increase the size of the glTF asset considerably.
  4. One has to ensure consistency of the extras data with the original data source

Points 3. and 4. can probably be alleviated by ensuring that the glTF only contains “Low Volume, High Value” data - that is, roughly speaking, only the additional data that is required for the visualization and the queries (glTF is still intended as a (pure, compact) transmission format). Points 1. and 2. might be an argument for a custom extension…

The resources linked above certainly cover some related questions, but I’ll have to invest more time to understand this thoroughly. Similarly, from the description in the notebook (as far as I understood, and please correct me where I’m wrong), the DataVisual is not directly related to glTF. Instead, the DataVisual is created based on a three.js scene. So in order to better understand the data flow, I’d have to take a closer look at three.js as well…

5

An extended article explaining the connection between DataVisual and glTF has been published at

6

Here are further examples on observable notebooks showing uses-cases of DataVisual:

1 Like