Anyone ever tried feeding preprocessed 3D Volumetric Data for training LLM?

Hi Everyone,
As a background, I’m a VFX & CG Supervisor, as well as CG Artists. I’ve dealt with lots of 3D data for CG animation and VFX.
And now I’m embarking into a new ventures, which deals in digitizing human based data, especially movement.
I’m on my way to capture large scale amount of movement and volumetric data, and storing it in industry standard 3d format. Already done prototype.
I would like to process such data to train LLM. The goal is to have a system that can identify, analyze, query, manipulate and such; of all given volumetric 3D data.
I’m wondering if anyone here have dealt with such project. Especially processing 3D matrix / voxel data to LLM training data.
Cheers !

Hey champ, and welcome to the community!

Large Language Models, are generally best at text based stuff, for visual stuff I’d recommend using a diffusion model instead :laughing:

Ps.: if your model also need to be rigged at some point, I’d recommend you have a look at openPose.

1 Like

Hi there

It’s for analysis and classification of animated models or animated volumetric shapes. So the LLM can tell user what kind of movement style that it looks at. For example … education AI assistant for motion performance

And not all 3d model needs to be rigged once the animation is baked per vertex.

You’ll likely need multiple different AI systems for that, an LLM can definitely handle the everything that’s language based, aka taking queries and talking to users :laughing:

You’ll need to build an entirely different AI for classification of movement data and such, an LLM won’t be able to do that particularly well, no matter how much you fine tune it :sweat_smile:

Yeah I agree, but it might be easier for an AI classifier to work with a bunch of stick figures without having to interpolate all the vertex data :laughing:

1 Like

:smile: sorry my questions probably out of scope.

No worries, there’s various different projects that is already working on this problem.

Do a Google search on stuff like OpenPose, PoseNet and MoveNet and take a look, you might find something you’d like :wink:

1 Like

Whoa , thank you so much ! Really appreciate it.

1 Like

You’re welcome!

I hope you make something super cool! And make sure to come back and show us the results!

Personally I would take all the data you’ve collected and one of the previously mentioned models, and wrap it with the assistant API so you can interact with it conversationally :laughing:

1 Like