[Unity Plugin] Support for SDF plugin

[fayezsalka]

The feature, motivation and pitch

Hi!

We’ve been looking to use Mujoco to simulate robots performing complex bimanual tasks in Unity, e.g screw and bolt action, however such use cases typically require SDFs and it’s nice to know that Mujoco has a plugin for it, but can’t seem to find any hints on how to get it to work with Unity. Are there any plans to have it supported?

Alternatives

No response

Additional context

No response

[fayezsalka]

I found that Unity has support for convex decomposition which can go around some of the issues of non convex mesh collisions, however I wonder what would be the limits of this approach vs using SDFs for collision detection?

https://github.com/Unity-Technologies/VHACD

[Balint-H]

Hello, I'm working on finishing up a PR to support MuJoCo plugins in Unity, primarily with the elasticity plugin in mind. The features should also work partially for the SDF plugin, with some additional work needed for it beyond that (e.g., to render the SDFs). I've been slightly delayed due to personal reasons, but the plan is to release at least the elasticity plugin support in the first half of December (from which you/we could add SDF support).

Decomposition may be sufficient for your needs, but depending on the resolution you need for your concave mesh it can inflate the collision detection costs. Limiting what can collide with what (e.g. through contact filtering) could alleviate some of this if it is a problem. Don't forget if you split your mesh up you will need to add separate geoms for each submesh if you want MuJoCo contacts with them.

[fayezsalka]

Awesome, that's great! Looking forward to it.

[fayezsalka]

Btw, with the elacticity plug in supprted, would it be possible to simulate cloth inside Unity (e.g with skinned mesh renderer)? That would be final nail in the coffin for us thinking about using Isaac Sim :D

[Balint-H]

That's the idea! I already implemented a mesh renderer to model the MuJoCo hammock scene in Unity (one that only works for sheet cloths), which was my starting point on supporting the elasticity plugin.

(Beach and sea from the Unity HDRP example scenes)

[fayezsalka]

Super cool!! Can't wait to do laundry folding with this + Quest :D