Cuda Solver info, Error Code: 13

[JonY]

Hi

Would like to find out how to resolve this issue of error code 13 when I enable CUDA collision solver 

My workstation setup:

Dual Xeon 2699v4 processors
RTX 2080TI
256GB ram

3ds MAX 2021

RTX 2080 TI: Nvidia driver 461.72 latest version

The drivers are install in C:\Program Files\Autodesk\3ds Max 2021\Plugins, where the Tyflow dlo is installed. 

Any suggestion is greatly appreciated as without the CUDA collision, my cloth simply passes through the collision object.

I have also checked Compatibility Mode under GPU, closed 3ds and reopened but still with same error code

Jon

   

[JonY]

Some observations:

a.) When CUDA collision solver enabled, I am able to run up to around 38 frames of collision from my full frame length of around 1200 frames. From 39 frame onwards, the CUDA solver will crash, with error 13 and no collision happened thereafter. Have to close 3ds and reopen again

b.) Force Affect - played around with Force Effect => Velocty Affect => Affect Both => change Multiplier value to a lower value. Wahlah! CUDA solver works throughout the whole 1200 frames with no crash. Perhaps when Force is too high, it penetrate the collision object which trigger a bug within CUDA solver? Just a thought.

Although I need to finetune the cloth effect further so that it does not look too rubbery - any suggestions on this?

Frame step still using Default value of 1F

[tyFlow]

Yea what you're seeing is the IZ solver transferring over to CUDA at only a certain threshold, because up until that point it's faster to do small local collision solves on the CPU. Higher forces = more simultaneous collisions = greater chance of falling back onto CUDA.

The difficulty in debugging these issues is that they're hardware dependent....because I have systems with different GPUs/configurations that do not run into any trouble during solves. That said, I do have access to a 2080ti system and I will do further testing on it when I get a chance.

Anyways, thanks for posting your setup/info...hopefully we can track these issues down.

[JonY]

Yea what you're seeing is the IZ solver transferring over to CUDA at only a certain threshold, because up until that point it's faster to do small local collision solves on the CPU. Higher forces = more simultaneous collisions = greater chance of falling back onto CUDA.

The difficulty in debugging these issues is that they're hardware dependent....because I have systems with different GPUs/configurations that do not run into any trouble during solves. That said, I do have access to a 2080ti system and I will do further testing on it when I get a chance.

Anyways, thanks for posting your setup/info...hopefully we can track these issues down.

If it helps, I can email you the scene I have to assist you further in optimising the plugin?

Also, what's the best approach for getting a cloth like simulation, as I have wind effector within Force and it looks very rubbery at the moment.

[tyFlow]

Sure, if you send the scene that could be helpful.

As for cloth, make sure you've got enough bind solver steps and reduce your overall sim steps as well. Lower steps = stiffer cloth.

[frascow]

I'm experiencing exactly the same problem. The CUDA is correctly loaded but after some frames of calc with the most simple cloth setup, it's no longer available giving me the error code 13

RTX 2080 TI: Nvidia driver 461.72 latest version
AMD Ryzen 9 3900x
64Gb ram
---------------------------------------------

Would like to find out how to resolve this issue of error code 13 when I enable CUDA collision solver 

My workstation setup:

Dual Xeon 2699v4 processors
RTX 2080TI
256GB ram

3ds MAX 2021

RTX 2080 TI: Nvidia driver 461.72 latest version

The drivers are install in C:\Program Files\Autodesk\3ds Max 2021\Plugins, where the Tyflow dlo is installed. 

Any suggestion is greatly appreciated as without the CUDA collision, my cloth simply passes through the collision object.

I have also checked Compatibility Mode under GPU, closed 3ds and reopened but still with same error code

Jon


[/quote]

[tyFlow]

Could you guys confirm what your WDDM TDR delay setting is set to?

Instructions for how to find it are here if you have NSight installed:

https://docs.nvidia.com/gameworks/conten...covery.htm

Or here if you don't:

https://www.pugetsystems.com/labs/hpc/Wo...ience-777/

Further instructions here:

https://docs.substance3d.com/spdoc/gpu-d...45489.html

[JonY]

I can only speak from the current setup I have as your problem sounds similar to mine:

a.) Do you have Force added with external forces? If yes, try playing around with the Multiplier value under Force to see if that helps. By changing the Multiplier value for my scene to a lower %, CUDA works with no code error 13

b.) If you want, send me your scene and I can help see if I can replicate your problem?

   

---

Could you guys confirm what your WDDM TDR delay setting is set to?

Instructions for how to find it are here if you have NSight installed:

https://docs.nvidia.com/gameworks/conten...covery.htm

Or here if you don't:

https://www.pugetsystems.com/labs/hpc/Wo...ience-777/

Further instructions here:

https://docs.substance3d.com/spdoc/gpu-d...45489.html

Mine is in its default setting of 60. Although the doc say change it to 10. Will change it to 10 and see if it helps

[tyFlow]

No, higher is better in this case

At least we've narrowed down that that's not the cause.

[JonY]

No, higher is better in this case

At least we've narrowed down that that's not the cause.

[tyFlow]

What version of windows are you guys running?

I tested on a machine with a 2080ti and could simulate all the CCCS example scenes without issue, with the latest NVidia drivers installed. However, it's a windows 8 machine...perhaps the problem is only happening on Windows 10?

[JonY]

What version of windows are you guys running?

I tested on a machine with a 2080ti and could simulate all the CCCS example scenes without issue, with the latest NVidia drivers installed. However, it's a windows 8 machine...perhaps the problem is only happening on Windows 10?

Windows 10 and 3ds 2021.

[frascow]

Hi there,

My values on TdrDdiDelay and TdrDelay are already 60, Redshift renderer asked long time ago to add those lines to the registry,
About OS, Windos 10 (Buid 18363) pro, 3dsMax 2021 and 3dsmax 2020

At least it seems that we both have exactly the same problem.

thanks!

[JonY]

Hi Tyson

I have attached a simple scene that generate the cuda error code 13 on my machine.

In the 3ds max scene, a simple cloth sticking to a box, and colliding with a box.

When the physical size of collider box is large, cuda solver works fine. But when the collider box becomes smaller (thinner), that is when cuda solver runs into problem - ie no collision after N frames. This correlate to your IZ solver point raised in your earlier post.

What I observed is rather straight forward:

. When collider object is large enough in size to collide with cloth, cuda solver works fine.

. When collider object becomes finer and intricate in size, cuda solver will crash after N frames. The way round this to what I tested was to play around with the Force parameters, by reducing the Multiplier value mentioned in my earlier post. This at least works for the moment.  


Anomaly observed in Point Cache (for suggestions):

. Upon a successful cuda solver cache of Tyflow with Mesh as its last operation (with Render off), I inserted a Point Cache to record the full simulation as individual cache files.
. But when I reload the cache on a copied plane from the original, either:
   . the new plane with loaded Point Cache disappears and not seen in viewport
or
   . the coordinates of the loaded Point Cache gets distorted, eg rotated and away from the original coordinates where cloth simulation was performed


  Project_Export Mesh.max (Size: 780 KB / Downloads: 246)

---

Hi there,

My values on TdrDdiDelay and TdrDelay are already 60, Redshift renderer asked long time ago to add those lines to the registry,
About OS, Windos 10 (Buid 18363) pro and 3dsMax 2021

At least it seems that we both have exactly the same problem.

thanks!

Frascow, you may want to see what my last post that discussed the observations and workaround I did in re to cuda crashing.

[frascow]

Frascow, you may want to see what my last post that discussed the observations and workaround I did in re to cuda crashing.

Hi JonY

I actually tested your workaround, setting the multiplier on a very low value and it works, unfortunately on the scenes I usually work the sense of scale and right physics is completely lost with this trick. Thanks for it but I really need a better solution, I'm sure Tyson will find it, meanwhile I prefer to work with the version 0.16108 that I keep running on 3dsmax 2020

__________________________________

I have attached a simple scene that generate the cuda error code 13 on my machine.

In the 3ds max scene, a simple cloth sticking to a box, and colliding with a box.

When the physical size of collider box is large, cuda solver works fine. But when the collider box becomes smaller (thinner), that is when cuda solver runs into problem - ie no collision after N frames. This correlate to your IZ solver point raised in your earlier post.

What I observed is rather straight forward:

. When collider object is large enough in size to collide with cloth, cuda solver works fine.

. When collider object becomes finer and intricate in size, cuda solver will crash after N frames. The way round this to what I tested was to play around with the Force parameters, by reducing the Multiplier value mentioned in my earlier post. This at least works for the moment.  


Anomaly observed in Point Cache (for suggestions):

. Upon a successful cuda solver cache of Tyflow with Mesh as its last operation (with Render off), I inserted a Point Cache to record the full simulation as individual cache files.
. But when I reload the cache on a copied plane from the original, either:
   . the new plane with loaded Point Cache disappears and not seen in viewport
or
   . the coordinates of the loaded Point Cache gets distorted, eg rotated and away from the original coordinates where cloth simulation was performed

---

Hi there,

My values on TdrDdiDelay and TdrDelay are already 60, Redshift renderer asked long time ago to add those lines to the registry,
About OS, Windos 10 (Buid 18363) pro and 3dsMax 2021

At least it seems that we both have exactly the same problem.

thanks!

Frascow, you may want to see what my last post that discussed the observations and workaround I did in re to cuda crashing.

[JonY]

Frascow, you may want to see what my last post that discussed the observations and workaround I did in re to cuda crashing.

Hi JonY

I actually tested your workaround, setting the multiplier on a very low value and it works, unfortunately on the scenes I usually work the sense of scale and right physics is completely lost with this trick. Thanks for it but I really need a better solution, I'm sure Tyson will find it, meanwhile I prefer to work with the version 0.16108 that I keep running on 3dsmax 2020

I have fine tuned the physics parameters so that I can get something near to what I was expecting, this workaround works for me at the moment  So ain't complaining for this

[JonY]

Tried again with the latest build of TyFlow, setting all values to default. Same issue/problem again.

Had also tried 0.16108 as frascow suggested, it seems to be working still for 2021.

Was wondering if this aspect is being looked at for resolution?

[tyFlow]

Hang tight for the next build....as of this evening a big CUDA bug happening on 2080ti (and newer) GPUs was found and fixed. Since I don't have a 30XX GPU to test I don't have a way of confirming that it fixes all the issues present on 30XXs too...but it fixes a pretty major one, and there's a possibility the bug in question being fixed will clear up some other issues too.

[JonY]

Hang tight for the next build....as of this evening a big CUDA bug happening on 2080ti (and newer) GPUs was found and fixed. Since I don't have a 30XX GPU to test I don't have a way of confirming that it fixes all the issues present on 30XXs too...but it fixes a pretty major one, and there's a possibility the bug in question being fixed will clear up some other issues too.

---

Hang tight for the next build....as of this evening a big CUDA bug happening on 2080ti (and newer) GPUs was found and fixed. Since I don't have a 30XX GPU to test I don't have a way of confirming that it fixes all the issues present on 30XXs too...but it fixes a pretty major one, and there's a possibility the bug in question being fixed will clear up some other issues too.

Which build will that be likely be seen in? Thanks for the hard effort

[frascow]

Unfortunately I'm still having the same error on the last build...

On the first example of CUDA cloths it gave me the error on frame 10, Is there is anyway I can help fixing the issue please tell me.

Thanks

[nearleyg]

Hey guys, also having this issue.
Interestingly I've only recently started having it after upgrading from tyflow v0.16107 to v0.16114. I'm using a 2080ti.

It's definitely something with the IZ steps switching to CUDA. It can be worked around by just cranking up the IZ threshold which lets my sim run fine without going crazy/cuda crashing.