Hey guys,
Right now the cloth solver has no dedicated vertex-to-face or edge-edge self/inter collisions. I worked hard to get that into the beta release but didn't make it in time since it's a very complicated task to get right. And now with the size of the todo list with regards to fixing bugs people have found and other features that need priority, the dedicated cloth mesh collision solver is pushed further back...
The solution at the moment is to rely on vertex-to-vertex collisions using a Particle Physics operator, and ensuring your cloth mesh is dense enough so that faces don't pass through each other. The Particle Physics operator is integrated into the main binding solver, so it will play nicely with cloth sims. A few key things to note:
Generally for cloth you want your flow time steps to be 1/4 or higher. Also, do not be afraid to crank your Particle Bind Solver substeps up really high. 100 or more for hires cloth. However, the higher you put your flow time steps, the lower you can put your PBS steps. For example, 1/4 flow step and 100 PBS steps can be changed to 1/8 flow step and 50 PBS steps without losing accuracy.
Then ensure that the collision radius of your Particle Physics operator is enough so that cloth vertices won't pass through each other. Cranking up the Particle Physics collision tolerance value will also help solve collisions with higher simulation substeps, as more collisions will be caught between fast moving particles. There are display settings you can use to preview your collision radius size. Also make sure to turn on "ignore binding neighbors" so vertices of the same face don't collide with each other, which is unnecessary to get proper collisions and can lead to visual artifacts.
With the right settings (ensuring your cloth is hires and the Particle Physics collision radius is roughly half the length of any given face on your cloth mesh), you can get very decent results with this setup, as seen in this post:
https://www.instagram.com/p/Bu1BEPVnwfy/
I have updated the example scenes with a specific cloth self-collisions demo that you can grab here, to see such a setup:
http://apps.tysonibele.com/tyflow/scenes...scenes_002