Clothmesh Polygon size

[tyFlow]

The self-collisions in the example scene are generated with a Particle Physics operator, which uses the mesh vertices as colliders with a given radius...so in your swapped meshes, any faces smaller than that radius will bulge as the verts try to move away from each other. In the (distant) future, tyFlow will get proper vertex-to-face and edge-to-edge cloth self-collisions. For now, vertex-to-vertex is the only option.

For now, the solution may be to subdivide your mesh so all faces are roughly the same size, then set the Particle Physics operator radius to roughly have the length of any given face on the mesh.