Problems with physx collison

[tyFlow]

Hmm, well PhysX Shapes have a penetration offset setting that you can use to give each shape a little penetration buffer, but you're never going to run into a situation where colliders can fully intersect like in your second image and the simulation will remain stable.

Post-processing your mesh with relax modifiers, prior to input into the sim (like your first image) will be your best bet.