Make sure the vertices of your cloth are packed tight enough together so that the collision radii of the grains collide against them and don't pass by...
Also, if your Particle Physics operators for your cloth and grains are in separate events, disable the "process only this event" checkbox in each.
Also, if your Particle Physics operators for your cloth and grains are in separate events, disable the "process only this event" checkbox in each.