Cloth collision does not come in correctly.

[tyFlow]

Since you're using Particle Physics for the collisions, what you're seeing is the result of point collisions between particles....your grain particles have a radius but so do your cloth particles....so the gap between the two is just the minimum distance between the particle radii before they're considered touching.

You can shrink the radius of your cloth particles with a Scale operator. Just place it after the Cloth Bind operator and set it to something like 40%.