If you disable realtime and it's still slow then it's something else, since subframes aren't calculated on whole frames. The subframe thing was just my guess since I haven't seen your scene or cache.
Are you sure scrubbing is really faster?
Are you sure scrubbing is really faster?