That is the effect of the memories that they have taken. Even if you come to a logical conclusion of what is missing, you'll experience these same effects and whatever conclusion you've made will be instantly unreachable again.

And aside from that, there are chronological variations too. Suzaku recalls events that I do not without feeling the effects of the taken memories. I expect there are similar gaps for Gino and Nunnally.