My fiancé of over 3 years constantly says... "remember when we did "such and such"......and I will say......."It WASN'T with me"....

I constantly confuses, forgets? Who he did what with. In the...