I believe Sam has three chronologies of memory that all feel valid. Feel like ‘his life’.
One: He can feel his life of 35 years. The person he was when Death put in the wall. That’s coming up on 35 years of chronological memories with about a year and a half missing.
Two: At the same time, when he looks at his memories as Soulless, he gets those memories, and that feels as continuous and true as the chronological experience of waking with the wall.