Everything that hinders us from moving forward is rooted in fear. In memoirs, all authors experience fear. Fear of our own story, of what we remember is 'correct', and how our words will be preserved. Questions like 'Will I be sued?' is wildly discussed, for instance. I believe the only way to combat fear is with love, and as we write alone we must master self-love. It becomes as important to the memoirist as mastering the art of creative writing, and it can take just as long. So, love yourself through the process of writing. Start with observing your own feelings and behaviour.
My technical answer is to; stop thinking, stop writing and start revising. Read what you've already written, lobe what you love and throw out or rewrite the rest.
I have 3 simple answers. 1. Frank McCourt's Angela's Ashes. 2. Mary Karr's Liars Club and 3. an overpowering need to write my own story and make sense of my life. My memoir took 7 years to write, revise and publish.