memory

A person I work with says that research shows that memory is very unreliable, so his approach is there is no way of knowing what really happened, just work with the feelings. He said he had had experience of having to realize that his own memories (topic unspecified) were systematically distorted, but he wasn't in any way trying to lead me in that direction. I find his views very threatening. I don't want to claim that all memories are literally true, but if I can't believe that my memories average out to a reasonably accurate summary of at least part of my experience then how can I trust myself? And if I can't trust myself then how can I change my world view from the one I grew up with to one that works for me and my children? He says he supports me in believing myself. It isn't going to be easy for me to do so knowing how he sees the issue.