The Graduate star has two children from his first marriage to Anne Byrne and four with his current wife Lisa Gottsegen.
However, Hoffman admits he should have been more involved as his kids were growing up, and is devastated not to be able to recall key events from their childhoods.
He tells Britain's Daily Mirror newspaper, ""The most terrifying thing for me is when I look at pictures of my kids growing up. I've been married for 31 years, raising all these kids. The youngest is 25 and my wife remembers exactly what happened the day the picture was taken, exactly where everyone was. I can't remember any of it.
""It's terrifying because you realise you were there, but you weren't there. Because if you were really engaged in your life you would retain those memories, and I didn't. The oddest thing is that I'll be surfing the channels on TV and suddenly see a movie I did. I'll stop and watch a few minutes and remember exactly where I was that day, exactly what the problems were filming.
""Everything about that day will come back to me and I say to myself, 'Why don't I have that kind of memory about my life? Why is it just my work?'""