Writing, goals, and how life turns out

When I was much younger, my expectations of my future life were high. I thought that one day, I'd be a writer, at least be able to make a living as a novelist. I thought that I'd quit my job working for the U.S. Postal Service, and be able to have a lucrative and fulfilling career as a novelist. Now at 52, I look back and am dismayed and depressed that the dream has gone unfulfilled. Of course, one has to realize that other more important things may have kept me from accomplishing that dream. Raising a family and making sure that they were taken care of (which I wouldn't trade for all the writing in the world), doing my duty to God and other responsibilities, exercising for good health, and of course, my full-time job. All those took me away from writing for long stretches of time.
I'm not the kind of writer who can easily sit down for five minutes and have something good written. I need at least an hour or two of free time, which throughout my adult life, has been difficult and sometimes impossible to find. And I get down on myself to easily, whether or not it's the writing aspect or the exercising aspect. Somehow, through all that though, I've been able to write one novel (still unpublished), and complete large chunks of two others (still in the process of completing them), a bunch of short stories, and numerous articles on my real passion in life--outdoors activities. So, I've written. I've even published one of my short stories for money.
But that isn't what my goal originally was. My goal was to be making a living writing. My goal was to be out of the postal service at a young age, living my dreams. So what am I trying to say? That sometimes dreams don't come true. That sometimes a modification is in order, and in order to be happy with those modifications, we need to decide to. We need to try and be happy in other ways. I think I can do that. I just need to figure some things out and decide where I'm going with my writing. Should I continue floundering in the world of fiction, or should I go where I've been more successful in my writing career, writing the outdoors articles? I'm leaning toward the outdoors articles, but then a part of me will always be wondering, could I have been good enough to be a novelist had I only persevered?