I am very involved in PCa work because of what happened to me. (My understanding now is that African Americans should begin PSA tests at age 40 and other men at age 50.)
I had my first PSA as part of my annual physical when I turned 50, and it looked a little unusual. So the next year I had it done again, and it looked even more unusual. They did a biopsy followed by a more thorough digital rectal exam than before and found that I had cancer in my prostate.
Of all the options that were available to me I chose surgery, which I had done in 2000. That makes me a 12-year cancer survivor. Thank you, God! (Contact me privately if you would like to know why I chose surgery over the other options. And if you'd like to know how I overcame common post-prostatectomy side effects.)
I know that a PSA test is not perfect; no medical test is (as far as I know, as I am not a medical professional myself). Any doctors worth their salt would follow a suspicious PSA with another one, and then do a variety of other tests before making a diagnosis and discussing options with the patient (and the options include active surveillance, aka watchful waiting).
No doctors worth their salt would say, "Oh, your PSA is [a certain number]. That means you have prostate cancer. Let's do surgery/radiation/chemo/hormones." A PSA number is not a magic tell-all finding.
All I know is that since prostate cancer does not have symptoms (until it has spread to the bones and then it's too late), PSA tests saved my life (and the lives of quite a few other men I know). But of course what persons do with their own bodies is up to them. A person's body is extremely personal, isn't it?
Screen, if you have it then assess the situation. Some cancers are so slow as to not be a problem, some are not. There is a concern about over treatment.
Trust me on this. You really need to keep the old body's systems checked. It's analogous to checking the underhood fluids in your car.
Far, far better to discover that your enlarged prostate (yes, I am that old) is not due to an inoperable cancer. The one you didn't get diagnosed because you didn't mind peeing half a dozen times a night.
You are wise (and fortunate to be able) to trust your personal physician, but the bottom line is that YOU are ultimately responsible for your health, not your doctor. Sounds to me like he really cares about his patients.
Learn what the signs of prostate cancer are. I'm not a doctor, and I don't put myself out there as one. But 40 does appear to be too young. Since your dad had it, you might be at increased risk, but your doctor is right: that doesn't mean you'll get it. Still, know the signs. Have anything suspicious checked out by a licensed physician.