The research – in the British Medical Journal – looked at 1.2 million pregnancies in Sweden over 20 years.
Men who had ICSI – a treatment specifically for male infertility – had an increased prostate cancer risk.
But Prostate Cancer UK said researchers must look at a much broader age range before concluding men who have fertility treatment are at higher risk.
Researchers from Lund University in Sweden used data from national birth and cancer registers.
They looked at more than a million births between 1994 and 2014, and at cancer cases.
Most babies – 97% – were conceived naturally, and 20,618 (1.7%) were conceived using IVF, although the data does not show if fertility issues lay with the man or the woman.
Some 14,882 (1.3%) births resulted from ICSI, where a single, good-quality sperm is selected and injected directly into an egg.
ICSI was first used in Sweden in 1992, with every case recorded by the register.
‘Offer the test’
Among the natural conception group, 3,244 (0.28%) were diagnosed with prostate cancer, compared with 77 (0.37%) in the IVF group and 63 (0.42%) among those who had ICSI.
Men in the ICSI group also had a higher risk of developing early onset prostate cancer, before the age of 55.
Prof Yvonne Lundberg Giwercman, who led the study, told the BBC: “The prostate cancer numbers are quite small, but these men are very young.
“They are a small, high-risk group, and we should be following them more closely.”
She said she hoped there would be further studies to investigate why the link existed.
Allan Pacey, professor of andrology at the University of Sheffield, said: “It has been proposed that male infertility might serve as a “canary in the coal mine” for men’s health, which both men and their doctors should be better attuned to.”
He added: “It is important to be clear that this is not because the techniques of assisted reproduction go on to cause prostate cancer, but probably because the two have a common cause in some way.
“Perhaps all men who are diagnosed with a fertility problem in their 20s and 30s should be given a leaflet explaining what this might mean for them in their 50s and 60s, so that they can be aware of possible future problems, and be encouraged to visit their GP a bit quicker than they often do.”
But Simon Grieveson, from Prostate Cancer UK, said it was important not to “leap to any conclusions” on the basis of this study.
“Prostate cancer is more common in men over the age of 50. The men involved in this study were younger on average, and therefore already have a very low risk of prostate cancer.
“This study would need to look at a much broader age range to fully understand whether men who undergo fertility treatment actually have a higher risk overall.
“If this can be proven, more research would then need to be done to determine the underlying cause. Until then, there is little evidence that there would be any benefit in monitoring these men more closely.”
He added: “We believe it’s important that all men are aware of the risks of prostate cancer, and men concerned about the disease should speak to their GP. However, couples considering fertility treatment should not be put off by these results.”
More: Chin surgery