A lot of things that show up in studies as 5AR inhibitors (rice, resveratrol, zinc) I consume fairly frequently with no problems. I take Wicked Supplements' TocoCaps every day and it contains beta sitosterol (a component in saw Palmetto, which is what fucked me) and it does nothing but benefit me (FWIW, it's only 8mg which is small in comparison to a Saw Palmetto extract). But I was kinda this way originally because of my experience but I took the plunge on TocoCaps, Res, rice, zinc, coconut oil (all shown in studies as 5AR/DHT inhibitors), yet I make monthly (sometimes more than that) progress to recovery, and I'm in a pretty good place right now to where I rarely think about PFS.
The main ones I would avoid are: Finasteride/dutasteride (obviously), saw palmetto berry or extract/beta-sitosterol, pygeum, or high dose zinc. Those are all proven to cause people PFS-like problems.