Another even more serious fume problem with scrap brass is the possible inclusion of lead in the alloy. Not all, but some brasses contain both. It's not just a problem of possible scrap solder, but actually formulated into the alloy.
I tend to avoid brass for the most part, but very occasionally cast with it for a decorative part. I am very careful not to overheat, use a covered crucible, and am careful about alloy. My brass melts are small, generally for small parts.
On the other hand I have made many zinc and zinc alloy castings. They are no problem for me and do not fume. The important part is to heat to the melting point but not to the boiling point, and those are far enough apart in pure zinc or zinc alloy not to have issues. At least for me.
