Mercury remains a concern with seafood. Some larger fish, like shark, swordfish and marlin, contain this metal that can cause brain and nerve damage. Pregnant women and small children should avoid these fish, and others should limit their consumption to once per week. Fortunately, the fish highest in healthful omega-3 fats are the lowest in mercury.