You probably won’t be able to terminate his rights, but you can get a court order for a passport. What does your current custody order say about travel?
Instead of trying to terminate his rights, go to court to modify the visitation to supervised visits. That way, his dad or someone else has to be present. That lets her stay connected with her paternal family safely.
At the same time, ask the judge for a passport order. That way, you can travel freely. The court is much more likely to approve those changes than full termination.
You can’t just terminate his rights. You’ll have to prove he’s unfit, and that’s a high bar. Instead, try for a custody modification.
Get a lawyer. If you have proof of him doing drugs around her, plus his lack of involvement, you can push for full custody and get the passport issue sorted.