By policy, don't make changes outside of Terraform. This will slow you down a little to start with, but developing the culture of infrastructure via terraform pushes is part of what makes IaC successful. Obviously you should maintain break-glass capability, but doing all changes via Terraform will get you what you want.
Its much better to think of them as two tools that complement each other. In Void Linux we use terraform for setting up resources on clouds and setting up SDN links between them. Provisioning hosts on the other hand is best left to a tool that is designed for that, and Ansible is a good tool for this. There are certainly points where I would rather have a single tool to rule them all, but I suspect in that case it wouldn't do either task particularly well.
If you absolutely had to pick one tool to do both use cases, I'd go with Terraform and have it invoke an ansible-like provisioning process.