Hmm. Well even then I think a million million miles will suffice, that's over ten thousand AU, .17 light years.

Maybe? But if (1) it's a few solar masses, and (2) the https://en.wikipedia.org/wiki/Oort_cloud exists - "from 2,000 to 200,000 AU" - then the black hole could send a lot of planetesimals flying off in random directions. Quite of few of those could be moving toward Earth's neighborhood, some at very high speed. Humans might soon join the dinosaurs.