Well, no, a name like Peter won't affect anyone's job options and that really isn't a reason to not use it as a first name.
One and only reason is that OP doesn't like it.
Totally agree with this. I’ve never looked at the name Peter and had any sort of negative thought about it. We have a few Peters of various ages in my family and DS had a couple at primary school.
But yes, the only reason to not use the name is because OP doesn’t like it and that’s fair enough.