You should avoid that unless your model is not using multiple CPUs already.

Most ML frameworks will use all your available CPUs automatically. So if you add more workers there, you will have multiple instances of your model competing for CPU time.

But it's best to always benchmark in order to find your sweet spot.