It depends. If the language barrier is a problem then yes, I would re-assign them.
There is nothing worse than wasted potential and if a team member can't get on because they don't properly understand you then it is up to you to give them the best possible chance of success! Remember, you will still get a benefit from this (matching VP etc)
