Analyzing Language Mismatch In Self-supervised Speech Representation Learning