Aligned representations throughout languages is a desired property in multilingual massive language fashions (mLLMs), as alignment can enhance efficiency in cross-lingual duties. Usually alignment requires fine-tuning a mannequin, which is computationally costly, and sizable language information, which frequently will not be accessible. A knowledge-efficient different to fine-tuning is mannequin interventions — a way for manipulating mannequin activations to steer era into the specified path. We analyze the impact of a well-liked intervention (discovering specialists) on the alignment of cross-lingual representations in mLLMs. We establish the neurons to govern for a given language and introspect the embedding area of mLLMs pre- and post-manipulation. We present that modifying the mLLM’s activations modifications its embedding area such that cross-lingual alignment is enhanced. Additional, we present that the modifications to the embedding area translate into improved downstream efficiency on retrieval duties, with as much as 2x enhancements in top-1 accuracy on cross-lingual retrieval.
- † Work executed whereas at Apple
- ‡ Equal contribution
- § AI Digital Assistant Lab, Georgia Institute of Know-how
