𝐋𝐋𝐌2𝐕𝐞𝐜 - 𝐓𝐫𝐚𝐧𝐬𝐟𝐨𝐫𝐦 𝐋𝐋𝐌𝐬 𝐢𝐧𝐭𝐨 𝐄𝐦𝐛𝐞𝐝𝐝𝐢𝐧𝐠 𝐌𝐨𝐝𝐞𝐥𝐬

LLM2Vec paper 👉 https://mcgill-nlp.github.io/llm2vec/

Intruction

LLM2Vec, a simple unsupervised approach that can transform any decoder-only LLM into a strong text encoder.

LLM2Vec consists of three simple steps:

1. enabling bidirectional attention,
2. masked next token prediction, and
3. unsupervised contrastive learning.

LLM2Vec not only outperforms encoder-only models on word-level tasks but also achieves new SOTA results on the MTEB benchmark.

To summarize, LLM2Vec shows that without expensive adaptation or synthetic GPT-4 data, LLMs can be transformed into embedding models (universal text encoders)