🌟 𝐋𝐋𝐌 𝐏𝐫𝐨𝐯𝐢𝐝𝐞𝐫'𝐬 𝐑𝐞𝐥𝐞𝐚𝐬𝐞 (2024 1H) & 📝 2024 𝐋𝐋𝐌 𝐒𝐮𝐫𝐯𝐞𝐲 (on Training / Data / RAG / Serving / Agent)

🌟 𝐋𝐋𝐌 𝐏𝐫𝐨𝐯𝐢𝐝𝐞𝐫’𝐬 𝐑𝐞𝐥𝐞𝐚𝐬𝐞 (2024 1H)

I have selected 21 LLMs from various providers for 2024 1H. [📣Release news] and [📋Tech report] are in this post. [📘API docs] and [🤗HF models] are in the comments.

1. 𝐐𝐰𝐞𝐧-2 (​Alibaba Group​, 2024.06.07)
   • • 📣News: https://qwenlm.github.io/blog/qwen2/
2. 𝐒𝐨𝐥𝐚𝐫-𝐌𝐢𝐧𝐢-𝐣𝐚 (​Upstage​, 2024.05.22)
   • • 📣News: https://www.upstage.ai/feed/tech/solar-mini-chat-ja
3. 𝐘𝐢-𝐋𝐚𝐫𝐠𝐞 (​01.AI​, 2024.05.13)
   • • 📣News: https://x.com/01AI_Yi/status/1789929378467426794
4. 𝐘𝐢-1.5 (​01.AI​, 2024.05.13)
   • • 📣News: https://x.com/01AI_Yi/status/1789869537317540016
   • • 📋arXiv: https://arxiv.org/abs/2403.04652
5. 𝐆𝐏𝐓-4𝐨 (​OpenAI​, 2024.05.13)
   • • 📣News: https://openai.com/index/hello-gpt-4o/
6. 𝐐𝐰𝐞𝐧-𝐌𝐚𝐱 (​Alibaba Group​, 2024.05.11)
   • • 📣News: https://qwenlm.github.io/blog/qwen-max-0428/
7. 𝐃𝐞𝐞𝐩𝐒𝐞𝐞𝐤-𝐕2 (DeepSeek, 2024.05.07)
   • • 📣News: https://x.com/deepseek_ai/status/1787478986731429933
   • • 📋arXiv: https://arxiv.org/abs/2405.04434
8. 𝐒𝐧𝐨𝐰𝐟𝐥𝐚𝐤𝐞-𝐀𝐫𝐜𝐭𝐢𝐜 (​Snowflake​, 2024.04.24)
   • • 📣News: https://www.snowflake.com/blog/arctic-open-efficient-foundation-language-models-snowflake/
9. 𝐏𝐡𝐢-3 (​Microsoft​, 2024.04.22)
   • • 📣News: https://azure.microsoft.com/en-us/blog/introducing-phi-3-redefining-whats-possible-with-slms/
   • • 📋arXiv: https://arxiv.org/abs/2404.14219
10. 𝐋𝐥𝐚𝐦𝐚-3 (​Meta Facebook​, 2024.04.18)
    • • 📣News: https://ai.meta.com/blog/meta-llama-3/
11. 𝐌𝐢𝐱𝐭𝐫𝐚𝐥-8𝐱22𝐁 (​Mistral AI​, 2024.04.17)
    • • 📣News: https://mistral.ai/news/mixtral-8x22b/
12. 𝐑𝐞𝐤𝐚-𝐂𝐨𝐫𝐞 (​Reka AI​​, 2024.04.15)
    • • 📣News: https://www.reka.ai/news/reka-core-our-frontier-class-multimodal-language-model
    • • 📋arXiv: https://arxiv.org/abs/2404.12387
13. 𝐂𝐨𝐦𝐦𝐚𝐧𝐝-𝐑-𝐏𝐥𝐮𝐬 (​Cohere​, 2024.04.04)
    • • 📣News: https://cohere.com/blog/command-r-plus-microsoft-azure
14. 𝐃𝐁𝐑𝐗 (​Databricks​, 2024.03.27)
    • • 📣News: https://www.databricks.com/blog/introducing-dbrx-new-state-art-open-llm
15. 𝐆𝐞𝐦𝐢𝐧𝐢-1.5 (​Google​, 2024.03.08)
    • • 📣News: https://blog.google/technology/ai/google-gemini-next-generation-model-february-2024/
    • • 📋arXiv: https://arxiv.org/abs/2403.05530
16. 𝐂𝐥𝐚𝐮𝐝𝐞-3 (​Anthropic​, 2024.03.04)
    • • 📣News: https://www.anthropic.com/news/claude-3-family
17. 𝐌𝐢𝐬𝐭𝐫𝐚𝐥-𝐋𝐚𝐫𝐠𝐞 (​Mistral AI​, 2024.02.26)
    • • 📣News: https://mistral.ai/news/mistral-large/
18. 𝐆𝐞𝐦𝐦𝐚 (​Google​, 2024.02.21)
    • • 📣News: https://blog.google/technology/developers/gemma-open-models/
    • • 📋arXiv: https://arxiv.org/abs/2403.08295
19. 𝐐𝐰𝐞𝐧-1.5 (​Alibaba Group​, 2024.02.04)
    • • 📣News: https://qwenlm.github.io/blog/qwen1.5/
20. 𝐒𝐨𝐥𝐚𝐫-𝐌𝐢𝐧𝐢 (​Upstage​, 2024.01.25)
    • • 📣News: https://www.upstage.ai/feed/product/solarmini-performance-report
21. 𝐒𝐨𝐥𝐚𝐫-10.7𝐁 (​Upstage​, 2023.12.23)
    • • 📣News: https://www.upstage.ai/feed/press/solar-10-7b-emerges-as-worlds-top-pre-trained-llm
    • • 📋arXiv: https://arxiv.org/abs/2312.15166

---

📝 2024 𝐋𝐋𝐌 𝐒𝐮𝐫𝐯𝐞𝐲 (on Training / Data / RAG / Serving / Agent)

I’ll read all below !!

𝐓𝐫𝐚𝐢𝐧𝐢𝐧𝐠

📌 A Survey on Self-Evolution of Large Language Models (2024.04.22)

• • arXiv: https://arxiv.org/abs/2404.14387
• • Github: https://github.com/AlibabaResearch/DAMO-ConvAI/tree/main/Awesome-Self-Evolution-of-LLM

📌 Continual Learning of Large Language Models: A Comprehensive Survey (2024.04.25)

• • arXiv: https://arxiv.org/abs/2404.16789
• • Github: https://github.com/Wang-ML-Lab/llm-continual-learning-survey

📌 Continual Learning with Pre-Trained Models: A Survey (2024.01.29)

• • arXiv: https://arxiv.org/abs/2401.16386
• • Github: https://github.com/sun-hailong/LAMDA-PILOT

D𝐚𝐭𝐚

📌 Datasets for Large Language Models: A Comprehensive Survey (2024.02.28)

• • arXiv: https://arxiv.org/pdf/2402.18041
• • Github: https://github.com/lmmlzn/Awesome-LLMs-Datasets

📌 A Survey on Data Selection for Language Models (2024.02.26)

• • arXiv: https://arxiv.org/abs/2402.16827
• • Github: https://github.com/alon-albalak/data-selection-survey

📌 A Survey on Data Selection for LLM Instruction Tuning (2024.02.04)

• • arXiv: https://arxiv.org/abs/2402.05123
• • Github: https://github.com/Bolin97/awesome-instruction-selector

𝐑𝐀𝐆

📌 RAG and RAU: A Survey on Retrieval-Augmented Language Model in Natural Language Processing (2024.04.30)

• • arXiv: https://arxiv.org/abs/2404.19543
• • Github: https://github.com/2471023025/RALM_Survey

📌 Retrieval-Augmented Generation for AI-Generated Content: A Survey (2024.02.29)

• • arXiv: https://arxiv.org/abs/2402.19473
• • Github: https://github.com/hymie122/RAG-Survey

📌 Retrieval-Augmented Generation for Large Language Models: A Survey (2023.12.18)

• • arXiv: https://arxiv.org/abs/2312.10997
• • Github: https://github.com/Tongji-KGLLM/RAG-Survey

𝐒𝐞𝐫𝐯𝐢𝐧𝐠

📌 LLM Inference Unveiled: Survey and Roofline Model Insights (2024.02.26)

• • arXiv: https://arxiv.org/abs/2402.16363
• • Github: https://github.com/hahnyuan/LLM-Viewer

📌 A Survey on Effective Invocation Methods of Massive LLM Services (2024.02.05)

• • arXiv: https://arxiv.org/abs/2402.03408
• • Github: https://github.com/W-caner/Effective-strategy-for-LMaas

📌 Beyond Efficiency: A Systematic Survey of Resource-Efficient Large Language Models (2024.01.01)

• • arXiv: https://arxiv.org/abs/2401.00625
• • Github: https://github.com/tiingweii-shii/Awesome-Resource-Efficient-LLM-Papers

𝐀𝐠𝐞𝐧𝐭

📌 Large Multimodal Agents: A Survey (2024.02.23)

• • arXiv: https://arxiv.org/abs/2402.15116
• • Github: https://github.com/jun0wanan/awesome-large-multimodal-agents

📌 Large Language Model based Multi-Agents: A Survey of Progress and Challenges (2024.01.21)

• • arXiv: https://arxiv.org/abs/2402.01680
• • Github: https://github.com/taichengguo/LLM_MultiAgents_Survey_Papers

📌 Personal LLM Agents: Insights and Survey about the Capability, Efficiency and Security (2024.01.10)

• • arXiv: https://arxiv.org/abs/2401.05459
• • Github: https://github.com/MobileLLM/Personal_LLM_Agents_Survey

Information about Tokens in LLsM

Why do we keep talking about “tokens” in LLMs instead of words?

It happens to be much more efficient to break the words into sub-words (tokens) for model performance!

The typical strategy used in most modern LLMs since GPT-1 is the Byte Pair Encoding (BPE) strategy. The idea is to use, as tokens, sub-word units that appear often in the training data. The algorithm works as follows:

• We start with a character-level tokenization
• we count the pair frequencies
• We merge the most frequent pair
• We repeat the process until the dictionary is as big as we want it to be

The size of the dictionary becomes a hyperparameter that we can adjust based on our training data. For example, GPT-1 has a dictionary size of ~40K merges, GPT-2, GPT-3, and ChatGPT have a dictionary size of ~50K, and Llama 3 128K.