Skip to main content

Llama 2 Resources

Free playgrounds #

Running it yourself on a cloud GPU #

  • 70B GPTQ version required 35-40 GB VRAM. Follow this guide

Hosted APIs #

Best model versions to use #

See more on GPTQ vs GGML versions of models here.

What about 34B? #

It’s coming soon. They wanted more time to red-team it.

Chat vs Base #

The base models are uncensored, and are not instruct-tuned or chat-tuned.

The chat models are censored, and have been chat-tuned.

Are the base models instruct-tuned? #

No.1 “These models are not finetuned for chat or Q&A. They should be prompted so that the expected answer is the natural continuation of the prompt.”

How can I prompt the base models? #

Create a fake document so that if the model naturally continues what would be expected next in the document, you get the result you want. For example, instead of “What is the capital of France?” you’d write “The capital of France is "

See also:

Are they censored? #

Base is uncensored, chat is censored.

Prompt template #

<s>[INST] <<SYS>>

{user_message_1} [/INST]


<s>[INST] <<SYS>>

{user_message_1} [/INST] {model_reply_1}</s><s>[INST] {user_message_2} [/INST]

See here.

Setting up an API endpoint #

  • Hugging Face
  • Docker/Runpod - see here but use this runpod template instead of the one linked in that post
  1. Devs playing around with it
  2. Uses that GPT doesn’t allow but are legal (for example, NSFW content)
  3. Enterprises using it as an alternative to GPT-3.5 if they can get it to be cheaper overall
  4. Enterprises using it as an alternative to GPT-4 if they can fine-tune it for a specific use case and get comparable performance

Fine-tuning #

Fine-tuning as a service for enterprises #

  • Databricks (Patrick, co-founder, and Xiangrui, ML Lead) incl Mosaic ML
  • (Lamini is a Llama inspired name. Llamas, Alpacas, Vicuñas and Guanacos are all subsets of the Lamini tribe.)
  • Radiant AI
  • Scale AI
  • Snowflake

Running locally #