Together AI

LiteLLM supports all models on Together AI.

API Keys

import os 
os.environ["TOGETHERAI_API_KEY"] = "your-api-key"

Sample Usage

from litellm import completion 

os.environ["TOGETHERAI_API_KEY"] = "your-api-key"

messages = [{"role": "user", "content": "Write me a poem about the blue sky"}]

completion(model="together_ai/togethercomputer/Llama-2-7B-32K-Instruct", messages=messages)

Together AI Models

liteLLM supports non-streaming and streaming requests to all models on https://api.together.xyz/

Example TogetherAI Usage - Note: liteLLM supports all models deployed on TogetherAI

Llama LLMs - Chat

Model Name	Function Call	Required OS Variables
togethercomputer/llama-2-70b-chat	`completion('together_ai/togethercomputer/llama-2-70b-chat', messages)`	`os.environ['TOGETHERAI_API_KEY']`

Llama LLMs - Language / Instruct

Model Name	Function Call	Required OS Variables
togethercomputer/llama-2-70b	`completion('together_ai/togethercomputer/llama-2-70b', messages)`	`os.environ['TOGETHERAI_API_KEY']`
togethercomputer/LLaMA-2-7B-32K	`completion('together_ai/togethercomputer/LLaMA-2-7B-32K', messages)`	`os.environ['TOGETHERAI_API_KEY']`
togethercomputer/Llama-2-7B-32K-Instruct	`completion('together_ai/togethercomputer/Llama-2-7B-32K-Instruct', messages)`	`os.environ['TOGETHERAI_API_KEY']`
togethercomputer/llama-2-7b	`completion('together_ai/togethercomputer/llama-2-7b', messages)`	`os.environ['TOGETHERAI_API_KEY']`

Falcon LLMs

Model Name	Function Call	Required OS Variables
togethercomputer/falcon-40b-instruct	`completion('together_ai/togethercomputer/falcon-40b-instruct', messages)`	`os.environ['TOGETHERAI_API_KEY']`
togethercomputer/falcon-7b-instruct	`completion('together_ai/togethercomputer/falcon-7b-instruct', messages)`	`os.environ['TOGETHERAI_API_KEY']`

Alpaca LLMs

Model Name	Function Call	Required OS Variables
togethercomputer/alpaca-7b	`completion('together_ai/togethercomputer/alpaca-7b', messages)`	`os.environ['TOGETHERAI_API_KEY']`

Other Chat LLMs

Model Name	Function Call	Required OS Variables
HuggingFaceH4/starchat-alpha	`completion('together_ai/HuggingFaceH4/starchat-alpha', messages)`	`os.environ['TOGETHERAI_API_KEY']`

Code LLMs

Model Name	Function Call	Required OS Variables
togethercomputer/CodeLlama-34b	`completion('together_ai/togethercomputer/CodeLlama-34b', messages)`	`os.environ['TOGETHERAI_API_KEY']`
togethercomputer/CodeLlama-34b-Instruct	`completion('together_ai/togethercomputer/CodeLlama-34b-Instruct', messages)`	`os.environ['TOGETHERAI_API_KEY']`
togethercomputer/CodeLlama-34b-Python	`completion('together_ai/togethercomputer/CodeLlama-34b-Python', messages)`	`os.environ['TOGETHERAI_API_KEY']`
defog/sqlcoder	`completion('together_ai/defog/sqlcoder', messages)`	`os.environ['TOGETHERAI_API_KEY']`
NumbersStation/nsql-llama-2-7B	`completion('together_ai/NumbersStation/nsql-llama-2-7B', messages)`	`os.environ['TOGETHERAI_API_KEY']`
WizardLM/WizardCoder-15B-V1.0	`completion('together_ai/WizardLM/WizardCoder-15B-V1.0', messages)`	`os.environ['TOGETHERAI_API_KEY']`
WizardLM/WizardCoder-Python-34B-V1.0	`completion('together_ai/WizardLM/WizardCoder-Python-34B-V1.0', messages)`	`os.environ['TOGETHERAI_API_KEY']`

Language LLMs

Model Name	Function Call	Required OS Variables
NousResearch/Nous-Hermes-Llama2-13b	`completion('together_ai/NousResearch/Nous-Hermes-Llama2-13b', messages)`	`os.environ['TOGETHERAI_API_KEY']`
Austism/chronos-hermes-13b	`completion('together_ai/Austism/chronos-hermes-13b', messages)`	`os.environ['TOGETHERAI_API_KEY']`
upstage/SOLAR-0-70b-16bit	`completion('together_ai/upstage/SOLAR-0-70b-16bit', messages)`	`os.environ['TOGETHERAI_API_KEY']`
WizardLM/WizardLM-70B-V1.0	`completion('together_ai/WizardLM/WizardLM-70B-V1.0', messages)`	`os.environ['TOGETHERAI_API_KEY']`

Prompt Templates

Using a chat model on Together AI with it's own prompt format?

Using Llama2 Instruct models

If you're using Together AI's Llama2 variants( model=togethercomputer/llama-2..-instruct), LiteLLM can automatically translate between the OpenAI prompt format and the TogetherAI Llama2 one ([INST]..[/INST]).

from litellm import completion 

# set env variable 
os.environ["TOGETHERAI_API_KEY"] = ""

messages = [{"role": "user", "content": "Write me a poem about the blue sky"}]

completion(model="together_ai/togethercomputer/Llama-2-7B-32K-Instruct", messages=messages)

Using another model

You can create a custom prompt template on LiteLLM (and we welcome PRs to add them to the main repo 🤗)

Let's make one for OpenAssistant/llama2-70b-oasst-sft-v10!

The accepted template format is: Reference

"""
<|im_start|>system
{system_message}<|im_end|>
<|im_start|>user
{prompt}<|im_end|>
<|im_start|>assistant
"""

Let's register our custom prompt template: Implementation Code

import litellm 

litellm.register_prompt_template(
        model="OpenAssistant/llama2-70b-oasst-sft-v10",
        roles={
            "system": {
                "pre_message": "[<|im_start|>system",
                "post_message": "\n"
            },
            "user": {
                "pre_message": "<|im_start|>user",
                "post_message": "\n"
            }, 
            "assistant": {
                "pre_message": "<|im_start|>assistant",
                "post_message": "\n"
            }
        }
    )

Let's use it!

from litellm import completion 

# set env variable 
os.environ["TOGETHERAI_API_KEY"] = ""

messages=[{"role":"user", "content": "Write me a poem about the blue sky"}]

completion(model="together_ai/OpenAssistant/llama2-70b-oasst-sft-v10", messages=messages)

Complete Code

import litellm 
from litellm import completion

# set env variable 
os.environ["TOGETHERAI_API_KEY"] = ""

litellm.register_prompt_template(
        model="OpenAssistant/llama2-70b-oasst-sft-v10",
        roles={
            "system": {
                "pre_message": "[<|im_start|>system",
                "post_message": "\n"
            },
            "user": {
                "pre_message": "<|im_start|>user",
                "post_message": "\n"
            }, 
            "assistant": {
                "pre_message": "<|im_start|>assistant",
                "post_message": "\n"
            }
        }
    )

messages=[{"role":"user", "content": "Write me a poem about the blue sky"}]

response = completion(model="together_ai/OpenAssistant/llama2-70b-oasst-sft-v10", messages=messages)

print(response)

Output

{
  "choices": [
    {
      "finish_reason": "stop",
      "index": 0,
      "message": {
        "content": ".\n\nThe sky is a canvas of blue,\nWith clouds that drift and move,",
        "role": "assistant",
        "logprobs": null
      }
    }
  ],
  "created": 1693941410.482018,
  "model": "OpenAssistant/llama2-70b-oasst-sft-v10",
  "usage": {
    "prompt_tokens": 7,
    "completion_tokens": 16,
    "total_tokens": 23
  },
  "litellm_call_id": "f21315db-afd6-4c1e-b43a-0b5682de4b06"
}

Advanced Usage

Instead of using the custom_llm_provider arg to specify which provider you're using (e.g. together ai), you can just pass the provider name as part of the model name, and LiteLLM will parse it out.

Expected format: <custom_llm_provider>/<model_name>

e.g. completion(model="together_ai/togethercomputer/Llama-2-7B-32K-Instruct", ...)

from litellm import completion 

# set env variable 
os.environ["TOGETHERAI_API_KEY"] = ""

messages = [{"role": "user", "content": "Write me a poem about the blue sky"}]

completion(model="together_ai/togethercomputer/Llama-2-7B-32K-Instruct", messages=messages)

Together AI

API Keys​

Sample Usage​

Together AI Models​

Llama LLMs - Chat​

Llama LLMs - Language / Instruct​

Falcon LLMs​

Alpaca LLMs​

Other Chat LLMs​

Code LLMs​

Language LLMs​

Prompt Templates​

Using Llama2 Instruct models​

Using another model​

Advanced Usage​