Instructions to use LeoLM/leo-hessianai-7b-chat with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use LeoLM/leo-hessianai-7b-chat with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("text-generation", model="LeoLM/leo-hessianai-7b-chat", trust_remote_code=True) messages = [ {"role": "user", "content": "Who are you?"}, ] pipe(messages)# Load model directly from transformers import AutoTokenizer, AutoModelForCausalLM tokenizer = AutoTokenizer.from_pretrained("LeoLM/leo-hessianai-7b-chat", trust_remote_code=True) model = AutoModelForCausalLM.from_pretrained("LeoLM/leo-hessianai-7b-chat", trust_remote_code=True) messages = [ {"role": "user", "content": "Who are you?"}, ] inputs = tokenizer.apply_chat_template( messages, add_generation_prompt=True, tokenize=True, return_dict=True, return_tensors="pt", ).to(model.device) outputs = model.generate(**inputs, max_new_tokens=40) print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:])) - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- vLLM
How to use LeoLM/leo-hessianai-7b-chat with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "LeoLM/leo-hessianai-7b-chat" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "LeoLM/leo-hessianai-7b-chat", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker
docker model run hf.co/LeoLM/leo-hessianai-7b-chat
- SGLang
How to use LeoLM/leo-hessianai-7b-chat with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "LeoLM/leo-hessianai-7b-chat" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "LeoLM/leo-hessianai-7b-chat", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "LeoLM/leo-hessianai-7b-chat" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "LeoLM/leo-hessianai-7b-chat", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }' - Docker Model Runner
How to use LeoLM/leo-hessianai-7b-chat with Docker Model Runner:
docker model run hf.co/LeoLM/leo-hessianai-7b-chat
Commercial use
I posted this on Twitter but reiterating here for transparency.
First of all, thanks for your efforts. It's great to see a focus on non-English languages! Second, you write that your models are the frist German-focused, commercially available models. However, the chat and instruction datasets that you trained on were translated with OpenAI services and they explicitly forbid that models trained with the output of their own models be used for commercial reasons.
So could you clarify what you meant? As far as I can tell your base models can be commercially licensed, but the chat models cannot.
Adding an explicit license would be very helpful!
Thanks for your interest and for bringing this up!
LAION does not believe that any AI model's output can be protected commercially and we therefore see no problem with releasing the models under the Llama-2 community license.
Aside from this, we do not recommend using the chat models in any production setting without further validation or more explicit, task-specific finetuning. While the chat models are a good baseline and serve their purpose of demonstrating the efficacy of our continued pretraining approach, they are not reliable enough as-is for any production environment and tend to hallucinate. Finding and creating better finetuning datasets (perhaps without the use of OpenAI) is up to the community. Our base models are there to accelerate this development and make German finetuning easier for the open-source and commercial community.
While I share the sentiment, you cannot just say that you do not believe something that is legally binding, unless you lawyer up and fight to have it changed. OpenAI's Terms of Use clearly state that their output cannot be used to create commercial LLMs. You use their data, so you must agree with those terms. Again, I agree with your sentiment that it is incredibly frustrating that OpenAI puts up these restrictions, effectively creating an oligarchy in the commercial LLM space. But just ignoring their terms does not make commercial use legal. In fact, this has unfair repercussions: others who explicitly state that their models cannot be used for commercial reasons - following legal requirements - might be passed by in favor of your model because they think yours allows for commercial use (which it does not).
I emphasize again that I admire the model and the efforts and the transparency. From a legal standpoint, however, this does not feel right.
Thank you for voicing your concerns on this. In case you are interested in the legal backgrounds of our decisions, please write a mail to contact@laion.ai