SGLang

SGLang is an open-source LLM programming and inference framework built by the team behind SkyPilot. It is a wrapper around vLLM that adds structured output generation capabilities, tool calling, and better prompt control. It is ideal for building AI agents and implementing tool calling architectures.

Getting started

code.txt
1pip install --upgrade pip
2pip install uv
3uv pip install "sglang[all]>=0.4.10.post2"

You can launch an OpenAI-compatible API server with:

code.txt
1python3 -m sglang.launch_server --model-path qwen/qwen2.5-0.5b-instruct --host 0.0.0.0

Usage

code.py
1from sglang import SGLang, system, user, assistant
2
3sg = SGLang(model="mistralai/Mistral-7B-Instruct-v0.1")
4
5@sg.function
6def chat_func(sg):
7    sg += system("You are a helpful assistant.")
8    sg += user("What's the difference between stars and planets?")
9    sg += assistant()
10
11# Call the function
12print(sg.chat_func().run().text)

This defines a chat function, compiles it to a prompt, sends it to the LLM, and parses the assistant's response.

Tool calling

code.py
1from sglang import SGLang, tool, user, assistant
2
3sg = SGLang(model="mistralai/Mixtral-8x7B-Instruct-v0.1")
4
5# Define a tool that can be called by the LLM
6@tool
7def get_weather(city: str) -> str:
8    return f"The weather in {city} is sunny."
9
10@sg.function
11def weather_agent(sg):
12    sg.user("What is the weather like in Tokyo?")
13    sg.assistant(tools=[get_weather])
14
15result = sg.weather_agent().run()
16print(result.tool_calls)  # Inspect tool call info
17print(result.text)        # Final answer