1818 - <a href =" https://github.com/janhq/cortex.cpp/releases " >Changelog</a > - <a href =" https://github.com/janhq/cortex.cpp/issues " >Bug reports</a > - <a href =" https://discord.gg/AsJ8krTT3N " >Discord</a >
1919</p >
2020
21- > ⚠️ ** Cortex.cpp is currently in Development . This documentation outlines the intended behavior of Cortex, which may not yet be fully implemented in the codebase.**
21+ > ⚠️ ** Cortex.cpp is currently in active development . This outlines the intended behavior of Cortex, which may not yet be fully implemented in the codebase.**
2222
2323## Overview
2424Cortex.cpp is a Local AI engine that is used to run and customize LLMs. Cortex can be deployed as a standalone server, or integrated into apps like [ Jan.ai] ( https://jan.ai/ ) .
@@ -28,142 +28,15 @@ Cortex.cpp is a multi-engine that uses `llama.cpp` as the default engine but als
2828- [ ` onnx ` ] ( https://github.com/janhq/cortex.onnx )
2929- [ ` tensorrt-llm ` ] ( https://github.com/janhq/cortex.tensorrt-llm )
3030
31- To install Cortex.cpp, download the installer for your operating system from the following options:
32-
33- <table >
34- <tr style =" text-align :center " >
35- <td style="text-align:center"><b>Version Type</b></td>
36- <td style="text-align:center"><b>Windows</b></td>
37- <td colspan="2" style="text-align:center"><b>MacOS</b></td>
38- <td colspan="2" style="text-align:center"><b>Linux</b></td>
39- </tr >
40- <tr style =" text-align :center " >
41- <td style="text-align:center"><b>Stable (Recommended)</b></td>
42- <td style="text-align:center">
43- <a href='https://github.com/janhq/cortex.cpp/releases'>
44- <img src='https://github.com/janhq/docs/blob/main/static/img/windows.png' style="height:14px; width: 14px" />
45- <b>Download</b>
46- </a>
47- </td>
48- <td style="text-align:center">
49- <a href='https://github.com/janhq/cortex.cpp/releases'>
50- <img src='https://github.com/janhq/docs/blob/main/static/img/mac.png' style="height:15px; width: 15px" />
51- <b>Intel</b>
52- </a>
53- </td>
54- <td style="text-align:center">
55- <a href='https://github.com/janhq/cortex.cpp/releases'>
56- <img src='https://github.com/janhq/docs/blob/main/static/img/mac.png' style="height:15px; width: 15px" />
57- <b>M1/M2/M3/M4</b>
58- </a>
59- </td>
60- <td style="text-align:center">
61- <a href='https://github.com/janhq/cortex.cpp/releases'>
62- <img src='https://github.com/janhq/docs/blob/main/static/img/linux.png' style="height:14px; width: 14px" />
63- <b>Debian Download</b>
64- </a>
65- </td>
66- <td style="text-align:center">
67- <a href='https://github.com/janhq/cortex.cpp/releases'>
68- <img src='https://github.com/janhq/docs/blob/main/static/img/linux.png' style="height:14px; width: 14px" />
69- <b>Fedora Download</b>
70- </a>
71- </td>
72- </tr >
73- </table >
31+ ## Installation
7432
75- > ** Note** :
76- > You can also build Cortex.cpp from source by following the steps [ here] ( #build-from-source ) .
77-
78-
79- ### Libraries
80- - [ cortex.js] ( https://github.com/janhq/cortex.js )
81- - [ cortex.py] ( https://github.com/janhq/cortex-python )
82-
83- ## Quickstart
84- ### CLI
85- ``` bash
86- # 1. Start the Cortex.cpp server (The server is running at localhost:3928)
87- cortex start
88-
89- # 2. Start and chat with the model
90- cortex run < model_id> :[engine_name]
91-
92- # 3. Stop a model
93- cortex stop < model_id> :[engine_name]
94-
95- # 4. Stop the Cortex.cpp server
96- cortex stop
97- ```
98- ### API
99- 1 . Start the API server using ` cortex ` command.
100- 2 . ** Pull a Model**
101- ``` bash
102- curl --request POST \
103- --url http://localhost:3928/v1/models/{model_id}/pull
104- ```
105-
106- 3 . ** Start a Model**
107- ``` bash
108- curl --request POST \
109- --url http://localhost:3928/v1/models/{model_id}/start \
110- --header ' Content-Type: application/json' \
111- --data ' {
112- "prompt_template": "system\n{system_message}\nuser\n{prompt}\nassistant",
113- "stop": [],
114- "ngl": 4096,
115- "ctx_len": 4096,
116- "cpu_threads": 10,
117- "n_batch": 2048,
118- "caching_enabled": true,
119- "grp_attn_n": 1,
120- "grp_attn_w": 512,
121- "mlock": false,
122- "flash_attn": true,
123- "cache_type": "f16",
124- "use_mmap": true,
125- "engine": "llamacpp"
126- }'
127- ```
128-
129- 4 . ** Chat with a Model**
130- ``` bash
131- curl http://localhost:3928/v1/chat/completions \
132- -H " Content-Type: application/json" \
133- -d ' {
134- "model": "",
135- "messages": [
136- {
137- "role": "user",
138- "content": "Hello"
139- },
140- ],
141- "model": "mistral",
142- "stream": true,
143- "max_tokens": 1,
144- "stop": [
145- null
146- ],
147- "frequency_penalty": 1,
148- "presence_penalty": 1,
149- "temperature": 1,
150- "top_p": 1
151- }'
152- ```
153-
154- 5 . ** Stop a Model**
155- ``` bash
156- curl --request POST \
157- --url http://localhost:3928/v1/models/mistral/stop
158- ```
159- 6 . Stop the Cortex.cpp server using ` cortex stop ` command.
160- > ** Note** :
161- > Our API server is fully compatible with the OpenAI API, making it easy to integrate with any systems or tools that support OpenAI-compatible APIs.
33+ You can install a nightly (unstable) version of Cortex from Discord here: https://discord.gg/nGp6PMrUqS
16234
16335## Built-in Model Library
164- Cortex.cpp supports various models available on the [ Cortex Hub] ( https://huggingface.co/cortexso ) . Once downloaded, all model source files will be stored at ` C:\Users\<username>\AppData\Local\cortexcpp\models ` .
16536
166- Here are example of models that you can use based on each supported engine:
37+ Cortex.cpp supports various models available on the [ Cortex Hub] ( https://huggingface.co/cortexso ) . Once downloaded, all model source files will be stored in ` ~\cortexcpp\models ` .
38+
39+ Example models:
16740
16841| Model | llama.cpp<br >` :gguf ` | TensorRT<br >` :tensorrt ` | ONNXRuntime<br >` :onnx ` | Command |
16942| ------------------| -----------------------| ------------------------------------------| ----------------------------| -------------------------------|
@@ -190,6 +63,7 @@ For complete details on CLI commands, please refer to our [CLI documentation](ht
19063Cortex.cpp includes a REST API accessible at ` localhost:3928 ` . For a complete list of endpoints and their usage, visit our [ API documentation] ( https://cortex.so/api-reference ) .
19164
19265## Uninstallation
66+
19367### Windows
194681 . Navigate to ` Add or Remove Programs ` .
195692 . Search for Cortex.cpp and click ` Uninstall ` .
@@ -205,83 +79,6 @@ sudo sh cortex-uninstall.sh
20579sudo apt remove cortexcpp
20680```
20781
208- ## Alternate Installation
209- We also provide Beta and Nightly version.
210- <table >
211- <tr style =" text-align :center " >
212- <td style="text-align:center"><b>Version Type</b></td>
213- <td style="text-align:center"><b>Windows</b></td>
214- <td colspan="2" style="text-align:center"><b>MacOS</b></td>
215- <td colspan="2" style="text-align:center"><b>Linux</b></td>
216- </tr >
217- <tr style =" text-align :center " >
218- <td style="text-align:center"><b>Beta Build</b></td>
219- <td style="text-align:center">
220- <a href='https://github.com/janhq/cortex.cpp/releases'>
221- <img src='https://github.com/janhq/docs/blob/main/static/img/windows.png' style="height:14px; width: 14px" />
222- <b>cortexcpp.exe</b>
223- </a>
224- </td>
225- <td style="text-align:center">
226- <a href='https://github.com/janhq/cortex.cpp/releases'>
227- <img src='https://github.com/janhq/docs/blob/main/static/img/mac.png' style="height:15px; width: 15px" />
228- <b>Intel</b>
229- </a>
230- </td>
231- <td style="text-align:center">
232- <a href='https://github.com/janhq/cortex.cpp/releases'>
233- <img src='https://github.com/janhq/docs/blob/main/static/img/mac.png' style="height:15px; width: 15px" />
234- <b>M1/M2/M3/M4</b>
235- </a>
236- </td>
237- <td style="text-align:center">
238- <a href='https://github.com/janhq/cortex.cpp/releases'>
239- <img src='https://github.com/janhq/docs/blob/main/static/img/linux.png' style="height:14px; width: 14px" />
240- <b>cortexcpp.deb</b>
241- </a>
242- </td>
243- <td style="text-align:center">
244- <a href='https://github.com/janhq/cortex.cpp/releases'>
245- <img src='https://github.com/janhq/docs/blob/main/static/img/linux.png' style="height:14px; width: 14px" />
246- <b>cortexcpp.AppImage</b>
247- </a>
248- </td>
249- </tr >
250- <tr style =" text-align :center " >
251- <td style="text-align:center"><b>Nightly Build</b></td>
252- <td style="text-align:center">
253- <a href='https://github.com/janhq/cortex.cpp/releases'>
254- <img src='https://github.com/janhq/docs/blob/main/static/img/windows.png' style="height:14px; width: 14px" />
255- <b>cortexcpp.exe</b>
256- </a>
257- </td>
258- <td style="text-align:center">
259- <a href='https://github.com/janhq/cortex.cpp/releases'>
260- <img src='https://github.com/janhq/docs/blob/main/static/img/mac.png' style="height:15px; width: 15px" />
261- <b>Intel</b>
262- </a>
263- </td>
264- <td style="text-align:center">
265- <a href='https://github.com/janhq/cortex.cpp/releases'>
266- <img src='https://github.com/janhq/docs/blob/main/static/img/mac.png' style="height:15px; width: 15px" />
267- <b>M1/M2/M3/M4</b>
268- </a>
269- </td>
270- <td style="text-align:center">
271- <a href='https://github.com/janhq/cortex.cpp/releases'>
272- <img src='https://github.com/janhq/docs/blob/main/static/img/linux.png' style="height:14px; width: 14px" />
273- <b>cortexcpp.deb</b>
274- </a>
275- </td>
276- <td style="text-align:center">
277- <a href='https://github.com/janhq/cortex.cpp/releases'>
278- <img src='https://github.com/janhq/docs/blob/main/static/img/linux.png' style="height:14px; width: 14px" />
279- <b>cortexcpp.AppImage</b>
280- </a>
281- </td>
282- </tr >
283- </table >
284-
28582### Build from Source
28683
28784#### Windows
@@ -355,8 +152,8 @@ make -j4
3551526 . Verify that Cortex.cpp is installed correctly by getting help information.
356153
357154``` sh
358- # Get the help information
359- cortex -h
155+ # Get help
156+ cortex
360157```
361158
362159## Contact Support
0 commit comments