How to Use an Free Offline AI in GPTits
Added 2024-01-16 16:46:25 +0000 UTCDownload the latest KoboldCPP, either koboldcpp.exe if your GPU supports CUDA, or "_nocuda" if it doesn't. https://github.com/LostRuins/koboldcpp/releases
Uncheck "Launch Browser," as the intention is to use it with GPTits.
ncrease the Context Size if the model and your hardware allow.
Select a valid model.
On this page, you will find a list of models, some of which are censored while others are not: https://github.com/Troyanovsky/Local-LLM-Comparison-Colab-UI/tree/main
The majority of the models are hosted on the Hugging Face website. In the "Files and versions" section, you can download the respective model. Within the "Model card", you will find instructions, and possibly a table indicating which variant of the model is most suitable for you based on considerations such as RAM, disk usage, and method.
The latest models conclude with the .GGUF extension; however, you may come across them labeled as .BIN or other file extensions.
In "AI" -> "MODE" use the "Direct Url".In network koboldcpp settings, the default port is 5001, so you can use the EXTERNAL_URL as "http://localhost:5001/api/v1/generate". In case you cannot use this port, feel free to make the necessary adjustments in both KoboldCPP and GPTits.
Enjoy! The speed and quality depend on your hardware, model, and settings.
