private ai server automation · ubuntu 24.04 · mit

Your models. Your data. Your hardware.

kwisatz turns a fresh Ubuntu box — an NVIDIA DGX Spark, an AMD Strix Halo, any workstation — into a machine that actually serves models. llama.cpp built for your GPU, models hot-swapped at :9292, a ChatGPT-style UI at :3333 — with hardened SSH and UFW on the way.

fresh box→ 2 YAML files→ 1 command→ working inference server

Get started in 4 steps Browse 60+ automations
automations
0
config files
0
commands
0
cloud dependency
0
verfeinerer@lab: ~/kwisatz

§ 01 why private ai

A cloud API is a subscription
to someone else's datacenter.

Cloud AI APIkwisatz
Your prompts, code & documentsa third-party datacenterstay on your machine / LAN
Cost modelper token, foreverhardware you already own
Model choicewhat the vendor offers todayany open-weight GGUF / vLLM model
Availabilityoutages, rate limits, deprecationsruns offline, pinned versions
GDPR / NDA / air-gapcontracts and trustdata never leaves the network

The real cost of a DIY inference server is not the GPU. It's the ten install projects wearing a trench coat: driver, engine, model proxy, web UI, SSH hardening, firewall, monitoring, backups. Done by hand that's a weekend of plumbing before the first token. kwisatz is that weekend, committed to a repo.

§ 02 how it works

Config is the runbook.

  • machine-config.yml

    Infrastructure. Which of the 60+ task scripts run, in which order, with which env vars and args. Enabled services are the whole truth of the box.

  • models.yml

    Model catalog. Which LLMs to download, serve, and wire into your coding agents — a declarative catalog, not a download folder.

  • tasks/*.sh

    Idempotent by contract. Every task is self-contained, safe to re-run alone, and converges instead of tearing down.

  • deploy.py

    One engine. The CLI plans the run, then executes it with a pyinfra deploy — so a single-target apply cannot install something different from a full one.

machine-config.ymlservices · env · args · order models.ymlcatalog · sync · agent wiring kwisatz applyplanner + pyinfra deploy tasks/setup-*.shidempotent · converge sync-models.pydownload · verify · index llama-swap :9292 · Open WebUI :3333 · Traefik :443one OpenAI-compatible surface for every agent you own
Two files, one engine, reproducible on every box you buy next.

§ 03 quick start

Four commands. Then coffee.

  1. 01

    Clone & copy the config

    git clone https://github.com/Kubementat/kwisatz.git
cd kwisatz
cp machine-config-inference.yml.example machine-config.yml
cp models.yml.example models.yml

    Both examples ship disabled — flip enabled: true on what you want.

  2. 02

    Install the orchestrator's only prerequisite

    curl -LsSf https://astral.sh/uv/install.sh | sh

    uv is the only thing the orchestrator needs. Everything else is installed for you.

  3. 03

    Preview, then apply

    ./kwisatz status   # what will run, in order
./kwisatz apply    # stops at the first failure — fix, re-run, converge

    Sudo is asked once, up front, and held. Task output streams live and lands in logs/kwisatz-<ts>-<target>.log.

  4. 04

    Or let an AI agent drive

    Read and execute the instructions in ./skills/kwisatz-assistant/SKILL.md

    The repo ships an Agent Skill, so Claude Code, pi or any compatible agent can pick, configure and run the right setup with you.

Ordering matters, and kwisatz knows it. setup-sshd must run before configure-firewall: sshd moves to its new port first, the firewall then opens it. Reversed, UFW goes default-deny around a port nothing listens on and your next reconnect is the last one. The shipped example keeps that order — keep it.

§ 04 the cli

One command,
JSON when a machine is asking.

There is no second implementation of apply. The CLI plans, then starts the same pyinfra deploy, and a conformance test asserts the (command, env, stdin) triple either way.

./kwisatz init

create machine-config.yml from the example, then run doctor

kwisatz status

what will run, in order, with env keys and the real command line

kwisatz apply

provision everything enabled, stop at the first failure

kwisatz apply traefik

one target — same env, no config edit, same code path

kwisatz services

what is running right now, and whether it is healthy

kwisatz doctor

OS, sudo, uv, docker, GPU, disk, port collisions, unhealthy stacks

kwisatz config enable vllm

flip one flag; comments and ordering survive byte-for-byte

kwisatz run forgejo -- --force

verbatim arguments the CLI cannot know about

# the whole CLI, as data — no README reading required
./kwisatz commands --json | jq '.result.commands[].name'

# install one target and report the outcome
./kwisatz apply traefik --json | jq '{ok, exit_code, items: [.result.items[] | {target, status}]}'

# what is running, for a dashboard or an agent
./kwisatz services --json | jq '.result.items[] | {name, manager, state, ports}'
exit codes are a contract
0ok5unhealthy stack / timeout
1task failed8partial (--keep-going)
2usage9another run holds the lock
3config / unknown target6,7reserved, published, unused
4prerequisite missing

§ 05 the catalog

Not just inference.

Every entry is a tasks/<name>.sh you can enable, skip, or run alone. Search by name or by what you need.

loading…

    § 06 model catalog

    A catalog, not a
    download folder.

    models.yml declares which models exist on the box, how they are served, and which coding agent they are wired into. kwisatz apply sync-models converges the reality: download what is missing, verify, index, and update the agent config.

    • Two-stage workflow: declare in YAML, then sync.
    • GGUF for llama.cpp, vLLM-native models, pinned versions.
    • Same catalog feeds llama-swap, Open WebUI and your agents.
    # models.yml — providers, models, agent wiring
providers:
  - name: local
    baseUrl: http://localhost:9292/v1
    apiKey: ${LLAMA_SWAP_KEY}
    models:
      - name: qwen3.8-27b
        type: local
        download:
          - "hf download hf://unsloth/Qwen3.8-27B-GGUF/...-Q4_K_XL.gguf"
        serve:
          cmd: ${llama-server-bin} --port ${PORT} -m ${models-dir-hf}/…
        agent:
          contextWindow: 262144
          reasoning: true
          input: [text, image]

# then:
./kwisatz apply sync-models

    § 07 docs & guides

    Read the manual, not the code.

    The README

    Full reference: orchestrator, YAML format, re-run policy, sudo contract.

    read →

    Automations

    Every task script with its env vars, defaults and gotchas.

    read →

    Traefik v3

    TLS via Let's Encrypt, security headers, rate limiting, protected dashboard.

    read →

    NetBird mesh VPN

    Reach your LAN from mobile data or a roaming laptop — self-hosted, no cloud relay.

    read →

    Matrix team chat

    Synapse, Element, MAS and wellknown wired into Traefik.

    read →

    LLM gateway

    One LiteLLM endpoint (OpenAI and Anthropic compatible) in front of several llama-swap boxes.

    read →

    vLLM

    High-throughput serving for the boxes where GGUF is the wrong shape.

    read →

    How to test

    How to run a task safely, verify a change, and not wreck your box.

    read →

    CLI specification

    The exit-code and JSON contract, section by section.

    read →