(2025-10-13) A short rant about "democratizing" genAI, hobbyist software etc
----------------------------------------------------------------------------
As you might have noticed, my break from phlog posting had been extended by a
week from what I'd been planning. Too much had been going on throughout this 
month, so I definitely needed to keep my attention span narrow enough. I 
promise that I am going to eventually return to the previous topics like 
homebrew VMs or abacus, but right now I'd like to rant about the state of 
genAI for "mere mortals" and some other adjacent topics without a particular 
structure. So, here's my chain of thoughts (no pun intented).

Recently, I had something remind me about the existence of KoboldCpp ([1]): a
single-binary LLM inference runner that uses llama.cpp under the hood but 
also incorporates a simple web-based chat UI, a set of various API endpoints 
(its own, OpenAI-compatible and Ollama-compatible, among many others) and a 
simple GUI to easily run not only text generation, but also image 
generation, TTS and speech recognition models (via the included 
stable-diffusion.cpp, TTS.cpp and whisper.cpp distributions respectively). A 
couple years ago, before LM Studio even was a usable thing, KoboldCpp 
already gained some traction among lusers who just wanted something fully 
local for roleplaying scenarios. It was one of the first pieces of software 
that was considered something to "democratize" running LLMs for a layman. 
For me, however, as fast and efficient as it is, there are two caveats: 1) 
it doesn't directly use the llama.cpp binaries but the underlying GGML 
engine instead (exposing a totally different set of commandline parameters), 
hence some strange defaults like the default maximum generated token limit, 
2) the GGML engine version it uses doesn't get updated as often as the 
upstream llama.cpp, so some models still can be out of reach, like the 
recent Granite 4 MoE series. It's only a matter of time and patience when 
the newer engine gets finally merged into Kobold.

As of now, I haven't yet explored its image generation or speech capabilities
but I run the text generation server as follows:

#!/bin/sh
# Sensible defaults to run KoboldCpp
CTXSIZE=32768
# hardware options
# HARDOPTS="--usecpu --gpulayers 0 --usemmap"
HARDOPTS="--usevulkan --flashattention --gpulayers 99"
# run it
koboldcpp --defaultgenamt $(($CTXSIZE<8192?$CTXSIZE:8192)) --skiplauncher
--contextsize $CTXSIZE $HARDOPTS $*

Then I just pass the GGUF file as the parameter to the script and that's it.
In the model list in the API, it gets exposed as 
"koboldcpp/filename_without_ext". For CPU-only inference, the second 
(commented) line is necessary. Also, as you can see, KoboldCpp can only 
generate 8192 tokens at a time at most, even when the context window is 
larger. The upstream llama.cpp doesn't have such limitations. On the upside, 
in case you need to only deploy KoboldCpp to non-GPU systems, there is a 
"nocuda" binary version that weighs much less, and also an "oldpc" version 
that disables the AVX2 instructions automatically (which requires a separate 
CLI flag in the other builds) and performs some other tricks to run the 
models on older hardware.

Well, why would I even consider using this instead of the usual llama-server
(or llamafile, previously covered here, if I need a single-file deployment)? 
Well... KoboldCpp really is easier on RAM consumption. I've purchased a 
really cheap VPS (around $23/year!) with the only caveat being it having a 
single core and 512 MB or RAM. Are there any modern LLMs suitable for such 
amounts of RAM? Sure, there are, like Gemma 3 270M in the 4-bit QAT version. 
But аre there any engines suitable for running this LLM on such amounts of 
RAM? Well, KoboldCpp is one of them, I just had to append the "--noavx2" 
flag for it to not throw the "illegal instruction" error and still get up to 
the speed of 11 to 14 tokens per second (given that the context window size 
had to be decreased to 4096 tokens). Ten times slower than on a "normal" 
system but still perfectly adequate. The llamafile-based deployment, on the 
other hand, showed about 5 to 6 t/s at most while obviously consuming much 
more RAM per inference. If/when time permits, I'm also going to dig up some 
of my older hardware and run some more tests.

Another side of the coin is the actual usage of all this goodness. Recently,
a controversial question has emerged: "If LLMs became so good at coding, 
where's all the hobbyist level and indie software, why aren't we 
experiencing a boom of it?" Well, from my own experience, I think I know the 
answer to this question, and it's not a simple one either, it's as 
multifaceted as the technology itself. First, true indie developers most 
often are resource-conscious, so they will be the last ones to adopt LLMs 
for any coding assistance. They, and some of them rightfully so, view 
themselves as artisans instead of code monkeys, and like to be in control of 
everything that happens in their creations. I share this view myself, so I'm 
never going to use LLMs for anything but some boring but inevitable 
boilerplate parts of code. The second reason we still aren't experiencing 
that indie boom to the fullest is that many "vibe coders" really thought 
that LLM could think FOR them throughout the entire development process, 
rather than merely assisting with the boilerplate coding part. Offloading 
design decisions and mission-critical bits of code to LLMs already has lead 
to some disasters, with many and many more to come. We just aren't at the 
"good enough" phase yet, no matter how corporate marketoids are trying to 
convince you otherwise. The third reason is something I already have 
discussed a bit in the past in this phlog, and it has little to do with LLMs 
per se: desktop software just isn't that popular anymore. An average vibe 
coder would try generating a mobile app at best, but most of the time, it's 
just some React-based browser-oriented crap. I mean, instead of React, you 
can insert Next.js, Expo or whatever RAM-hogging framework is popular this 
week of the month, that doesn't really matter. What matters is that, in the 
eyes of the masses, the definition of "software" really has shifted towards 
this. And because of the amount of already existing JSX/TSX garbage to train 
on, it's pretty much the only coding thing LLMs are *kinda* good at now. Can 
it be usable by the general public? Maybe. Can it compete with the industry 
giants? Probably. Does that qualify as "indie development" in my eyes? Hell 
no. Truly independent software never requires a gigabyte-sized browser 
engine to run it.

Want to create independent desktop software? Learn Tcl/Tk, or at least
Tkinter if you already know Python (just because Python already ships with 
it). Wanna go mobile too? Learn Go + Fyne then. Wanna go online? Learn 
Elixir. There are lots of better ways of doing stuff than just succumbing to 
the mainstream crapware frameworks for languages that aren't supposed to be 
used for that stuff in the first place. By choosing the right tool for the 
job in the beginning, you make it much easier for your future self. And 
then, and only then, like I already said some time ago, you may use LLMs to 
assist you. Not think for you but to help you with writing the boring parts 
of your implementation while you're still in control. Sober, aware and 
independent.

--- Luxferre ---

[1]: https://github.com/LostRuins/koboldcpp