Vllm is a bit better with parallelization. All the kv cache sits in a single “pool”, and it uses as many slots as will fit. If it gets a bunch of short requests, it does many in parallel. If it gets a long context request, it kinda just does that one.

You still have to specify a maximum context though, and it is best to set that as low as possible.

…The catch is it’s quite vram inefficient. But it can split over multiple cards reasonably well, better than llama.cpp can, depending on your PCIe speeds.

You might try TabbyAPI exl2s as well. It’s very good with parallel calls, thoughts I’m not sure how well it supports MI50s.

Another thing to tweak is batch size. If you are actually making a bunch of 47K context calls, you can increase the prompt processing batch size a ton to load the MI50 better, and get it to process the prompt faster.

EDIT: Also, now that I think about it, I’m pretty sure ollama is really dumb with parallelization. Does it even support paged attention batching?

The llama.cpp server should be much better, eg use less VRAM for each of the “slots” it can utilize.

I’ll save you the searching!

For max speed when making parallel calls, vllm: https://hub.docker.com/r/btbtyler09/vllm-rocm-gcn5

Generally, the built in llama.cpp server is the best for GGUF models! It has a great built in web UI as well.

For a more one-click RP focused UI, and API server, kobold.cpp rocm is sublime: https://github.com/YellowRoseCx/koboldcpp-rocm/

If you are running big MoE models that need some CPU offloading, check out ik_llama.cpp. It’s specifically optimized for MoE hybrid inference, but the caveat is that its vulkan backend isn’t well tested. They will fix issues if you find any, though: https://github.com/ikawrakow/ik_llama.cpp/

mlc-llm also has a Vulcan runtime, but it’s one of the more… exotic LLM backends out there. I’d try the other ones first.

AFAIK some outputs are made with a really tiny/quantized local LLM too.

And yeah, even that aside, GPT 3.5 is really bad these days. It’s obsolete.

“a lot of people around him did the same.”

Your friend is in la la land.

I know a rich couple that would love to do this and 100% can’t because it’d be ludicrously expensive, even with no kids. And that’s in a place way cheaper than New York City.

…Maybe it was more practical when his parents were working, though?

Bloefz has a great setup. Used Mi50s are cheap.

An RTX 3090 + a cheap HEDT/Server CPU is another popular homelab config. Newer models run reasonably quickly on them, with the attention/dense layers on the GPU and sparse parts on the CPU.

This is the way.

…Except for ollama. It’s starting to enshittify and I would not recommend it.

The iPhone models are really bad. They aren’t representative of the usefulness of bigger ones, and it’s inexplicably stupid that Apple doesn’t like people pick their own API as an alternative.

If there’s ever a need, you could try whatever open generation models use for post processing.

I’m a huge iced tea fan.

Compression throws me off.

That AI ‘hiss’ is ringing it my ears, but its very, very similar to YT uploads that have been re-encoded like 10 times. Which is a lot of them.

Out of curiosity, I made spectrograms of the AI song and concert that sounds ‘clean’ yet kinda noisy/compressed to me. I won’t tell you which one is which:

Source song, if you are curious: https://www.youtube.com/watch?v=MydFq0io-tQ

…It’s because Mamdani is not a jerk?

Look, Trump is a human being in an incredible distortion sphere. His actions/reactions tend to skew more “interpersonal” and emotional than Machiavellian, like how he acts immediately after he meets Zelensky or a lawmaker or sees a bad Tweet or whatever; he was like before politics.

Step in Trump’s shoes: he sees Mamdani on Fox News, on his Twitter feed, from his circle. This guy is Socialist Satan from his perspective.

Then the guy walks into Trump’s office, is actually super nice and reasonably balanced, and loves New York too? He’s clearly not an alien lizard like a Clinton.

I’m not surprised Trump reacted this way at all. It’s totally in character. And if you understand this, you understand why Trump has so much popularity: in spite of the massive toxicity he commands, he’s quite human for a politician.

And IMO… your 3080 is good for ML stuff. It’s very well supported. It’s kinda hard to upgrade, in fact, as realistically you’re either looking at a 4090 or a used 3090 for an upgrade that’s actually worth it.

Oh no, you got it backwards. The software is everything, and ollama is awful. It’s enshittifying: don’t touch it with a 10 foot pole.

Speeds are basically limited by CPU RAM bandwidth. Hence you want to be careful doubling up RAM, and doubling it up can the max speed (and hence cut your inference speed).

Anyway, start with this. Pick your size, based on how much free CPU RAM you want to spare:

https://huggingface.co/ubergarm/GLM-4.5-Air-GGUF

The “dense” parts will live on your 3080 while the “sparse” parts will run on your CPU. The backend you want is this, specifically the built-in llama-server:

https://github.com/ikawrakow/ik_llama.cpp/

Regular llama.cpp is fine too, but it’s quants just aren’t quite as optimal or fast.

It has two really good built-in web UIs: the “new” llama.cpp chat UI, and mikupad, which is like a “raw” notebook mode more aimed at creative writing. But you can use LM Studio if you want, or anything else; there are like a bazillion frontends out there.

Since there is generated video, it seems like someone solved this problem.

Oh yes, it has come a LOONG way. Some projects to look at are:

https://github.com/ModelTC/LightX2V

https://github.com/deepbeepmeep/Wan2GP

And for images: https://github.com/nunchaku-tech/nunchaku

Video generation/editing is very GPU heavy though.

I dunno what card you have now, but with text LLMs (or image+text input LLMs), hybrid CPU+GPU inference is the trend days.

As an example, I can run GLM 4.6, a 350B LLM, with measurably low quantization distortion on a 3090 + 128GB CPU RAM, at like 7 tokens/s. If you would’ve told me that 2-4 years ago, my head would have exploded.

You can easily run GLM Air (or other good MoE models) on like a 3080 + system RAM, or even a lesser GPU. You just need the right software and quant.

It’s a lot to keep up with

Massive understatement!

The next project will be some kind of agent. A ‘go and Google this and summarize the results’

Yeah, you do want more contextual intelligence than an 8B for this.

The image models I’m using are probably pretty dated too

Actually SDXL is still used a lot! Especially for the anime stuff. It just got so much finetuning and tooling piled on.

Do it!

Feel free to spam me if I don’t answer at first. I’m not ignoring you; Lemmy fails to send me reply notifications, sometimes.

Also, I’m a quant cooker myself. Say the word, and I can upload an IK quant more specifically tailored for whatever your hardware/aim is.

You can run GLM Air on pretty much any gaming desktop with 48GB+ of RAM. Check out ubergarm’s ik_llama.cpp quants on Huggingface; that’s state of the art right now.

Yeah, exactly.

Cromite’s explicit focus is, literally, antifingerprinting. With the goal of breaking cross site tracking I guess.

A more accurate goal for Tor/Mullvad is anonymizing, e.g. “blending in with the crowd.”

It’s like radically changing your clothes every day vs wearing super incognito stuff. Different means, each more optimal for different aspects of security/privacy.

You should be running hybrid inference of GLM Air with a setup like that. Qwen 8B is kinda obsolete.

I dunno what kind of speeds you absolutely need, but I bet you could get at least 12 tokens/s.