And IMO… your 3080 is good for ML stuff. It’s very well supported. It’s kinda hard to upgrade, in fact, as realistically you’re either looking at a 4090 or a used 3090 for an upgrade that’s actually worth it.

Oh no, you got it backwards. The software is everything, and ollama is awful. It’s enshittifying: don’t touch it with a 10 foot pole.

Speeds are basically limited by CPU RAM bandwidth. Hence you want to be careful doubling up RAM, and doubling it up can the max speed (and hence cut your inference speed).

Anyway, start with this. Pick your size, based on how much free CPU RAM you want to spare:

https://huggingface.co/ubergarm/GLM-4.5-Air-GGUF

The “dense” parts will live on your 3080 while the “sparse” parts will run on your CPU. The backend you want is this, specifically the built-in llama-server:

https://github.com/ikawrakow/ik_llama.cpp/

Regular llama.cpp is fine too, but it’s quants just aren’t quite as optimal or fast.

It has two really good built-in web UIs: the “new” llama.cpp chat UI, and mikupad, which is like a “raw” notebook mode more aimed at creative writing. But you can use LM Studio if you want, or anything else; there are like a bazillion frontends out there.

Since there is generated video, it seems like someone solved this problem.

Oh yes, it has come a LOONG way. Some projects to look at are:

https://github.com/ModelTC/LightX2V

https://github.com/deepbeepmeep/Wan2GP

And for images: https://github.com/nunchaku-tech/nunchaku

Video generation/editing is very GPU heavy though.

I dunno what card you have now, but with text LLMs (or image+text input LLMs), hybrid CPU+GPU inference is the trend days.

As an example, I can run GLM 4.6, a 350B LLM, with measurably low quantization distortion on a 3090 + 128GB CPU RAM, at like 7 tokens/s. If you would’ve told me that 2-4 years ago, my head would have exploded.

You can easily run GLM Air (or other good MoE models) on like a 3080 + system RAM, or even a lesser GPU. You just need the right software and quant.

It’s a lot to keep up with

Massive understatement!

The next project will be some kind of agent. A ‘go and Google this and summarize the results’

Yeah, you do want more contextual intelligence than an 8B for this.

The image models I’m using are probably pretty dated too

Actually SDXL is still used a lot! Especially for the anime stuff. It just got so much finetuning and tooling piled on.

Do it!

Feel free to spam me if I don’t answer at first. I’m not ignoring you; Lemmy fails to send me reply notifications, sometimes.

Also, I’m a quant cooker myself. Say the word, and I can upload an IK quant more specifically tailored for whatever your hardware/aim is.

You can run GLM Air on pretty much any gaming desktop with 48GB+ of RAM. Check out ubergarm’s ik_llama.cpp quants on Huggingface; that’s state of the art right now.

Yeah, exactly.

Cromite’s explicit focus is, literally, antifingerprinting. With the goal of breaking cross site tracking I guess.

A more accurate goal for Tor/Mullvad is anonymizing, e.g. “blending in with the crowd.”

It’s like radically changing your clothes every day vs wearing super incognito stuff. Different means, each more optimal for different aspects of security/privacy.

You should be running hybrid inference of GLM Air with a setup like that. Qwen 8B is kinda obsolete.

I dunno what kind of speeds you absolutely need, but I bet you could get at least 12 tokens/s.

That’s not strictly true.

I have a Ryzen 7800 gaming destkop, RTX 3090, and 128GB DDR5. Nothing that unreasonable. And I can run the full GLM 4.6 with quite acceptable token divergence compared to the unquantized model, see: https://huggingface.co/Downtown-Case/GLM-4.6-128GB-RAM-IK-GGUF

If I had a EPYC/Threadripper homelab, I could run Deepseek the same way.

Most aren’t really running Deepseek locally. What ollama advertises (and basically lies about) is the now-obselete Qwen 2.5 distillations.

…I mean, some are, but it’s exclusively lunatics with EPYC homelab servers, heh. And they are not using ollama.

Already done, see: https://github.com/uazo/cromite

When I go to the fingerprint test, a bunch of the values like canvas resolution and timezone are randomized.

…Not everything, though.

I’m kinda surprised it’s still around, and popular.

I guess it still has a lot of SEO and such.

I don’t get the analogy… of course a bunch of students using different tools with different inputs will yield different results? But if they use the same model and input at zero temperature, they will, in fact, get the same results, just like any code.

Predictability has never been a strength of ML, of course.

…That’s not really what it’s for. It’s for finding exotic stars in a mass of astronomical data on a budget, or interpoliating pixels in an image, or for identifying cat videos reasonably well. That’s still a useful tool. And the modern extension of getting a glorified autocomplete engine to press some buttons automatically is no different if structured and constrained appropriately.

The obvious problem, among many I see, is that these Tech Bros are selling underbaked… no, not even half cooked agenic systems as sapient magic lamps. Not niche tools for very specific bits of automation. Just look at the language Suleyman is using:

I grew up playing Snake on a Nokia phone! The fact that people are unimpressed that we can have a fluent conversation with a super smart AI that can generate any image/video is mindblowing to me.

Some linux terminals like cosmic and wezterm support tabbing.

The Windows one is neat though.

It’s trained in to the point its unconscious now. It’s like part of humanity’s collective psyche.

Well, according to Microsoft, your AI agent should be… err, neutering AI agents for you?

It will entirely depend on the price.

If it’s expensive, odds are you’re better off, like, finding or cobbling together a used RTX 3080 deskop or something like that.

^ This.

It’s a neat, under-construction tool.

A. Tool. An ‘agent’ to do niche things is neat.

…But I don’t need a chatbot on my fucking toaster.

I’m BlameTheAntifa and I have a distro-hopping addiction.

“Hi, BlameTheAntifa.” The circle of disto-hoppers echos.