Hey it’s never too late to get things better under control! My parents only just started that journey and they’re old enough to be grandparents

Honestly the questions you’re posing require a level of market analysis that could fill an entire white paper and be sold for way more money than I want to think about. Its a level of market analysis I don’t want to dive into. My gut instinct from having worked in the tech industry, working with datacenters and datacenter hardware at large companies is that the AI industry will contract significantly when the bubble pops. I’m sure I could find real data to support this prediction but the level of analysis that would require and the hours of work are simply more than it’s worth for an internet comment.

You have factors including what hardware is being deployed to meet AI bubble demand, how the networking might be setup differently for AI compared to general GPU compute, who is deploying what hardware, what the baseline demand for GPU compute is if you simulate no present AI bubble, etc. etc. it’s super neat data analysis but I ain’t got the time nor appetite for that right now

That’s super interesting! I didn’t know other languages handled side thoughts better!

Oh yeah machine learning as a technology will survive, and eventually it will be implemented where it can do what it’s really good at, but right now it’s being shoved into everything to do things it isn’t good at, so you end up with a super expensive to run, energy inefficient tool that runs worse than with traditional algorithms that can be run client side or on a single much cheaper server (I’m oversimplifying the server architecture for brevity)

Think customer service chatbots on ever car dealership’s website. Traditionally these were extremely simplistic and usually just had canned responses based on keywords in the customer’s written message and would quickly cascade the customer to a real customer service rep as soon as things got out of scope. Now with LLMs companies are running those as the customer service chatbots and the LLM can do anything from agreeing to sell a new car for a dollar to providing scam or invalid contact info to referring the customer to a competitor. There’s no knowing what the AI will do because it’s non-deterministic and you don’t want that in customer service!

Right now we’re in the bubble phase where every single company is finding some way to shoehorn AI into its business model so they can brag about it. Fucking Logitech added a remappable AI button that brings up a ChatGPT interface and just spends Logitech’s money on tokens with ChatGPT. That’s pure bubble behavior. Once the bubble pops we won’t have literally every single time you open a car dealership page spending an LLM token or 5, you won’t have Amazon running AI chatbots on every product page just for asking about that product, you won’t have every website just giving away free unrestricted access to LLMs. That’s what I’m talking about.

AI demand will drop when the bubble pops, and while it will be higher than it was 8 years ago, everyone is going to be very skeptical of anything AI, just like folks are still skeptical of mortgage backed securities over 15 years later, or just like people are skeptical of commerical websites without a clear method of financing 25 years after the dotcom bubble. People remember these things and will take a while to warm up to the idea again

Is there enough demand for thousands of servers with purpose built ARM processors (which may or may not have any publicly available kernel support) driving 4-8 600w a pop Nvidia datacenter chips though? Yes some will be repurposed but there simply won’t be the demand to fill immediately. Realistically what will happen is companies operating these datacenters will liquidate the racks, probably liquidate some of the datacenters entirely and thousands of servers will hit the secondhand market for next to nothing. While some datacenter structure city empty and unmaintained until they’re either bought up to be repurposed, bought up to be refurbished and brought back into datacenter use of torn down, just like an empty Super Walmart location

Some of the datacenters will be reworked for general compute, maybe a couple will maintain some AI capacity, but given the sheer quantity of compute being stood up for the AI bubble and the sheer scale of the bubble, basically every major tech company is likely to shrink significantly when the bubble pops, since we’re talking companies that currently have market caps measured in trillions, and literally a make up full quarter of the entire value of the New York Stock Exchange, it’s going to be a bloodbath.

Remember how small the AI field was 6 years ago? It was purely the domain of academic research, fighting for scraps outside of a handful of companies big enough to invest in am AI engineer or two on the off chance they could make something useful for them. We’re probably looking at a correction back down to nearly that scale. People who have drank the coolaid will wake up one day and realize how shit the output of generative AI is compared to the average professional’s human work

Machine learning models have much different needs that crypto. Both run well on gaming GPUs and both run even better on much higher end GPUs, but ultimately machine learning models really really need fast memory because it loads the entire weights into graphics memory for processing. There’s some tools which will push it to system memory but these models are latency sensitive so crossing the CPU bus to pass 10s of gigabytes of data between the GPU and system memory is too much latency.

Machine learning also has the aspect of training vs inference, where the training portion will take a long time, will take less time with more/faster compute and you simply can’t do anything with the model while it’s training, meanwhile inference is still compute heavy it doesn’t require anywhere near as much as the training phase. So organizations will typically rent as much hardware as possible for the training phase to try to get the model running as quickly as possible so they can move on to making money as quickly as possible.

In terms of GPU availability this means they’re going to target high end GPUs, such as packing AI developer stations full of 4090s and whatever the heck Nvidia replaced the Tesla series with. Some of the new SOCs which have shared system/vram such as AMD’s and Apple’s new SOCs also fill a niche for AI developer and AI enthusiasts too since that enables large amounts of high speed video memory for relatively low cost. Realistically the biggest impact that AI is having on the Gaming GPU space is it’s changing the calculation that AMD, Nvidia and Intel are making when planning out their SKUs, so they’re likely being stingy on GPU memory specs for lower end GPUs to try to push anyone with specific AI models they’re looking to run to much more expensive GPUs

Huh I’d never looked into that! Thank you!

I mostly used zram because there’s an easy-peasy script in the Debian repos making it dead simple to setup and never looked deeper into it

Side thoughts in the middle of sentences are definitely weird in written form. Heck they get messy in spoken form too! Some punctuation to help the reader understand what’s being communicated can go a long way, and in the format of a forum discussion where folks will quickly tap out a brain fart from a 5" slab of plastic and glass, when I see what appear to be multiple sentences mashed together into one incoherent one, I’ll generally assume it’s a writing error, because folks don’t proof read, they aren’t writing literature with multiple drafts. They’re just quickly jotting down a thought or two and somethimes errors compound with that level of quick communication

Yikes you’re literally financing your hobby! Better financial move is to get a used system to start with (usually a used gaming PC can be had for like $500ish, and I’m sure there’s plenty of people online you can ask for help speccing something out), squirrel away money for a couple of years (I like to keep a dedicated savings account just for big purchases like tech upgrades. $40 biweekly dissearing into another account you don’t touch is $2k every 2 years, so a 4 year complete refresh cycle for 2 people) and buy when you feel like it. Good news is it’s a small enough amount of cash to easily right the financial ship but still yikes!

Are people throwing 128GB in for shits and giggles these days? I’m kinda out of the loop and still used to people arguing about if 32GB is excessive or not. Used to be only needed for media production and honestly it’s still debatable for pure gaming systems if more than 16GB is overkill

Its not just electricity but also water consumption, and noise pollution if not particulate pollution too.

They provide little benefit to the surrounding communities with very few jobs by design and just consume local resources at usually discounted bulk rates to sell a service that possibly nobody locally would be in the market for

Also the scariest part of this datacenter inflation is how much of these new data centers are going to be abandoned within the next 5 years when the AI bubble pops and suddenly the companies spending like crazy on datacenter growth need to cut back. There’ll be lots of big empty buildings outside of small towns costing taxpayers a ton of money, much like when any big box store closes up shop. You can either spend a ton of money tearing it down, a ton of money rebuilding it into something useful, a ton of money attracting another business which may or may not front the cost for remodeling the space or a ton of money maintaining the empty property so it doesn’t fall over and become even more of a blight. There’s no winning for these small municipalities that just get used and abused by large businesses

Zram on Linux is awesome! I’ve used it heavily in both memory constrained systems and systems with 16+GB of memory running very poorly optimized code

Running for example, Cities Skylines with 40GB of mods can easily lead to running memory usage being 20-30GB uncompressed. With zram I can load that same mod load out on a 16GB laptop with no swap and it won’t crash where it would crash for being out of memory before.

Another example is Proxmox with over-provisioned lxc containers. Since it’s still the kernel scheduler running all of the processes in those containers zram can keep them all running nicely even when a heavily modded Minecraft server gets a few players online and starts pushing past memory limits, where before I set it up I’d have some of the Minecraft server processes get killed to free up memory resources without warning or proper logging by Minecraft

Edit to add: my daughter’s first laptop has only 4GB of memory and runs a decade old Celeron booting from a spinning hard drive, the definition of budget ewaste. Zram makes it so it’s CPU limited running Minecraft rather than memory limited!

I mean since it’s driven by demand from companies running AI models it’s mostly impacting higher speed modules with lower speed/capacity kits being less significantly impacted. Granted that’s also pushing demand up for those lower speed kits since for gamers quantity matters far more than speed, meanwhile AI really cares about speed most

The good thing about new AM4 boards being available at this point in time is you have options to keep older hardware running. Usually the CPU and memory will out-survive motherboard. Much like those new Chinese motherboards supporting 4th and 6th gen Intel CPUs, this is great for longevity and reduces how much production is needed

In a sane world, the limitations of a CPU socket would be reached, and then newer SKUs would no longer be released

I’d argue that it would be best if computers were more like cars, a new platform gets released each decade or so, and small improvements are made to individual parts but the parts are largely interchangable within the platform and produced for a decade or two before production is retired. More interchangable parts, slower release cycle and more opportunities for repair instead of replacement

Gaming GPUs during normal crypto markets don’t compute fast enough to mine crypto profitably, but if crypto prices get high enough such as during a boom cycle, it can become profitable to mine on gaming GPUs

Edit to add: For crypto there’s basically a set speed that any given GPU mines at. The hash rate specifically. It really doesn’t change noticably over time through software updates, nor does the power consumption of the GPU. Its basically a set cost per cryptocurrency mined with any given hardware. If the value earned by mining can exceed the cost to run the GPU then GPU mining can quickly start making sense again.

There was a nice window from about a year or two ago to about 3 months ago where no individual components were noticably inflated. Monitors took the longest to recover since the pandemic shortages so that was arguably around the beginning of this year that they seemed to fully normalize

Its funny because at work we’ve been pushing hard on Windows 11 refreshes all year and warning that there will likely be a rush of folks refreshing at the last possible minute at the end of the year inflating prices. And we ended up being correct on the inflated prices part but it was actually the AI bubble that did it

You can load a can’t stress how good planetary map into RAM wholly

This is the part people are struggling with, because it’s probably 3 difference sentences mashed together, whether in your head, by your fingers or by an autocarrot, but regardless its completely incomprehensible as a result

16GB is fine for most games, 32GB is handy if you run a lot of poorly optimized mods, but also memory compression helps with that significantly. 8gb of memory gets pretty cozy for gaming these days though, but again memory compression makes a big difference.

Both my and my wife’s computers have 32GB of RAM because it was cheap enough to not be worth worrying about, but if RAM is expensive you don’t need a ton of it to enjoy your games

This assumes machine learning models are able to get better than they currently are. Newer models have been plateauing in quality of outputs (improvements have been noticable in video and image generation, but even that is slowing down)

I don’t think we’re going to see machine learning models that perform well enough to create printing press level change to the world