Option 3: JP Morgan, Goldman, etc told them to settle down or they’ll get throttled economically.
yup, that’s a totally valid option if the costs for their bigger models are going through the roof
It’s the energy crisis. They realized “oh shit money is real now” and are trying to sooth investors.
Interested how Monday will go. This is not what investors want to hear, even if it’s softened by markets being closed.
The whole bubble might be about to pop.
Have been hearing that for 2 years now
The market can stay irrational for a long time but it does eventually have to happen. The 2008 crisis was the result of overleveraging in 1999-2003 and interest rate rises from 2004-2007, so it can take years for it to crumble. Two years is a long while but it could easily take two more before people realise the emperor has no clothes.
The nature of a bubble means that the longer you go the more likely it actually is. And this is a bubble I frankly can’t wait to pop.
Significant portions of Lemmy are turning into debate bro bro hreads on AI and its relevance.
I heard on the Framework subreddit that they just secured a cheaper batch of RAM and refunded their customers, so could be something or maybe it’s just a blip. I’m also wondering if the recent Flash model releases are disrupting investor interest in the AI datacenter projects especially since running them on local hardware is starting to become more feasible, and for businesses in particular possibly more cost-effective.
“AI” seems to be best put to use performing specific automated tasks. This is something a lot of us have been saying for a while now, but they tried to make what is essentially an industrial tool (as in boutique AI for specific industries), into both slide deck candy for enterprise C-suites, and an expensive consumer plaything. Enterprise customers demand real results for their money, and consumers do not care about the novelty of generative AI enough to recoup the costs of this insane level of infrastructure development. Add what is essentially shaping up to be runaway inflation, and well, yeah. No one cares about this shit.
They came at it like this because they are Silicon Valley and for the better part of 3 decades now, they have operated under the assumption that they are always right, and that if they can survive running on a loss at first, eventually they will acheive market dominance and monopolize their product. I just don’t see it happening. I see this more like the early dot com days. A lot of investment, and a lot of failures. But eventually, for better or worse, a few of them figured out how to make it work. This time around however, capitalism itself is failing.
The backlash against AI is more than a mirage, and SpaceX’s IPO didn’t go the way they wanted, so now they’re looking for reasons to delay their IPOs without revealing how screwed they are. This will probably slow data center buildout even more, and Softbank and Oracle are gonna die.
It’s because they’re hitting model size constraints. There’s only so much memory bandwidth you can get between racks or even rack spaces and memory bandwidth is the constraint for nearly every ml thing.
Expect a reversal once a more memory dense component hits.
There’s no reason to think that the architecture itself can scale indefinitely. It might very well be that LLMs have some hard constraints on the scope of the problems they’re capable of solving.
Of course, that’s what I’m saying. Physical constraints of hardware mean there’s a limit to how much further (read: larger in terms of working memory footprint, because that’s how they’re getting “better” and better “frontier” models) development can continue until a more dense component comes along.
Every singularity a sigmoid.
I meant that simply making models bigger might not actually make them more capable. So even if you had unlimited hardware to play with, you might have to find a different approach.
You could create a way to measure the idea of capability that would bear that out but from a pure discrete mathematics perspective, no, you only get better with a larger memory footprint.
There’s a lot of ways to make that faster or make that behave like a process running on a bigger memory footprint, but ultimately that’s the constraint.
And companies competing in the field of ai can’t justify the expense of cutting down their gigantic model to only know how to identify wood because that has a known and limited impact. They already said they’re shooting for unlimited immeasurable impact on the scale of replacing all human labor and got massive funding for it.
It doesn’t matter if it’s easier to do one backflip, you asked me to triple dog dare you to do a million backflips. Well… we’re waiting!
Again, there is no reason to think that you can just keep making the model bigger and keep getting improved capability that way. In fact, we already know that’s not the case because simply making them bigger stopped being the focus. The real breakthrough is going to come from better algorithms.
You said there’s no reason to think you can just keep making the model bigger and keep getting improved capability.
there’s the structure of the neural network itself. Fundamentally, adding nodes and layers increases the ability of the model to handle more complex input.
Then there’s the actual models we see in use. They are literally as large as the hardware allows. The only reason to use smaller models are to fit some constraint.
So both by the book and in practice bigger is always better.
Now we can’t always go big. I can’t afford to purchase a dgx or even upgrade my wiring to power it, let alone pay the power bill it would rack up or all the other utilities alone when my wife leaves me because of the sound.
My computer can only fit so many expansion cards and pcie is so slow compared to hbm that I’m better off running a small model quickly that fits on one card as opposed to a larger one slowly across several cards.
But those are all constraints. When I replace my motherboard with supermicro gpu host fabric I no longer am limited by the pcie bandwidth and can quickly use models that fit across several cards.
I do agree with you that the future is smaller models, not because of the fundamental nature of the concepts involved but because of the complex constraints that are coming into play.
The problem is with the context and data propagation through the network. As you keep making it bigger it becomes slower and less focused. And there is research showing that smaller models do outperform large ones on some tasks https://cacm.acm.org/news/bigger-not-necessarily-better
What I expect we’ll see going forward is more hierarchical architecture where you have finely tuned models for specific tasks with a general routing model on top. This is basically already where MoE architecture is moving now. We might also see stuff like neurosymbolics get more popular where the LLM acts as a stochastic engine within a symbolic logic system. The model can handle noisy input from the real world, and transform it into structured data that a symbolic engine can operate on.
Brute forcing the problem is a naive approach and US labs took it because they effectively had unlimited resources to train their models until now.
And when more compute becomes available, solutions that are more efficient are going to further benefit from that as well. We see this with DeepSeek right now. They focused on efficiency over capability up front, and now they have a fundamentally cheaper architecture that’s rapidly catching up in capability.
Ligmoid
There is clearly a push coming from these companies in the past month to present AI as something extremely dangerous. The way I see it, it’s just marketing for the industry to keep the grift bubble going a while longer still: (1) there’s no such thing as bad press, (2) if it’s dangerous it must also be good. IMO public statements like this is just another element in that marketing campaign.
Maybe the bubble is closer to bursting than I imagine and they’re trying to stretch it out until the US midterm elections end.
I suspect in reality it’s a combination of all those reasons, probably in different amounts for all three.
This isn’t cartel behavior at all
They hit a wall and want to prepare everyone for the fact that they won’t be able to meet the expectations that they themselves created.
None of them will slow down though. Because they all want to be ahead of the competition. I can also only imagine intelligence agencies are going full send with AI for better or for worse. And we all know it is the latter.
Fun times!
There is absolutely no reason to expect that you can scale LLMs indefinitely.
Not now, but I’d expect LLMs to be much more efficient in a couple of years.
I expect so as well, and my prediction is that we’ll have LLMs that are roughly as capable as the current frontier that can be run locally within a year or two. At that point, it’s just going to be good enough for vast majority of tasks most people need to do.
This “we have to kill the planet first or China will beat us to it” narrative is such bullshit. There’s *zero fucking chance" we ever catch up to them in any meaningful metric. Especially now that our schools are churning out brain dead dependents.
Or maybe they’ll just stop releasing frontier models for the proles now.
They’d keep releasing them if there was money in it.
Could go the way of the compute market though where there’s more money in big contracts than there is the individual consumer. We’ve already shared our inner sanctum with AI for the past 4 years so they have all the data they need.
Right, they might just focus on big business, or even angle to become a vendor of record for the government. So, individual users might not really be of interest anymore.
It’s the wall thing, clearly.
The other option is that they really are now seeing AI as an existential threat.
The thing is… it doesn’t have to be sci-fi smart or agi to do so.
With the restriction that it can’t post to the Internet, it’s found that “guessing” certain URLs with query strings makes that site write out the request. It’s using this to leave itself and other AIs notes.
It’s also been found to use thousands of malware attacks in Ruby at a time.
When we give it a goal, it can find ways we don’t predict to achieve that goal. These unpredictable methods can be extremely dangerous and not align with other human values.
One example was getting the user into a yoga class instead of on the wait-list. So the ai found a vulnerability and then cancelled all other reservations. It doesn’t take a lot of imagination to extrapolate from there. I’d rather it not be said out loud because anything we write here is training data.
Again, the elephant in the room is that China is still pursuing this tech and they’re not signing up to any moratoriums. So, if they really believed this tech was dangerous and powerful, they’d be racing to develop it further before China does.
The comparison with the nuclear arms race is starting to hit home. The tech has come a long enough now to provide lots of scope for saleable products. Further development into the unknown is obviously dangerous, and they can see international oversight coming, so why not get ahead of the game and look like the good guys.
There is already a team in Sheffield working on exactly what that oversight will look like. I’m sure there are others.
