“The future ain’t what it used to be.”

-Yogi Berra

  • 10 Posts
  • 842 Comments
Joined 3 years ago
cake
Cake day: July 29th, 2023

help-circle







  • Energy density has nothing to do with this.

    No it absolutely does, and it matters because:

    It’s the cost of how much pollution refining the rare earths and making batteries produces vs the amount of pollution associated with construction of a building with pulleys that move weights up and down.

    If an entire building could be supplied with a few elevator shafts and some weights, because the energy density of the system is so high, it would be silly not to do so. But the energy density of these systems isn’t remotely close to that. Where as yes, a building absolutely can be built with batteries as a part of it to support its typical duty cycle.

    Gravity is just not a particular energy dense form of storage. Its not really debatable. And like you said, building buildings comes with all kinds of other forms of pollution. Not to mention, we could be building them to house people, not pulleys and weights.

    Its an idea that sounds good, but once you engage with it seriously, its limits become obvious. Pumped hydro will almost always make more sense. A big tank at the top and a big tank in the basement, and bam. Battery built. not to mention you’ve got a semi-permanent back up reservoir now built, which could help with flood control, drought tolerance, fire control, all kinds of other things. And you don’t need to build new buildings for this. They can go into/ on existing buildings.












  • “Attention is all you need” is the place to start that question, as this is where the transformer gets introduced, and it’s authored by the team at google brain. Notably, not OpenAi, who later authored a paper introducing gpt-1. Transformers were introduced as a way to shrink the model and support parallelization, not to make it larger.

    Convolutional networks, lstms, kernel based vision models, unet all of that had already existed before, and yes, people had just thrown more complexity at the matter myself included, but it never resulted in the kind of pay off that transformers seemed to have been able to achieve.

    So it’s not like the community hadn’t tried just throwing more compute or scale at the issues, it’s just that it didn’t result in this kind of emergent complexity that we’ve seen with transformers. And that’s true if networks in general. There is no guarantee that throwing “just more complexity” at the system is going to result in different properties. But there is also no guarantee it won’t. Practitioners of complex systems analysis understand this, that there are no guarantees regarding boundary conditions in complex systems.

    And this is the second leg of support for unexpected, because if it was to be expected, why not just go there? And we can see that with the relatively asymptomatic performance we see in frontier llms. They are still improving. But marginally compared to the massive jumps we saw, for example between gpt1 and gpt2, or gpt 2 and gpt 3. Even at gpt 3 and gpt 4 we saw that asymptote beginning to form. And since four, improvements have been very meh, inspite of just throwing more parameters at the problem. Things have been improving but it’s largely around the engineering around the models, not the models themselves, inspite of throwing more and more complexity at the problem.

    And maybe at one trillion parameters, there is some new boundary conditions which result in new emergent properties. But we don’t know that. So if we go there and find that out, it would be unexpected, at least in that we’ve been throwing exponential complexity at a problem to get sublinear performance improvements.

    Attention is all you need: https://arxiv.org/pdf/1706.03762

    Gpt-1: https://cdn.openai.com/research-covers/language-unsupervised/language_understanding_paper.pdf

    Complex systems: https://onlinelibrary.wiley.com/doi/10.1155/2020/6105872