> A new reversible algorithm
>
> Designing a quantum circuit involves two constraints that ordinary programming doesn’t have. The first is that a quantum circuit can’t make “if-then” decisions the way an ordinary program can. An ordinary program can check a value and then run only one of two pieces of code. But a quantum computer runs on a superposition of many inputs at once, some of which would run the first piece of code and some of which would run the second, so the circuit has to include the operations for both every time it runs.
If you squint, this looks like SIMD programming (for a quick introductory reference, see [1]) where there is no branching logic in the lowest level. Looking from far away, it seems like high-performance programming is converging from these two points, isn't it? One with the current classical hardware, the other with the new quantum one.
> Real progress happens by expanding the area of the envelope, not by coloring in the blanks.
Respectfully disagree. What one person may classify as expanding the envelope, another one with broader knowledge may classify as coloring in the blanks. Much knowledge in one field comes from cross-pollination from other fields.
Let me give you an example I just learned about. The database community spent decades trying (to some extent) to solve the implementation of materialized views to arbitrary complex SQL queries. Then, a few years ago, someone with experience in digital signal processing brought their expertise to the DB field and completed the puzzle. [1]
Someone with DB expertise but no signal processing expertise might say that is expanding the envelope, but another person with experience in both may say it is coloring in the blanks.
AI work now may seem like coloring in the blanks, but given it's vast expertise in multiple fields, my bet is that for most people it will look like pushing the envelope very soon.
That is a very interesting example but it serves very much to illustrate my point: this is latent knowledge, we effectively already have it but we just haven't put 2 and 2 together. Unfortunately AI excels at such problems because it oversees a much larger slice of our knowledge than most individual humans. I've always said that cross domain application of knowledge is the easiest way to make progress and I haven't really seen anything conflicting with that to date.
So while you start off with 'disagreement' in fact it appears we agree far more than we disagree. Yes, it is important that this work is done. But you clearly illustrate that the envelope did not in fact expand. But all the time scientists around the world do in fact expand the envelope of our knowledge and one of my fears is that by undercutting a lot of such knowledge work by putting this digital combine to work on it that we will lose the infrastructure to do the rest. We'll see a lot of progress in a very short time and then we'll find out that because we took away that class of problems that it has become almost impossible to keep the infrastructure, incentive structure and educational paths in place to be able to do the next round.
You see a similar thing happening in software development. All of a sudden it has become a lot harder to start as a career. And lots of people are not even going to try this direction if there is no first rung on the ladder.
At this point, it’s better to ponder a question: Are there fundamental reasons why a search trajectory during reinforcement learning can’t leave the envelope? And: what is the envelope, anyway? A metaphorical surface in a metaphorical space of ideas with no defined procedures to make it into an actual metric space?
Reinforcement learning can definitely (in principle) leave this envelope, but it depends crucially on whether the feedback generates new information from reality, rather than just repackaging what humans have already discovered in the form of RL environments and densified rewards. So the question is not about RL per se, but what is the information source, what is the bandwidth of information it supplies, and then only finally the effectiveness of RL in distilling information that into models.
> Unfortunately AI excels at such problems because it oversees a much larger slice of our knowledge than most individual humans. I've always said that cross domain application of knowledge is the easiest way to make progress and I haven't really seen anything conflicting with that to date.
What is unfortunate about this? I suspect we concur in not buying into the AGI/ASI/RSI/ hype, but this isn't this kind of breakthrough extremely valuable regardless?
Maybe it's not expanding the convex hull of what is possible, but the fact is that human progress is extremely jagged even within that convex hull -- for myriad reasons limitations related to funding, cross-domain expertise, number of scientists, etc.
----
If you're worried about what it takes for a novice setting out in the field, it might be useful to see how specific "mature" subfields have evolved in their training structures. Eg. I have a background in theoretical physics and it is common for grad students to do courses for a couple of more years and take a few more years to reach the same productivity compared to grad students in (say) CS/AI.
I wonder if nowadays we should be focusing less on scalar data structures and more on languages that facilitate vectorized/SIMD instructions. Languages like Vx lang, Mojo, and Futhark that work both in the CPU or the GPU, although each one works in a different level of abstraction and control.
GPU compilation is definitely something I intend Valen to support. I wrote about it back in 2022, [0] and I learned a lot of lessons on how to do it (and how not to do it!) from working on the Mojo compiler.
The biggest decision for Valen is what to lower to:
* Rust MIR, since rustc has CUDA now, [1] (perhaps other cards soon?)
* SPIR-V, like Zig does for its GPU compilation [2]
* MLIR
Rust MIR is looking pretty nice. Valen already has Rust interop by doing some rustc sorcery, [3] and it would be somewhat straightforward to switch from emitting LLVM to emitting Rust MIR. I just need to figure out if Rust MIR can support the optimizations I have planned for Valen.
In a perfect world, Rust would be able to lower its MIR to MLIR, since MLIR has so many backends. I recall there were some efforts to do that, unsure where that ended up.
I don’t want to hijack Evan’s post by posting a link, but if you’re interested I’ve implemented many of the ideas in that presentation in the language I’m working on. You can check out the latest link in my post history for a rundown of how it works and a demo.
I remember listening to Jane Street’s Ron Minsky on their podcast talking about this a few months ago, how TCP becomes the bottleneck in AI clusters. As an electrical engineer, I remember that circuit switching gave away to packet switching due to very sparse usage of the network when there are many actors going through it. It is not very efficient, but that wide variety of traffic makes it hard to optimize it since the flow patterns are too dynamic. A good analogy with car traffic is that downtown there are so many cars going to a large variety of places, that traffic lights – as inefficient as they are – are a solution that at least works good enough.
On the other hand, if the traffic follows a very predictable pattern, a custom implementation can be much more efficient. Specially nowadays machine learning can find much better solutions through reinforcement learning. And AI cluster data flow is much more predictable than what goes over the internet as a whole.
Google TPUs use optically routable fibers like circuit switching to be able to change the whole topology of the setup depending on training load/matrix dimensions.
I remember reading about their setup and it blew my mind. They have MEMS mirrors that can rotate to redirect one of thousands of fiber lines into any other fiber line by using other optics. That way you can take in like 1000 lines and optically connect it to any of the other 1000 lines.
Basically they use lens array (one lens per input fiber) to focus each specific beam on its assigned mirror and mirror directs the light to the chosen output that also each has a lens in front of the fiber going out.
> Most software that requires hiring and paying software engineers has low risk tolerance:
I think a few of the industries listed like defense and aviation have low risk tolerance. However, from my (somewhat brief) experience of working in two health techs for a couple of years, I strongly disagree that healthcare has low risk tolerance for tech. Granted, they make run-of-the-mill CRMs, but I was baffled at how tolerable it is to have egregious user experience that makes users waste multiple hours per month with clerical work that is very painful because the UIs are very slow and buggy.
"Risk" has nothing to with designing functional and elegant UIs, so I'm not sure why you would even make the comparison.
It means risk that the software stops working after an update. Which usually trades off iteration speed and best practices (i'm pretty sure the average startup has way better security practices by just delegating to google/aws than the average manufacturing software business) in exchange for a rigorous testing and rollout schedule.
So I'm also not sure that the article has a point at all, the human writing the code was never relevant to avoiding the "risk" in these industries in the first place.
There is a huge amount of pending work in health research that is nowadays ignored because smart people have better paying jobs available.
Once many of those jobs get automated, I bet there will be many more people working in health research, which hopefully should lead to better health outcomes for society as a whole.
So when our lives will become so much emptier because we are leaving all intellectual and physical work to machines, at least on the way there we can reallocate all our greatest minds to improve healthcare, so we can turn 200 in our paradise where we don’t need to work or think?
This is true, and it's strange that people don't think much about the fact that approximately zero % of people on planet earth are actually trying to cure diseases.
I have been trying to tell the folks at work the same thing. Ideally, using a language that goes in that direction. For frontend web development, for example, there is Elm, which is pure and functional.
If you squint, this looks like SIMD programming (for a quick introductory reference, see [1]) where there is no branching logic in the lowest level. Looking from far away, it seems like high-performance programming is converging from these two points, isn't it? One with the current classical hardware, the other with the new quantum one.
[1] https://mitchellh.com/writing/everyone-should-know-simd
reply