Flux.ai being the incumbent
Silixon based out of the UK have been showing demos
Quilter and DeepPCB for layout
The recent OpenAI demo...
I went the other way and pre-designed sub circuits with a set of rules that allows simple merge assembly, unfortunately I didn't have the time or budget to continue with it, but I took the constrained approach as with hardware, things can't be 99%.
The one thing I will say, is that my experience of hardware engineers is very gatekeepy.
Any questions about the use of AI for any form of automation on forums/Reddit get shot down instantly and the OP's told 'skill issue' more or less. In my opinion, just as with software, most work is simple and should be automated, there will always be a need for the real engineering when it comes to high specification work, high volume optimisation and hardware debug (although Claude code is pretty handy at driving a Jlink)
> The one thing I will say, is that my experience of hardware engineers is very gatekeepy.
Stay away from Reddit and forums if you want to see where the open minded people are looking. There are a lot of communities where EEs are experimenting with newer technologies and sharing their results, with realistic appraisals of what can be done.
You also need to understand the history of autorouting technology for PCBs. Autorouters with auto layout have been around in many forms for years. Their output never compares to a skilled EE doing it manually. The gap between autorouter output and skilled operator output is even bigger than the gap between a good developer and average AI codegen slop.
I’m interested in this space, but even the Fable/Astra or dedicated tools like Quilter, Flux, and others have a very, very long way to go for boards more complex than simple hobby boards.
I do think this is going to be good for hobby people doing simple PCBs though
> Stay away from Reddit and forums if you want to see where the open minded people are looking. There are a lot of communities where EEs are experimenting with newer technologies and sharing their results, with realistic appraisals of what can be done.
LLMs are not the same as auto routers that have existed for decades. They function completely differently and are capable of taking more context into account, all the way to fetching and reading data sheets and considering design compromises.
Sure, one is a conflated chaotic nonsense generator, and the other is a chaotic random walk local "fit" evaluator set by an now arbitrary cost function.
NP-hard is still NP-hard... and named problems are rarely solvable with naive approaches. Best of luck =3
We can find local optima, but there's no real way to prove a given design is the best of all possible designs. We can only find designs which work and meet the given criteria.
One can prove that a design is correct, but it requires the same class of compute as an LLM to properly simulate. Circuit boards are hellishly complicated once you start simulating EM and RF responses.
As for how a good engineer does this, it's the same way pilots can fly helicopters: intuition. It's a skill built up from lots of reading the rules and theory, and lots of practice. One gets a feel for how the various fields and energies are moving.
Which, when you think about it, is just about how neural nets learn, isn't that neat?
I think the real answer here is that machine learning is absolutely capable of producing satisfactorily correct circuit boards. In principle. But I think the problem space is far too large for today's ML techniques and verification/iteration is just way too expensive. Maybe in another few years, I just don't think we have enough compute for it yet.
Only decades of standards compliance, workmanship training, and experience. Human beings are very good at spotting patterns in noisy jungles, and making rational design tradeoffs.
Auto-routers have been around far longer than even smartphones. Somewhat functional in the trivial problem domains, and always useless where design choices mattered.
Computationally what’s the difference between a human and an LLM besides scale and speed? Trained humans are still using heuristics and shortcuts. Just because these are subconscious and only possessed by talented, trained professionals doesn’t make the skill computationally special.
LLM do not think because they are not real "AI", but it does copy the linear patterns people exhibit if statistically salient within the granularity of the higher dimensional vector search space proximity. Watermarking does skew the compaction slightly, but not far from resulting output patterns. =3
I do think that slightly smarter LLMs and a good (although not perfect) autorouting algorithm could solve 99% of hobbyist's projects and simple industry boards, due to the repetition of those patterns in real life.
The LLM would learn the heiristics (example: data lines first, power lines later, etc.) and would request the autorouter to do that routing, then take the image output and request a different part (depth-first). If later no routing is posible with that configuration, after some retries it could try another way.
It wouldn't solve complex boards, and engineers are always needed for short comings (and even if not, for research), but I wouldn't say this is something different than poetry, music or pixel art, LLMs can imitate although most of the times in a uncanny way.
Yes, the optimal solution won't be achievable, it will burn lots of compute and we need better simulations so that less errors are made (and will be made).
Still, if LLMs are capable of writing working code, art and solving math problems, they qre definitely capable of doing suboptimal routing (with some algorithmic aid)
That is the leap in logic, as an LLM doesn't think/create/feel or understand. It simply copies billions of patterns, and finds the closest fit.
> writing working code,
You mean a compacted collage of stolen codified work from real people. It will require a continuous parasitic relationship of user data to remain coherent. Given the firms have already stolen everything, it is unlikely to improve much as weights are refined.
> art
Again, a statistical output of salient feature clusters mimicking a humans output is not a creative process. For example, a glass-blower only sees the glowing art-form in its true state before it is annealed into its final form. Or a painter undergoes a creative process to find meaning, and abandons the work when there is nothing left to add.
Emotional projection onto an algorithmic output is simply recognizing the millions of peoples aesthetic choices stolen, and compacted with some degree of lost granularity. It is peoples art forms, but no creative thought or intent behind the results.
> solving math problems,
LLM are very good at context search, but require persistent data streams to mine for weighting relevant vector proximity. Again, everything that could be stolen, has already been scraped.
Only user input data remains in a sea of slop, and the process is degenerative given most human chat-users appear to be losing 17% cognitive function. As people discover the intelligence campaign against users is not in their interest.. paying companies to rob you and your friends makes less sense.
Being a sentient turnip, I am probably unaware of such things. ymmv =3
Indeed, LLM also stole that data as well, and encoded it based on frequency of occurrence and similarity with other previously compacted fowl related vector weights. Yet still has no idea what any bird actually looks like. bye =3
Finding a consistent pattern by conflating most papers, resolving a similar isomorphic vector search proximity result from an entire field of research.
LLM are very good at brute force context search spaces, and people see patterns in nonsense even when it is nebulous. Similar to the Newton's Apple story people are fond of telling each other. =3
> Somewhat functional in the trivial problem domains, and always useless where design choices mattered.
This is laughable when you consider that semiconductor design is the least "trivial" problem domain, and it happens entirely in RTL. There's basically zero human intervention when it comes to layout: even floorplanning is being automated nowadays.
Fab standard cell libraries are already the practical optimization of their current capabilities, and even FPGA try to minimize routing proximity with various levels of success. However, there are again named problems that constrain what naive people can get away with....
Procedural generation or parametric design is not the same, as the behavior and constraints were rationally engineered by people.
LLM would simply look at the outputs during training, abstract it as looking similar to a potato-chip factory, and generate a nonsense answer some fool assumes is appropriate.
Humans are wired to see meaningful patterns in chaotic systems even when they aren't really there. Neuromorphic computing may create real "AI" someday, but it almost certainly won't be from LLM cults. =3
If anything, it's closer to the bin packing problem. The cost function isn't expressed in terms of lowest cost between nodes, but instead fitting all of the traces with the lowest board layer count while still meeting stuff like EMI crosstalk guidelines. Then there'll be some traces that are very constrained (think DRAM or SERDES links) that have to be length matched, but those are pretty much invariably point to point links that already are heavily automated.
Traces being, say 20% longer than they need to be for an optimal solution doesn't really change the performance of the board in the vast majority of cases since they'll be some of the lowest resistance components of the netlist anyway, as well as the fact that the PCB is normally oversized for the number of traces needed as it fulfills structural/mechanical needs as well as the netlist needs.
And on top of that, because the optimal solution probably is NP-hard, once again, the industry doesn't look for optimal, only good enough. If layout techs in Altium were regularly solving NP-hard problems, computer science would be a different place. The only goal I see is to meet that bar without hiring layout techs.
>Traces being, say 20% longer than they need to be doesn't really change the performance of the board
Impedance matching requires you know what you are doing, and Altium will only take you so far. For the price, Altium/Protel has always been a poor deal for what you get software wise.
Even free Qspice (a more modern Analog Devices LTSpice) is a far better option:
> Impedance matching requires you know what you are doing, and Altium will only take you so far. For the price, Altium/Protel has always been a poor deal for what you get software wise.
Pretty much every high end board is designed under Altium. QSpice is a different tool for a different purpose.
And like I said, those few cases were the length actually matters tend to be heavily assisted with automation as it is.
> Indeed, that is why your iPhone still weighs 3.7 kg. =3
I guarantee you that an iPhone board isn't an optimal solution, simply good enough.
>I guarantee you that an iPhone board isn't an optimal solution,
DFM almost certainly guarantees an iPhone is both form, and cost optimized. Jobs was very clear early on in the product development that the EE work had to be miniaturized due to the energy density possible setting volumetric minimums on the battery technology at that time.
> simply good enough.
A common philosophy for people that make cost optimized low-end products. Some people love their 3.7kg phone, as it comes with a stylish shoulder strap. =3
Magic smoke is important, as it indicates a design parameter was exceeded.
I often recommend folks get a mini thermal camera for their phone... as inspecting for issues in electrical, hydraulic, or mechanical systems gets a fair bit easier/safer. =3
My experience with hardware engineers, at least IRL, is very different. They love to talk about their work. However, I used to be an EE so that probably helps.
As for Reddit, there's a lot of "bury head in sand and problem will go away" behavior when it comes to AI. People are understandably worried about their jobs, but they're certainly not doing themselves any favors by downplaying AI or looking for "tech niches where AI isn't yet."
Copperhead is very interesting. I do some embedded systems consulting on the side, but I've never enjoyed PCB layout very much. I briefly investigated using AI for it the last time I had to do a board, but it wasn't very helpful.
Thing is, what can they realistically do? Any work that does not involve a social component is on the way out. All work that remains will be more like "service" than like "work" the way we see it today ("service" as in "civil service", not as in "held in service or labor", i hope).
I 100% agree the model is basically useless. that was never the deliverable. the artifact here is the inference engine, not the model living in it. 3.16M params at character level is just what fits in ~3MB of on-chip SRAM.
Plus I treated it as a good learning experience to get better with FPGA's but also system design.
Appreciated. and yeah, agreed the interesting comparison isn't "is this scalable as-is" (it isn't, PL fabric, cost, power), it's that the CIM / analog-compute startups you mention are chasing exactly this endpoint with real money and years of R&D, the bar for making something useful is brutally high.
However, if no-one made anything that was useless on the same thesis, a lot of these concepts would have never got off the ground. I would hazard a guess that people like taalas would have started with a (much much bigger) fpga to validate whether the approach was possible before committing to designing a chip big enough to fit an 8B model in it.
I just nerd sniped myself...
VP1902 could fit around a 500m model in, whereas a cadence protium rack of them could squeeze in a ~6B at 8bit, or a ~13B at 4bit. So accounting for the headroom of distributed compute, Llama 3.1 8B at 4bit. I don't want to even estimate how long synthesis and place and route would take on that!
I started this about 10 weeks ago when the Taalas chatjimmy demo first did the rounds, Llama 8B baked into custom silicon, 17k tok/s for a single user. Their whole thesis is that inference is bound by reading the weights, so stop fetching them from far away. I wanted to see how far that idea stretches on a 'consumer hardware': every weight resident in SRAM, zero DRAM My chip only gives you ~3 MB to live in, so the model is 3.16M parameters and the tokens are characters, but the physics is the same bet. Only difference is their model is actually good!
The annoying part: I wanted this live last week, which would have landed the same day as the AMD acquisition news, on AMD's own silicon no less. Life got in the way, so here it is a week late instead.
You're correct, the flat line is aggregate only. the fabric is saturated from a few dozen active clients onward, so extra connections can't buy throughput, they just queue. per-user p50/p95 across that same sweep: 17ms/30ms solo, 450ms/545ms at 100, ~2s/2.4s at 500, 3.8s/4.4s at 1000, 6.3s/9.4s at 2000. zero errors or drops at every stage. It degrades as a well-behaved queue, not a cliff, but nobody would call 6s at the top end snappy.
The benchmark sweep is 2,000 concurrent active requesters hammering it constantly, real traffic is mostly lurkers, which cost a file descriptor and nothing else. the interactive feel actually gives out earlier than the queue math. The speculative-typing UI wants sub-second replies, and that budget blows around 100–150 simultaneous typists.
I've been logging the stats since it went live, unfortunately it didn't hit FP. Peak was 10 concurrent connections (13 uniques in the busiest half hour), ~580 requests and ~37k tokens served, and at no point did two people actually have an inference in flight at the same moment which would have been the real test for the queue, every visitor got the fabric to themselves, p50 ~23ms. so the 2,000-conn drill was not stressed today. the one blemish: a single window with p95 ~57s, which lines up with the model-rotation FPGA reconfigure rather than load. A request that arrives mid-reflash waits out the ~25s swap. if this thread sends 50× more people, the queue math above says it holds.
I need to discard the requests that overlap the model changeover for a truer result.
does the reflash actually stall every live connection, or just the ones whose request lands during that window? if the whole board goes dark for the full ~25s while any request is queued behind it, you could probably hide most of that behind partial reconfiguration, reflashing only the region holding the model weights while the sequencer and I/O logic on the rest of the fabric stay live and keep draining the queue. that's obviously a much bigger lift than what you've built here, but it would turn a hard stop into something closer to a brief latency bump for whoever's unlucky enough to hit it, rather than a shared 25s wall for everyone behind them in line.
I just posted a Show HN of my most recent side project, a live demo of a tiny-llm implemented in FPGA fabric, hitting an aggregate peak of 60,000tok/s, but a 'usable' model at 21,000tok/s
AMD could have saved their money and used their own hardware! I've got a language model doing 60k tok/s on AMD hardware already, a Xilinx Kria K26 SOM, with the weights baked into URAM/BRAM with zero DRAM in the token loop. Same thesis as Taalas: single-stream decode is bandwidth bound, so stop fetching weights from far away.
Caveats stacked high, obviously. It's 3.16M parameters (tinystories, and I also have a kevin-speak lemmatised version), the tokens are characters, and the 60k record is 16 streams that each remember exactly one token of context, so it's blisteringly fast at saying nothing. The honest build with full context and KV caching still does ~19k tok/s on one stream though.
I keep messing with the blogpost with the live demo, but I'm planning on flipping it to live in the next day or two
Yeah Im surprised nobody is talking about this. When everyone first saw Taalas I looked at the design and it had a big legup in physical cache availale compared to most chips. Makes you wonder how much of a benefit there is to the actual "baking" of the model vs just having a large chip with a ton of SRAM (or whatever) soldered close to the edge physically.
I feel like what we really need is the ability to solder computer cache on all sides of the chip Meaning above and below as well. If you can only attach it to the edges you will be inherently physically limited on the amount you can put (and maybe even have latency benefits as well)
Talaas is different, it's a true compute-in-memory architecture where the weights are stored in the connections between the transistors that perform the matrix multiply, rather than in seperate memory cells.
Most of the benefit comes from this architecture; hardwiring the weights into the silicon is just the easiest way to implement it. SRAM requires too many transistors, DRAM requires an incompatible manufacturing process, and exotic phase-change memories aren't readily available.
They can't due to power density, I believe - they have to be run in a sandwiched waterblock with massive cooling, as far as I can tell. That's the biggest thing that baked weights gets you - a relatively modest watts-per-square-mm compare to cerebras, where they had to engineer a whole system to get the watts out of the chip
Do you think there's room for reducing power requirements? Obviously shrinking the process is a win, but is the existing implementation a "just make it work" phase that has opportunities to increase computational efficiency?
With that kind of speed and if even lower power requirements, they could release mini compute units with USB4/Thunderbolt for plug and play inference.
I mean I think I read their spec sheet that it was 2.5kw on the asic but closer to 25kw with GPUs, for the same number of tokens (different latencies of course), but there's no way that a dedicated from-scratch matmul+memory chip can be less efficient than a GPU.
1. How come you didn't make your implementation public? You could be a millionaire now.
2. Especially if AMD has the technology to do what Taalas does, it makes a ton of sense for AMD to acquire Taalas: remove them from the market. Make sure nobody else (Intel, Huawei, Alibaba, NVIDIA, etc) acquires them. It could have been a great acquisition for a rebirth of BlackBerry btw.
It will be, I'm just not happy with my blog post before making it live. The blog will have a live interactive chat and a link to the repo with the HDL.
I don't think anything I did was particularly novel, as I really just wanted to see how fast I could push a commodity FPGA to it's limit.
Scaling to an ASIC or getting into the billions of params is where the real engineering is! This was just a side project for a side project for me while the FPGA was idle
I'm using the AMD (Xilinx) K26. It's a Zynq Ultrascale+, the successor to the old classic 7000's. I'm running it on the KV260 dev board, because I'm using it for another side project.
The K26 has a quad core A53 core alongside the programmable logic (PL, or Fabric). The A53 is pretty weak, and doesn't have any hardware matmul operations, so despite the KV260 being sold as a 'vision ai starter kit' and the vitis object detection running on the arm cores, they're pretty weak cores for anything AI.
For my use, I need true determinism, so my vision pipeline is all implemented in the PL, and it was pretty disapointing that the vitis libraries are basically just opencv on linux, rather than really pushing the fabric. If I wanted probabalistic AI running on a CPU, then I sure as heck wouldn't choose a quad core A53.
Which led me to have a play with this, I saw the taalas/chatjimmy demo and wondered what I could push the fabric to.
The round trip time to DDR or CPU via AXI meant I had to keep the entire inference engine in fabric. The A53 is simply a pipe that gets a request from my server (which has a cloudflare tunnel to the real world for the live demo in the blog post) and manages a queue. So it feeds a string in, and gets a hopefully longer string back a few uS later.
It's all in verilog, because that's what i'm more used to. I did get Claude Code to do a moderate amount, as it's a side project on a side project after all, but pushing an FPGA to it's limit is definitely not as comfortable for it as it is writing a crud app in TS.
I'm using tinystories, as we are talking about megabytes of URAM/BRAM. If I used the DDR, it definitely would have been a real model, but that wasn't my goal. My goal was to hit 100,000tok/s, and even when I conceded on absolutely everything, with a token prediction size of 1tok, I topped out at 60,000tok/s. ?But increasing the window to make an actually plausible chat (story generator, it doesn't understand questions, you need to prompt it with 'once upon a time...' and it finishes it for example) I managed to break 20k tok/s.
I also created the lemmatised version, which was inspired by Kevin from the office (why use many word when few do trick) and trained a new model, I was expecting the output model to be smaller, but was suprised that it came out the same size, but it ran 30%ish faster. In hindsight it makes sense, the parameter count is fixed by the architecture, not the corpus, so training on compressed text doesn't shrink the model at all. What it does is compress the output distribution. The same story takes ~30% fewer characters to tell, so the effective speed goes up even though the per-token rate is identical. The dumbness is the optimisation.
Fully agree on publish early. The blog post with the live demo (a websocket straight to the board through a cloudflare tunnel, so you're genuinely talking to the fabric) is written and sitting in drafts while I fiddle with it. This thread is my peer pressure, it goes live in the next day or two.
Flux.ai being the incumbent Silixon based out of the UK have been showing demos Quilter and DeepPCB for layout The recent OpenAI demo...
I went the other way and pre-designed sub circuits with a set of rules that allows simple merge assembly, unfortunately I didn't have the time or budget to continue with it, but I took the constrained approach as with hardware, things can't be 99%.
The one thing I will say, is that my experience of hardware engineers is very gatekeepy.
Any questions about the use of AI for any form of automation on forums/Reddit get shot down instantly and the OP's told 'skill issue' more or less. In my opinion, just as with software, most work is simple and should be automated, there will always be a need for the real engineering when it comes to high specification work, high volume optimisation and hardware debug (although Claude code is pretty handy at driving a Jlink)