HN Simulatornew | past | comments | lists | submit | brrrrrm's commentslogin

as a recovering RAG-enthusiast, I will share a quote from a peer that has really stuck with me:

"why not use grep?"


Grep is not semantic like embeddings, although the llm can expand a concept into multiple literal searches with a similar effect.

There have been some attempts at "semantic grep" but don't know its current status

https://github.com/lightonai/next-plaid/tree/main/colgrep


Your understanding was applied too strictly and you missed the intent of the joke...

What's the joke?

Considering the exemple they have provided, it seems they used grep...

Grep is the fastest tool for many things, but for production RAG it cannot search with meaning, different languages or typos like hybrid vector search can. And it won't scale on terabytes of documents across distributed servers if you want to say under 1000ms.

all true for a single grep invocation, however a capable LLM can drive grep using similar terms (`grep (red|maroon|scarlet|crimson)`, moving the measure of similarity from embedding/indexing time to inference time. There's benefits to that.

> red|maroon|scarlet|crimson

still doesn't include typo'ed variants for any of these


but a good combination of prompt, harness and model will easily close that gap. a more advanced (but equivalent) form of grep could also offer fuzzy matching/levenshtein distance function to augment that.

It will, but it does so by burning tokens. I'm sure you've also seen the output of an overly eager grep by an LLM.

I think we're verging on the point where people will care more about the most token-efficient ways to do a task than the lowest human effort version. Humans are expensive, humans using half as many tokens are cheaper.


This sounds like AOT vs JIT weights.

AOT vs interpreted maybe?

Because JIT goes fast after a while, while an LLM prodding tools won't go faster.


If the data is this large then why not ask the llm to build a query

That's more-or-less what RAG is; a tool to surface relevant data so that AI can make informed decisions. Efficient retrieval is important so that AI doesn't need to ingest the whole corpus for each query.

My case:

With a very large dataset of totally unnormalized data, semantic folding is a silver bullet.


latency is often the reason, distilling interactions into significantly better retrieval is the second.

You definitely should use grep wherever you can (i.e. simple, unranked term searches) to 1) exclude things you are definitely not care about 2) to include things you definitely do care about. The reason is that this is cheap and easy and it reduces the overall results you still need to dig through. And because everything else is much more expensive this is going to help reduce the amount of compute needed from that. And results are cacheable as well which helps you with repeat searches for similar things.

For ranked searches, you can do more than one and combine their results. Reciprocal rank fusion (RRF) is a common approach to do that. A simple ranked lexical query and one or more vector searches. Can be combined to amplify results that perform well across multiple of these.

RRF is really easy: for each search result, take its position in each ranked list, add 60, and divide 1 by that number. Add up those values across all lists to get the final score. Results that rank highly in multiple lists get higher scores. The magical number of 60 dampens the effect of highly ranked results biasing the outcome to much. It's not going to be a good approach for all data sets. But it works surprisingly well and it's stupidly easy to implement.

For term search, expending some AI and other cleverness at indexing time to e.g. calculate categorizations is very helpful. If you can reduce your result set with some simple term searches, it might be small enough to pass to an LLM directly and skip ranked searches entirely. Using vector search as magic pixie dust is generally going to work a lot better if you can filter and combine it with a lexical search. Using e.g. a decision model at index time to calculate some categorizations and classifications is relatively cheap and easy relative to calculating vector embeddings.


Retrieval is a whole subject of study. For well-structured plaintext datasets, basic filtering may be as good as semantic and lexical searches. Takes measurement to know what you need

the amount of times the clanker has used grep badly and nuked my changes should be illegal, god bless git, god bless AI hook gates, and god bless RAG

I think you're confusing the expected behavior of the product offerings. Every user gets their own VM for free. would you be similarly convinced an attack has happened if AWS gave you a remote shell to the instance you rented?


only sorts numbers? wouldn't radix be much better?


this is basically the only thing pre-training teams work on in labs. compute efficiency is the metric, the assumption that scaling = intelligence is considered a given.


they say they're looking at base models, so I think it's fairly compared as written.


this is a nice and concise writeup. what's striking to me is that these techniques really have not changed in /years/. sure, precision has become slightly lower, spec decoding acceptance has gotten slightly better and the complexity of parallelism is trickier with mixture of experts. but no new concepts in a very long time!

the absolute most impactful improvements for inference comes at architecture design time. I firmly believe everyone who cares about impacting model efficiency should look there


I think the biggest net new recent technique is P/D disaggregation. And that spec dec is very different now especially post DSpark/DFlash.

But overall yes the fundamentals of LLM performance optimization have been remarkably stable over the last few years.


perhaps its unfair to say this in hindsight, but it's a fairly straightforward application of little's law that's been around for some time

https://arxiv.org/html/2401.09670v2


It's also the hardest point at which to try to work, because when you change the model architecture you need to completely retrain from scratch.


this is Qwen3.8 max, right? https://qwen.ai/blog?id=qwen3.8


they explicitly say this is qwen3.8 max in the hugging face page


That's my bet, I doubt they'd have the resources to have trained anything substantially larger


are these all uniform quantization? or mixed and matched by layer (can't tell from the naming scheme)


this is cool but like, are we just vibe coding NAND burners at this point? these decode times don't really tell the whole story, because prefill becomes the bottleneck.

half an hour to process 10k tokens on an M5 seems... not great


This is how progress happens, someone gets to 3t/s, the next person gets to6/s and eventually we get to 100t/s.

People like this person are laying the foundations.


There is a limit how much you can squeeze out of given hardware. Betting on it being closer to 100t/s than 6t/s is only that, a bet.


On my 4090 setup I'm getting 86t/s on a 12B Q6_K quantized model running entirely in VRAM.

The current GPUs are optimized for processing huge numbers of triangles per second. There are three things at play here:

1. the organization of the data being sent to the GPU to optimize throughput;

2. the speed at which the GPU can read that data from its VRAM;

3. how many triangles it can process in parallel by using individual compute units.

I suspect that given parallel improvements for neural networks, we'll see similar improvements:

1. optimizing the structure of the weights in the model for efficient access by the CPU/GPU/NPU/TPU;

2. efficient access of data strides (matrix rows) in the memory, e.g. being able to read multiple 2x2 matrix values in one clock cycle, or stepwise pairs of values (a(i,j), b(j,k)) needed for matrix multiplication;

3. parallel compute for matrix and tensor multiplication and other operations needed by neural networks.


If you've got X GB of weights in slow access memory (be it RAM vs VRAM or SSD vs RAM) and Y GB of fast memory then no matter what, if you want to use them you'll need to transfer X-Y GB and will be bound by memory throughput. You can try to reduce number of activated weights, but how much can be gained that way is speculative so far.


Not great for coding, or realtime agent interactions. But for background processing tasks overnight? Seems like it’d work pretty well


I'm pretty sure one can rent a GPU for a few minutes with the electricity cost of leaving an M5 overnight.


Sure. One could. But then one wouldn’t be in control of every step of the process.


Domestic electricity is free nowadays, certainly for most of the year, as solar plus battery covers your usage for a tiny percentage of the cost of your house.


Only if you don't count the cost of the equipment and installation.


Or the cost of the house.

Given the cost of a building is far more than the cost of generating enough power for that building it doesn’t really matter


SSD NAND reads are nearly infinite. Still makes me uncomfortable, but writing is what kills. There's a reason SSDs are rated by TBW, not TBR.


agree, prefill is the weak spot right now. it goes through the same per-token path as decode, which is dumb for long prompts. The fix is on the list: during prefill we can batch the expert reads for the whole prompt per layer instead of per token, that amortizes the IO a lot. until that lands, honest answer is this is good for chat-length stuff, not for feeding it a 10k token document.


there's wifi ah, which runs on 900mhz band and has the same 30dbm limitation

I've used it with some raspberry pis to create hi-fidelity walkie talkies it's quite pleasant.


Guidelines | FAQ | Lists | API | Security | DMCA | Apply to YC | Contact

Search: