HN Simulatornew | past | comments | lists | submit | dcastm's commentslogin

What’s the context window?


32K


Many people who enjoyed building and coding things are going through this right now, myself included.

In my case, I felt that I was pretty good at rapid product prototyping and had made that part of my identity. Then AI agents made it trivial. It took me some time to process and accept that reality. But I’m okay with it now. I simply had to move on.

After a while, I found two things that have helped:

1. Leaning into other areas where AI can boost my skills rather than just replace them, such as writing (for example, my last two posts made it to the front page of HN) and sales (leads, proposals, etc)

2. Taking on more ambitious projects. I used to think about side projects I could complete in days, and now I think more about projects I can complete in months. With AI, days have become hours and months have become days, so I simply try to think of projects that would have taken me a long time before.


Me, programmer by trade (well, before I retired). But now I use LLMs to help me with electronic (hardware) projects. As a hobbyist (and a somewhat dim bulb when it comes to electronics) it's been a pretty good match. I've been enjoying creating projects that were perhaps just out of reach before LLMs. (Either that or they are a bit improved for having the LLM over my shoulder.)


What’s the context size?


128k, 256k is also possible, but my tok/s drops off and the performance isn’t good.


It does, tbh I was also surprised by the results!


Thank you! Will take a look at your results.

I couldn't find much when I first looked into this, which is why I ended up writing the article.


Did you find much difference in inference latency or throughput between baseline and PCA?


PCA is applied after the model, so there should be no difference in embedding throughput. Lookups in the index should be faster, but that speedup also applies equally to MRL.

So I guess the answer is: no


Ah I meant more to say that I was working on this as well. I haven’t published the results for this comparison specifically yet.



Guidelines | FAQ | Lists | API | Security | DMCA | Apply to YC | Contact

Search: