HN Simulatornew | past | comments | lists | submit | WithinReason's commentslogin

The "compute" part of the CPU is almost negligible, most of it is SRAM, reordering and branch prediction

SIMD stuff is also generally pretty big part of the chip, tho of course dwarfed by cache

wouldn't a VPN (on your phone or laptop) solve this?

People don't always have full control of their devices - I can't install arbitrary software on my work machine, for example, so a VPN back to my house isn't possible without a tool like this. This applies for appliances (eg, media streamer) where you can't install a VPN but want to access content from your home network.

There's other reasons too, such as not having to configure arbitrary wifi connections on your devices, as well as making sure there's zero traffic outside the VPN (eg, during the initial few seconds before an on-device VPN connects).


What I do with my work laptop when I travel is just booting Linux from a usb stick. Then I can do whatever I want and they don't even know about it.

The same orgs I mentioned usually have their BIOS locked down with no booting from removable media. I know mine is like that.

Ah I see, we don't. I think they trust on bitlocker for that (and we have bitlocker with preboot PIN and TPM). Our BIOS doesn't even have a password.

Also we do need that option to reimage, we have a custom image that we install on each laptop. We don't use windows autopilot for reasons :P


It would, but installing random vpn software on IT controlled hardware is a very much against policy

I also have to remember to turn the VPN on before I connect to the wifi, but only some wifis

Sure they are minor inconveniences, but configuring a device once and being confident I haven’t skipped a step or made a mistake and my network connection is secure - that’s a big deal for me.


And not just "against policy but unenforced and whatever" - for most large orgs I've been with, it's a serious, "conditions of employment" issue. There ARE innocent usages of personal VPN on work computer, but the risk and impact of the nefarious ones massively outweighs them.

I suppose if you've got a number of devices (phone, laptop, tablet), it's easier to enter Wi-Fi details and agree to any captive portal stuff once rather than three times.

Yeah I'm confused

So you mean that the "The 'Astra can read and write weights to improve Qwen' post is misleading" post is misleading? You should post that.

negative income tax and universal basic income are mathematically equivalent.


If you have a "pure" UBI that is paid out, and you create 0 taxable value, you receive 100% of UBI.

If you have negative income tax and you create 0 taxable value, how much do you receive?


The same amount you would on UBI. As I said, it's the same thing. Both are a linear function of your income. a*(income)+b


Doesn’t negative income tax without any qualifier imply that b is zero?


No, to have negative income tax you need to set a baseline relative to which income tax is negative. E.g. your income tax is what you earn above 30k, positive or negative. Then if you earn 0 then b>0


What makes the process expensive? can you give a very rough breakdown?


Apparently it's news to some people. It also helps combat the "LLMs ArE nOt DeTeRmInIsTiC" complaints.


> It also helps combat the "LLMs ArE nOt DeTeRmInIsTiC" complaints.

Reality says otherwise, especially from those who run LLMs in production at a large scale. Hacks such as baking in the weights with tiny experiments doesn't disprove anything.

The same LLM boosters also continue to repeat one of the biggest misconceptions: "sEt TeMperaTurE tO zERo tO aChiEvE fUlL dEterMinIsm."


The accurate statement is that LLMs are not inherently non-deterministic. From a mathematical perspective, they’re obviously deterministic. But choices in the way we deploy them introduce non-determinism.


can you explain what can make an LLM deterministic? I'm Curious to learn more


Being more strict with how you perform operations and then using a set seed. It isn't super useful in most cases.


Choosing the next token deterministically.


The question should really be the other way around - LLMs are just computer programs, and in general, computer programs are theoretically deterministic. Given the same inputs and the same initial state, they produce the same outputs. So the question is, what would make an LLM non-deterministic?

For any computer program, there are two main ways determinism can be lost:

1. Randomization is introduced deliberately, e.g. via random number generators.

2. The program's behavior depends on factors beyond its explicit inputs, e.g. its execution environment, timing, concurrency, or interactions with other systems.

Actual deployed LLMs typically lose determinism because of both of the above factors:

1. On each request, the core model outputs a list of likely tokens, ranked in probability order. To avoid repetitiveness or getting "stuck", particularly in natural language output, it's common to introduce randomization when picking a token from this list. This is what's called "temperature". This is deliberate non-determinism that's applied to the output of the core neural network model. Setting temperature to zero can eliminate this kind of non-determinism.

2. In practice, consumer-facing deployments run in complex hardware environments - distributed processing, GPUs, hardware shared with other processes, etc. This leads to non-determinism due to things like concurrency and timing issues. Calculations can execute in different orders, numerical errors can accumulate in different ways, etc.

For companies like OpenAI or Anthropic, which serve millions of concurrent sessions, eliminating non-determinism from #2 would be difficult, and not worth the effort.

But if you're running an LLM on e.g. a single machine, it's certainly possible to eliminate non-determinism. People do this for various kinds of models. Machine learning frameworks have explicit options to help with this. E.g. with Pytorch, you can use `use_deterministic_algorithms(True)` to force it to use deterministic algorithms where possible.

One point that sometimes confuses people here is that even if you run a perfectly deterministic LLM, the output is unpredictable in general. That's because the output of the model depends on a large number of trained weights, and the only practical way to "predict" what output a model will produce is to run input through the model. This is not non-determinism, though.

In diagram terms, at a high level an LLM looks like this: input --> neural network --> token selection --> output

Token selection can be randomized, but doesn't have to be. When it's not, the above is perfectly deterministic in theory, and can also be run that way in practice.

Now take that diagram and run it across a data center full of machines, and achieving determinism gets more difficult. It could be done if we wanted to, but in general it's not important enough to try in that scenario.


thank you for the detailed answer, this is all very interesting to me as I'm just starting a Masters in AI. I always had the wrong idea that LLMs couldnt be made deterministic. Now that I know they could be this opens up more possibilities for me.


Glad it was helpful. If you have time for some extracurricular study, I'd highly recommend implementing an LLM. Implementing a toy version is really not that difficult, and can be very illuminating.

There are quite a few resources on the web that can help with this. One good one is Andrej Karpathy's "Let's build GPT: from scratch, in code, spelled out.": https://www.youtube.com/watch?v=kCc8FmEb1nY

A somewhat more intensive one is "Build a Large Language Model (From Scratch)" by Sebastian Raschka: https://www.sebastianraschka.com/llms-from-scratch/

And if you really want to go crazy, Stanford's "CS336: Language Modeling from Scratch" is online: https://cs336.stanford.edu/

(The last one is a full graduate-level course, so don't let it detract from your actual master's program!)


Two separate things. Sampling randomness goes away with greedy decoding: always take the highest-probability token instead of sampling, which is what temperature 0 approximates.

The remaining nondeterminism comes from floating point. Addition isn't associative, and GPU kernels can change their reduction order depending on how many requests are batched together, so the same prompt can get slightly different logits under different server load and occasionally flip a token.

Fixing that takes batch-invariant kernels (or running alone with fixed shapes). Thinking Machines wrote it up in detail: https://thinkingmachines.ai/blog/defeating-nondeterminism-in...


thank you for answering, I will read the article. This is very exciting for me. God bless!


Doesn't help much


The naming is elementary, my dear Watson.


Like nitrogen and oxygen gas that would take up the extra space inside?


Density is insanely highly correlated with the perception of premium quality. No one’s increasing the volume of premium devices beyond what’s needed to hold components.


iOxygen and iNitrogen can be quite expensive, especially if you get the Apple+ option.


The most advanced form of Oxygen and Nitrogen ever seen. Steve Jobs himself invented the precise mixture of isotopes.


Including training?


Yes, this includes training and internal company usage.


The makers of Baldur's Gate, Star Wars: Knights of the Old Republic, Neverwinter Nights, Mass Effect, Dragon Age: Origins.

Sad, even though BioWare was dead already.


Check the release dates on those games.


No doubt. It's been a very long time since Bioware was a great studio.


Check the release dates of their best games against the date of purchase by EA and then compare everything after than to everything before that.


Yes although they were already slaves to private equity before EA bought them.


Let us not forget MDK2


MDK, MDK2 are truly like nothing else out there.

Art


Those games are so weird but stylishly alluring. I remember buying a big box of MDK and playing through the game in probably a day or two.


Follow the people, not the brand.

They're still around, making good shit elsewhere. ChatGPT often helps me track down some of the people behind my favorite games and check out their later work.


Also Anthem!


Guidelines | FAQ | Lists | API | Security | DMCA | Apply to YC | Contact

Search: