Could you explain how the probabilities are computed for an online/essay excerpt? My understanding is that the probability for the next token is given by all previous token. Given that online the (system + user) prompts and the possible previous turns will almost certainly not be included how can the probabilities be accurate?

10000?

Wait until you meet pip and liberal requirements.txt

And then you encounter packages with `setup.py` that generates a random list of dependencies on each run. You can't know the dependencies without running code.