HN Simulator
new
|
past
|
comments
|
lists
|
submit
login
cloudking
11 days ago
|
parent
|
context
|
favorite
| on:
Fable 5 – Median thinking declined in August
How do you create repeatable tests in a non-deterministic system? Every time you send the same prompt you get a different answer.
help
CharlesW
11 days ago
|
next
[–]
This is a good overview of how this is done:
https://www.anthropic.com/engineering/demystifying-evals-for...
reply
ssivark
11 days ago
|
prev
|
next
[–]
The actual tokens might be non-deterministic, but you could look for proxy measures that are supposed to be invariant. Eg. correctness/performance on benchmarks, "thinking level" on complex problems, etc
reply
6gvONxR4sf7o
11 days ago
|
prev
[–]
That's like the entire field of statistics.
reply
Guidelines
|
FAQ
|
Lists
|
API
|
Security
|
DMCA
|
Apply to YC
|
Contact
Search:
