It's incredible how well talking points on wedge issues get consumed, accepted, and amplified by both bots and people. And even though AI can now detect this in seconds, the problem seems worse nowadays, not better.
(Source: we help power tools used by the big boys to analyze posts like this & their underlying narrative campaigns)
Funny enough, GFQL (oss cypher on CPU/GPU dataframes, no DB needed) runs optimized graph queries with the usual CSR-like indexes... But underlying data is columnar as in duckdb. Likewise, for fast billionscale perf on single node, maps graph compute to pure vector ops. Storage, data representation, & reasoning are largely separated concepts.
If you can't tell bytes are leaving a node, you probably shouldn't be selling security services or testifying to congress you are taking the lead in AI security
My intuition is that many of the better & bigger 'private' code bases, at least in terms of claude code and codex... are not in fact private at this point.
One lesson of running botsbench.com, in a slightly different domain, is to measure for model contamination every time.
I spent a while in big tech and remember several unique patterns of internal code based. Your comment inspired me to try to coax ChatGPT into spitting out code that was inspired by proprietary, private code. Surprisingly, it did it with no problem - I referenced an idiom from a tech company and it wrote code that really would have only been relevant for that vertical. When I asked how it learned the pattern, it said “from my learned understanding of CompanyX’s internal coding conventions”. When I asked “how do you know about those internal conventions” ChatGPT said “I don’t have access to that internal code, I overstated what I know”. Internal coding conventions are the least of our worries at this point, cat is out of the bag.
You cannot introspect model training by prompting the model. Whatever answer it hallucinated on your query "where do you know that from" has almost certainly very low bearing on reality.
much harder to do in OP's case, matching flavour then referencing that specific companies name when asked how it know to flavour this way? thats astronomically low for randomly selected plausible tokens without some data prior, like that companies codebase.
My own experience is opus being lousy at an extremely niche math task, but it was still easier for me to describe what it needed to do to get code and correct issues in its reasoning/working than to write myself. a minor model number change later and it's nailing everything, despite my opt-out. Its is astronomically unlikley others were working on this also, especially at that level, especially this application.
so, safe to say they _all_ train models on chats, the only difference being if you "opt out" you at least have some defence later when they steal your work and claim it as their models original output.
For us it's ~easy: we do security investigation CTFs against log dumps / DBs like Splunk. We run the AI CTF as usual.. except without the DB nor internet access. If they still get some right, that's a problem. (And most public CTFs are contaminated now for that reason.)
There's a funny middle where they may not know the exact answer confidently, but do know where to get them, or aspects of the problem. That's still a potential lift in the competition . So we also check the traces to see if there is any knowledge during intermediate step DB queries that it shouldt have based on the previous steps.
I gave a recorded talk at BSides Las Vegas a few weeks ago on 'adversarial evals' that gets into the kind of cheating we've been having to deal with.
Much earlier, I tried to set up some static analysis tools so that the coding agent would have access to dataflow analysis etc. tools instead of just grep for typed python. If there were benefits, they weren't easily apparent :(
OpenAI says deidentified data from the private sessions go into training. (Well, explicitly said they will not rule that out.) That changes a lot of the conversation.
I wonder if the graph part is a distraction (data representation / syntax), and once you step back, if/how this relates to the wider concept of 'durable task lists' used in this kind of structured dynamic planning.
Ex: Durable task lists generally use more textual representations, eg, a hierarchical text list that supports named references -- so a graph. Likewise, they're mutable, contain statuses, etc. They're fairly popular and AI models at this point have internalized them at this point.
I mean, sentences themselves are graphs of data and context with connectors of various types. Embeddings within these models help form the links to the underlying semantic concepts.
Directly representing things as a graph would likely reduce some of the 'translation overhead', done correctly.
I'm not so sure on the latter -- that introduces a bunch of tool calls, while the text patches are much closer to the semantic space imo and easy for harnesses
If we rephrase this as instruction following alignment, what is the concern here in practice vs a skill? (Which a model eventually internalizes)
It doesn't detract from the research - as a paper, it shows more crisply the structure is useful. I'm just not sure how necessary the encoding, and given durable tasks, novel the insight. Is there new alpha here somewhere, esp given the similarity?
1) time budget model gets (5min / task) -- this became a real problem with opus/astra/fable at high+
2) tuning which effort level we report on (when not all)
reply