Did I parse the jsonl files? No. I just asked Claude to create a skill to read all previous conversations across all projects or anywhere else in Claude, Cursor, Github Copilot and Windsurf. For the last one it failed (they are encrypted). The rest it just picked up whatever location and format they are stored in. So yes the skill is parsing the jsonl files in projects/. Only exception are Claude Desktop sessions that are only stored in the cloud.
The skill itself is also the basis for my memory across sessions and projects. With that Claude can recall what I can remember. It's using a subagent so only the important bits land in context.
From there it's easy to get some benchmarks. Let it find common themes across your sessions and let it mock them to create benchmarks strong enough so not everything gets the same score. My benchmarks are now based on me designing new tools, debugging live issues going through log files, finding clues in bigger code bases etc. pp.
But no they are not open. First of all they depend on my use cases (but I also got one colleague to contribute cases) and second of all they are based on the work at my company, so I can't share them.
Perhaps, but I imagine that it would be difficult for the receptionist to spec such tools sufficiently such that they would want to use the tool. Also, how would this tool be distributed or accessed? Most people don't have package managers or even desktop environments that they can meaningfully modify. To use your example, the receptionist is likely running windows on a thin client that is maintained by an msp, so how would they install and use the tool?
People talk about AI solving coding, sure, to a large extent. The problems of design and ops completely confound the layperson. Just last year, a cousin mentioned they were using like lovable or something to build a webpage for their business idea. I offered to just throw it together for them, very simple crud. They gave me what they had on lovable, and they got no further than the landing page before they had no idea what to ask for, let alone how to ask for it. Prompts were literally "ok so now make the app" and the models like "well what about x y z"? You have to be technically inclined at the very least to get something remotely useful out of these things.
Edit to say that the best way such a person might use llms to make work easier is through whatever client they are provided, probably copilot, to do toil like data entry, scheduling, etc. but those lock you into that ecosystem and interface, and the solution:problem ratio is close to 1; one cannot extend or modularize such solutions because the concept isn't even in such a person's head. Perhaps this will be solved in time, but we are not there yet, and we're not close.
Deployment is only an issue because it's assumed to be unnecessary in a current environment. As that assumption changes, so will the default setups. Of course that introduces new vulnerabilities, but as LLMs also make a mess of everything security related (and future LLMs will likely make a mess current era 'LLM hardened' setups, onward) our approach towards security and the internet will change in ways that are difficult to predict today.
I agree that plenty of people are currently hesitant to use LLMs for technical purposes, but I don't think it's because they literally could not do it. LLMs will now code up entire workable apps from minimal/vague/contradictory descriptions. If somebody only got to a landing page it's because they only prompted for a landing page, possibly trying to mimic how a human developer might have previously built a site - and quickly getting lost. But if you just say 'make a website where people can order widgets, manage their orders, and do all related tasks' it'll get done, no technical knowledge needed.
So I think we're mostly in a transitory phase. As people realize how powerful these tools have become, adoption is simply unavoidable.
By far the skill I use the most is one I wrote called "resolve-review" from back in the 4.6 days. Just uses gh CLI and graphql to get inline comments, which I key with standard tags like nit/fixit. So during review, I open vscode so I can have an lsp and jump around refs more easily (only reason I still have vscode installed; frozen at v1.110.1 with ai completely disabled and I haven't got around to getting the same setup in neovim), leave comments, send review, then resolve review. I'm sure one could have hooks or scheduled tasks listening for this but it's good enough for me. Working on a thing to build a pipeline of such work, which I describe in a top-level comment.
I do not have any pressure to deliver anything (amateur), so I can only imagine how much it must suck for anyone on that hamster wheel given how much it bothers me. If anyone wants to work on something like this, drop me a line; I miss working with people.
The problem the author describes is real. Expecting humans to consume and reason about pages and pages of technically dense, oddly packed prose with pointers to many external files (tickets, prs, plans, code) is not feasible. Perhaps models can hold that context, but I suspect it is incredibly inefficient. No idea why markdown became the lingua franca for the text stream interface outside of the llm tool calls--one must enable flags to get e.g. json streams. What's worse, models treat such markdown content as ground truth, almost at the level of a system prompt, especially when they receive it from, say, a call to the GitHub mcp to read some issues.
I have been developing a workflow that is strikingly similar to what the author describes. Github's new pr UI is solving the wrong problem. Pushing up buttons in the dom is not going to solve the root cause, which is context overload, incorrect global state (MEMORY.md, stale/superseded decisions). I think we just need a decent openapi thing that can be parsed, tagged, and validated in manageable chunks.
For me, just using GitHub issue and pr metadata, labels for typing, blocking/blocked-by as directional edges, then building a graph. Great for giving cheap context, at least for smaller projects. Goal is to then build a queue of issues to close, then stack the prs accordingly. Problem I've hit in dogfooding it is merge conflicts and restacking, but that can be automated to some extent. Not as easy when a design decision made during the initial approach is found to be insufficient or invalid partway through the stack, though.
The mailbox approach is exactly what I landed on as a first pass, but I'm sure one could develop much more sophisticated channels and a decent interface to work with them. I am waaay too obsessive to push one out, but looks like others are picking up on it, and my experience has been that these solutions usually become standard within months anyway. Just something I've been hacking on.
Can confirm, not arrested but threatened with litigation/employer contacted for responsible disclosure of unsanitized user input allowing arbitrary access to protected data of presumably any user. Just a url query param value flipped from true to false. Incrementing int user id too, so enumeration is trivial and the response payload was json so no need to do anything.
It is amazing what a casual observer with a background in web dev will see if they just open devtools. It has instilled a deep aversion in me to give any service any of my data that I care about protecting.
There should be an anonymous org way to report these without putting yourself in personal jeopardy, some kind of "innocent observer" protections built in...
If your system has gaps im not at fault for discovering them sort of deal
If not, did you parse the .jsonl files in the projects/ directory?
reply