GPUs, wifi, power/thermal management, firmware. Especially in portable computing there is still a huge amount of hardware support surface area to contend with.
Game consoles worked a lot like this until around the PS3 era, where the disc/cartridge the game came on was shipping binaries for everything including all the hardware support.
This led to some hilarious implementation shenanigans for the Wii as a transitional console, where the Home button pause screen is not in fact a task switch to some underlying console OS but rather a piece of the SDK that is separately-delivered from each individual game.
Obviously they tried it multiple ways have brought receipts, but nonetheless it seems surprising that it wouldn't be of benefit to bring as much of that kind of metadata to the model as possible. You'd think depth and segmentation in particular would basically just be a straight shortcut without which the model spends its own time and effort re-deriving that stuff.
I'd also be interested in how post-processing fits in with this. Like if you've got weather effects, film grain, tone mapping, etc, I would have thought the model would do better working on the image before those processes.
Hmm well the depth buffer only has accurate depth for opaque objects (and even that's not really true). Things like hair wouldn't be in there. Its not ground truth depth.
It is data, It doesn't have to be ground truth Depth. In fact the difference between what optically appears to be at one depth and the depth map depth is in-itself information. You have described a mechanism that the model can use to identify that what it is supposed to be enhancing is more likely to be hair.
I guess in this example since the model wasn’t trained on that, just like they said.
If that data is some kind of missing key, go make your own and stop nitpicking someone who made a great point as to why they didn’t want that oh so precious data
I think it has more to do with what kind of data they have access to at runtime - IIRC DLSS upscaling has only required the previous frames and motion vectors, so requiring depth buffers would mean it was no longer a "drop-in" replacement
> I'd also be interested in how post-processing fits in with this.
I think screenspace effects like film grain and tonemapping are excluded in the same way UI elements are rendered separately from the game.
This is what I've been working toward as well. It's interesting how having the agent do its own reasoning about what it thinks is the most relevant knowledge to carry forward into the next pieces of work is vastly more effective (and even fast sometimes) than whatever the mystery-meat "compaction" process is.
Mentioned by the author in a recent HN thread, I'm also experimenting with automating this through a tiny issue tracker called epiq [1] that basically lets the agent sessions themselves file tickets with the follow-on tasks and relevant handoff right in them, and then a dispatcher automatically launches those tickets into new agent sessions.
The idea to kill "mystery-meat compaction" and use an external handoff primitive is brilliant, but doesn't letting the agent author its own handoff tickets re-introduces the same failure mode?
You would think so, right? But as with others in the thread, I'd found directing the agent itself to prepare the handoff does deliver much better continuity.
I assume Anthropic & friends have noticed this as well and will change how they handle long running sessions, so the gap will likely close over time, but this is definitely where things stand today.
canada.ca is the Canadian government's equivalent, as far as a web 1.0-style portal/index into many other sites and resources.
Neither the UK or CA versions have "here's an LLM you can make general inquiries to" though; at least for now that appears to be unique to america.gov. Unclear to me why this required a separate domain with all this branding on it vs just being one of those "let our chatbot help you out!!!" popup widgets on the existing usa.gov.
I've recently gotten religion on the workflow that is many (relatively) short-lived agent sessions passing planning/handoff docs between themselves. It's better for my own task tracking, better for handling "oh btw I noticed XXXX", and better as a clear review point. Overall it just feels like it takes a lot of the formerly implicit context that was whatever we happened to have talked about and turns it into a much more explicit "this is what you need to know, now go".
Currently looking for a framework for managing this in a more formal way, and I think it's probably beads, but interested to hear from others.
Spent the past 1,5 years building a tool that might be relevant, helping keep durable task state between agent sessions. It is an issue tracker persisting state as immutable event logs, allows you to inspect workflows after the fact, lets you inspect diffs inline in the tickets and it is much more lightweight than Jira/Linear. There is no central service to integrate with, as it is Git-backed and lives with your code in your repo.
This looks really nice and I'm definitely going to give it a go. I been relying on Jira so this will be a breath of fresh air. Beads was great in principle and I haven't given it a look in a while but this looks much cleaner.
I have some older projects that use beads (I still run an old version without dolt that's imho pretty good overall) but lately with Fable also have a few newer projects where I just have the agent write docs and keep a worklog with the what/why/decisions etc. (I think I read it here on HN somewhere and figured I'd give that a try.)
The latter seems to work pretty well for now (slightly better than beads) but I'm always looking for ways to improve it. This could be an interesting replacement.
I think it works well for tracking intent. I tell my agents to use a "human-input-needed" tag when there is a high stakes decision fork, and then I just filter on that tag, immediately highlighting blocked workflows. I think you can get a long way with clever tagging, filtering + time travel.
Checking this out, the skill it wants me to add to my project seems to reference issues that wouldn't make sense in the context of my project. IE: `QPPR7PE` and `MBB6WY5` ?
You are totally right in that it makes no sense. It seems like the skill.md had regressed as I had updated it via agent proxy. Thanks for pointing it out, the skill has been brushed up.
Got it! I am not up to speed on Claude cloud sessions yet. Interested to hear how that works out for you. Let me know if you figure out what's getting in the way, I'd be happy to look into it.
I have a lot of little projects and I also prefer this way of working with agents. Sometimes I would start to interrogate on a specific portion or ask questions to better understand a concept, and the session would get poisoned and the agent would fixate on that topic for all the remaining turns.
I asked fable to look at my interaction patterns and clearly stated my frustrations and the problems I wanted solved, and it designed a simple process to track things in git and built a couple simple session hook skills. It’s pretty lightweight and I’ve been very happy with it for a couple months.
The power of this mode of work is that after you deconstruct the task into smaller subtasks, it's a lot easier to use cheaper models to implement that task.
I get a long way using models like Opus to make a plan of action and a bunch of tasks, and then using Deepseek to implement that plan of action. Saves a bunch of money and is fast.
- I have a record of work done and work to be done that helps _me_ when I come back to the project after several months. It’s committed and lives with the code.
- when a task inevitably ends up more complicated than I thought, I can in that session break it up
- I initiate sessions from multiple computers, so things stay in sync (through git)
- I also have a “tooling” repo that builds out some views of the work and hosts it for me to see when I’m on my phone.
- The hooks let the agent manage all of the workflow/task management, so there’s very little management overhead for me.
I rejected beads and JIRA. I wanted something more lightweight.
Nice. I found myself doing a lot of repetitive stuff handing off between agents, time to fix that. I think I agree with your approach of keeping it lean.
I also sometimes rewind the conversation after going off on a tangent, sometimes not if I want the agent to have the sorts of things I’m thinking about in context. I probably justify keeping too much crud in context than I should
I've been on this kick since I realized the primacy of the initial part of the session context. I created a python app that reads a phased plan and kicks off a new session for each phase. There is a standard prompt and handoff mechanism to determine if we encountered any unforeseen issues that we need to address in chat, but otherwise it will just grind with a clean session with appropriate context for each phase.
Just kicking off a subagent does this automatically...?
My main conversation is usually with an orchestrator that hands off work to various (usually cheaper) subagents to plan / review / etc. It has instructions to find the correct model for each task and not to do too much itself so a multi-phase plan automatically gets a fresh subagent for each phase.
i disable subagents. it also helps that i work in microservice size repos, so a single agent can keep the full relevant context. subagents waste alot of unnecessary tokens doing code exploration that the main agent is going to end up duplicating, i think they make more sense for monorepos.
anecdotally I've tried the same plan on different branches with both approaches (single agent with subagents implement the full plan vs. my little session per phase app) and then had agents judge the code quality and my approach worked better and saves tokens. horses for courses.
I use Jobs [0] to manage this—it's an agent-first CLI to track issues and tasks. A single `job orient` command gives the agent the current task in the context of the larger plan. It's a replacement for Plan Mode and issue trackers, and it has allowed me to execute massive plans in parallel with minimal oversight. There's a web UI, but it's a work in progress.
Cinemania 97 was a big part of this era for me too, though it was far less widespread than Encarta. Fantastic resource for classic film, and into the 80s and 90s, though it was bit quirky watching it get further and further out of date, like the big 1999 films that defined my youth (The Matrix, American Beauty, Fight Club, etc) all completely unknown to it of course.
I believe the Orin AGX is in a similar position. It does a UEFI boot but you absolutely have to supply a correct dtb and if you don't then key peripherals like USB and ethernet can just completely not work, or in one case I experienced, subtly malfunction in a way that appears to be fine but throws off a bunch of extra radiation that fails a certification test.
Thanks for speaking to the battery question; I'm very interested in how to run this kind of thing (the display portion at least) cordlessly too, especially when lipo pouches in that range are so dirt cheap.
I might be misunderstanding your comment but the display is run/powered by the board so all you need to do is provide power to the board. For e-ink it uses no power to "hold" an image, only to write/refresh (sorry if I'm telling you something you already know!).
So reiterate, with a 2000mAh lipo battery I should be able to drive the board _and_ the screen for over 3 years if I only refresh it at most once every 4 hours (with "push" capabilities, I don't have to deep-sleep the board to get that battery life). At least, that's what the calculator says, I _just_ got my hardware to play with and so I can't speak from experience. Unfortunately my board is not charging the battery so I need to get a replacement which will probably take a week or two to get here.
Yeah no we're aligned; I have some raspis that I'm interested in doing long term battery stuff with (remote sensing and the like) and it definitely does seem like connectivity is the huge battery killer, like even if you wake up only periodically to take a reading or update an epaper display, the cost in battery just to negotiate a wifi connection and do tcpip and http things is awful.
And then you're stuck with a device that can only be communicated with when it chooses to wake up, whereas with BLE it seems you can listen for a remote OTA wake much more cheaply.
reply