Try to use your IGPU as well if you have enough RAM, almost everyone has dual GPUs in their computers.
Use Vulkan build of llama.cpp to run low parameter MoE models in your IGPU and CUDA llama.cpp to run dense models in your DGPU.
Ofcourse you need to set your expectations right on what problem you want to solve, I've not been happy with using quantized models for coding even on a decent laptop; where as for NLP, ASR, STT etc. work great with this setup.
I tried "write binary tree in rust and ensure it compiles with rustc" using the coder model with their chat it didn't produce the correct output and their chat can't call tools. Then with opencode it went on thinking loop until it exhausted the thinking budget.
4090 16gb, 96 gb RAM.
But I like this overall approach of neutering experts and loading them from ssd to make the model run in low VRAM systems.
IME, local models (so far) cannot do anything resembling "vibe coding" or what I call "high ambiguity prompting."
Ex:
Won't be good at it: "make a web app where you can download files with background jobs"
Much better at: "make a new dbmate migration to create a table with this schema, then write REST-style CRUD methods for it: [pseudocode table definition, you can gesture at types and indexes]"
So, they don't demo oneshots as well, but for actual "LLM-augmented software engineering tasks" they do super well.
With limited VRAM, you can still use local LLMs but you have to set your expectations correctly to solve the right problem.
I run small models on various kind of devices including 1.7B model on the original Jetson Nano (4B) abandoned by Nvidia, I had to upscale the software (OS, Lllama.cpp etc.) to run the model but the model runs at 17t/s.
Small models are great at NLP stuff like classification (e.g. bookmarking), ASR etc. I have built custom browser extensions to save time with bookmarking and categorizing to use with local LLMs.
LosslessCut is great, but I've had issues with the final export like timing being off, having cut sequence etc. I think it is very particular about encoding/container.
Shotcut is my default video editor, if anyone's trying it on Linux please use their AppImage as it solved most of my headaches when I was using Flatpak version.
If it's lossless, it may change the cut position to enable lossless. If you want cut at an exact place, then all apps need to re-encode which will never be lossless.
For your particular requirement, you can't get lossless quality all the time by any software.
LosslessCut gives you the option to cut at the nearest keyframe, or to re-encode the small portion from the chosen timestamp to the next key frame (at most a few seconds). Sounds lossless enough to me.
HF CEO keeps posting about how great local models are running in his Mac, it would be funny to see if he replaces his MacBook with a 500W Nvidia laptop after the deal gets through.
But more importantly for business, if HF fails to use the upcoming ASIC inference chips for their ZeroGPU after the acquisition, their competitors using them might have an advantage.
At this point manufacturers should just open-source their firmwares as there's no barrier for entry to reverse engineer it. They will instead gain from army of end-users willing to put their time and tokens into fixing their bugs for free.
One of the main features of Logi Options + is the plugins which this application doesn't seem to support; you can load it locally or from their marketplace[1].
I myself have an accessibility related plugin to make clicking easier [2] built using their Logi Actions SDK. It finds the clickable elements near the cursor and displays them in the Actions Ring and the LCD keypad. If you have the Logi Dialpad then you can have a complete mouse free workflow with Access Ring.
reply