> a columnar DB will typically produce produce a bitmap from of col2
Bitmaps fell out of fashion after the branch predictor became the slowest part of vectorized execution.
Most columnar dbs use compact selection vectors and loop over the values within them, rather than an if() over each row.
Imagine you had 1024 rows in the col1 column and only 1 row is selected, then this optimization has to only do 3-4 branches before the vector is done vs a bit-map which has to always do 1024 checks.
This became relevant somewhere in the mid 2010s and is called out in the Hive design[1] docs from that era.
> It is mostly a bitmap index, generated from your struct so you don't have to write and maintain it yourself.
That explanation makes complete sense, because with this in mind, I can compare it to Apache Druid which does the same thing for data spilled onto S3.
Since this is immutable, you only had to solve "naming things", since the caches didn't need to be invalidated.
Druid is technically mutable, since it lets you mutate whole segments & then cache these bitmasks without having the original data in memory.
But if your dataset is already in memory, this is a neat implementation of an index and it should be faster than Roaring since you can skip the extra translation to the row-id that requires.
I've been building out a sort of "clickwise"[1] for agents to talk to each other without resorting to English as a communication format.
My problem is approximately ~2000 rules in various levels of conditional nesting from a central context agent to the execution side of things.
This is a better worked example than my confused mess of financial rules.
The structure is a bit more fleshed out, but you are still sending across data which is hard to decipher for a human, but perfect for a machine to bootstrap and learn a set of nested rules about the task.
Finance is a bit less consistent than chess on the rules, but packing into binary "Telepathy" protocols is going to be the next big thing for multi-agent orchestration.
This was pretty much Kerberos, but the embrace-extend Microsoft version was better than the regular krb5.
AD was the only consistent implementation across deployments, because it was nearly identical across customers.
Every linux implementation would be a unique snowflake you needed to know exactly how it worked each time, while the Microsoft version had quirks but it was the same issues in every customer (RBCD).
> Instead of joining the tables completely and sorting at the end, a Rank Join streams its inputs in descending score order (sorted access, or SA), building candidate result-sets as it goes.
Your algorithm feels like a variant of the WAND algorithm the Snowflake Top-K pruning paper [1] pulls from.
You can go distributed with this approach if you have a "gossip" protocol between workers to distribute the heap boundaries periodically.
Before the AI age, a "papers we love" group was a great thing within any company.
Now, Claude or Codex will know the SOTA literature around a topic like this and "uniformly distribute" the past into every single implementation of a similar idea. The future isn't uniformly distributed, but the past is getting the treatment & that's good.
Yes! Search engines like Tantivy have long employed WAND to score across terms. The difference though is that WAND is not joining terms in descending order by score: rather, by doc-id.
This algorithm is closer to Fagin's Threshold Algorithm (also mentioned in your link): the Threshold Algorithm was the first iteration of a rank join.
> Sensors and actuators require millisecond level responses, that's something you're not going to get out of an LLM on the other side of a communications link.
We've gotten to slightly a frame behind for the models I'm using to play video games - 36ms to get the next action.
The idea is probably not have a frontier model doing this, but to make a frontier model build these out of its own brain (well, directly edit the weights instead of RL tuning).
My team did call these sort of slices of a main brain a "Reflex agent" in similar fashion, since they do not reason only respond.
The faster you are, the more systems you're able to control directly. At ~30ms per "choice"/output, you can choose what lower level motors/servos/systems drive towards, but you can't generate the signals that drive them. You're looking at 1kHz and maybe higher there.
> maybe he enjoys the technical tinkering more than the gardening
Por que no los dos?
For me, gardening switches between being an old testament god with a spray bottle and a general process god who sets everything in motion, then takes eternity off.
The hybrid power train is simpler than a regular transmission and made of more durable parts.
The clutch itself might not have to disappear for more fine grained power delivery, the new Ferrari clutch-by-wire[2] is a similar bit of a few hall sensors talking to the ECU.
You can build a simulacrum of a manual low-mode transmission without needing to use it constantly in stop-go traffic or while sliding backwards on loose dirt.
In kei trucks I think the manual transmission is mainly just about cost of ownership. A new 4WD kei truck with MT costs ¥1,226,500 ($7,727 USD) compared to ¥1,303,500 ($8,212) for an automatic transmission - about 6% less [1]. And longer term repairs/replacement of a manual transmission tend to be less costly than an automatic transmission. Not a huge difference - but the applications where these trucks tend to be used are not exactly high margin, either.
I don't think we're likely to see a Toyota hybrid system or Ferarri-style drive-by-wire clutch at that price point. Although maybe in the future we could see EV kei trucks at a comparable price [2].
Personally I think it's worth spending slightly more for the convenience of a manual transmission (and my license doesn't actually permit me to drive a manual on public roads) - although some of my buddies laughed at me for buying an automatic kei truck. I think at this point though automatic transmission sales probably exceed manual transmission sales.
I always found it interesting that Europe and other international locales separate manual vs automatic in license grades. The USA does not - my license does not care if the car I operate is manual or automatic. I'm classed based on weight and axles and length only.
On one hand, you would think it makes sense. It IS an additional skillset to operate a manual transmission vehicle. On the other hand, if you can't operate a manual properly, you aren't getting very far anyway. Was there really that big of a spate of "I didn't know how to operate the manual, so I wrecked the vehicle" sort of issues in Europe and elsewhere that pre-empt this and justify the extra bureaucracy?
Japan is a lot stricter than the US when it comes to driving in general. The minimum age is higher, the driving test is more difficult, it's easier to lose your license for traffic violations, speed limits are lower, mandated vehicle inspections are much more expensive/comprehensive, there is zero tolerance for drinking and driving (even one sip), etc. This is reflected in traffic fatality figures - 12.8 deaths per 100,000 people in the US vs 1.9 in Japan [1].
So while I'm not sure about the origin of manual vs automatic licenses specifically, it's not really surprising in the broader context of Japan's strict stance on driving.
In practice it means if you want to drive manual, you need to take the driving test in a manual - which is not unreasonable. But seeing as how only 13% of foreigners pass Japan's practical driving test on the first attempt (to be fair, it was easier back when I took it) [2], and I hadn't driven manual in years, I wasn't looking to make things any more difficult for myself.
Or there is also the option to take some lessons at a driving school and get certified to drive a manual (with the assessment at the end being easier than taking the official test). But I haven't really had a good reason to bother with this.
It's the same in France (and most of Europe), you have two options for the practical driving test: and automatic only one, which you can attempt the exam for after 13h of lessons, and a manual + automatic one, which you can attempt after 20h of lessons. (in reality almost nobody manages it after 13/20h if you haven't driven before, the average is around 30h of lessons for manual).
If you took the automatic-only test, and have an automatic-only license, you can take an additional 7 hours of lessons to remove the automatic-only restriction from your license.
It does indeed make sense, since manual requires more skill, and attempting to drive a manual on the street without knowing how could be dangerous.
Funnily enough you can visit europe, and drive here on a canadian/US license, without ever having driven a manual.
I had driven manual for years, including in the UK on my Canadian license, but when I swapped my Canadian license for a UK one, I got an automatic only license because I had never taken a test in a manual. I haven't bothered upgrading because I drive electric these days, but it is mildly annoying.
> This is reflected in traffic fatality figures - 12.8 deaths per 100,000 people in the US vs 1.9 in Japan
Deaths per X km driven would be a more useful statistics. Japan has such a well functioning public transport system which just leads to people driving less.
As of 2024 it was 4.7 fatalities per 1 billion kilometers driven in Japan, vs 7.8 in US - or about a 66% increase in relative risk in the US per kilometer driven vs Japan.
Road fatalities per 10,000 registered vehicles were also higher in the US, at 0.4 in Japan vs 1.3 in the US.
Japan does have a significantly larger share of deaths coming from pedestrians (36%) and cyclists (15%) vs the US (18.2% vs 2.9%, respectively) - which is one of the challenges of driving in Japan. You constantly have to be on the lookout for pedestrians and cyclists coming from any direction, often on narrower streets with obstructed visibility. In the US the largest share of deaths is to car occupants themselves (~60%, though it's a little hard to determine the exact value since for some reason SUVs aren't classified as passenger cars, so deaths to SUV occupants count as "other").
I live in Japan and it's pretty insane how close drivers get to pedestrians on narrow streets. And how everyone is so used to it the drivers barely slow down at all.
Even worse was when I lived in the countryside and I'd try to go for walks down the road and motorbikes absolutely maxing out would rocket past me not giving a shit about who or what was on the road. Kinda funny looking back how the most dangerous place I lived was the Japanese countryside
The Japanese countryside (which includes suburbs with some farms scattered about) makes American suburbs look pedestrian friendly by comparison. Talking from personal experience two days ago[0], I was in Nagano at night walking around unlit streets barely wide enough for one car, being constantly passed by both cyclists and cars. I felt like I needed high-vis. The closest I can think of would be the American East Coast, which also has lots of chaos layout streets, but even they usually have the money for streetlights and they don't have the severe space constraints of Japan.
Bonus points: near the end of the walk there was a two-lane road that I had to walk half a km to get to the nearest crossing for, and then half a km back. Bonus points for the one car dealership in front of my destination that was only fenced off on the back side just to spite me (or, more likely, they don't want people using it as a pedestrian cut-through). The road blocking my path was constantly jammed with cars. Somehow, Nagano invented the one-lane stroad! Also, Google Maps doesn't have a good understanding of crosswalk locations, because there was a closer one on the other side that you can't really see on foot at night.
[0] I am trying to go to as many Round1s as possible to get a title in Music Diver for playing the game in a bunch of prefectures. The game gets EOS'd at the end of the month (but only in Japan, which is unheard of). Round1 likes building giant locations in the middle of nowhere with terrible transit access. I think the one in Nagoya even had a shuttle bus from the train station - which I didn't use, lol.
Wonder if it started because of hilly terrain, back when a MT was a nightmare on steep city roads (all over Mediterranean/Alpine cities) and just kept on out of inertia.
Here is a funny one, at least for Europe. Having moved from the US, from a state (not all have!) which has such an arrangement with France, I got my license "swapped" from the US one, to the French one, w/out automatic vs manual restrictions, so no issues about having driven the last 35 years out only automatic, before coming to France.
IDK why some parts of europe do that. Frankly it's stupid. Once you've driven your manual car for, IDK, a week, it's second nature like riding a bicycle.
The US does have separate license endorsements for manual in commercial trucks.
Well sure, but what do you do in the first hours of driving a manual? IF you've never driven one before, how can you legally be allowed to drive one on the road without training? Like, we could say that after you drove a truck for a week it's probably second nature, but there are very good reasons why we don't let people with a B class licence drive them. Ditto for trailers and buses and anything that's not a normal car up to 3.5 tonnes.
Arent you specifying type of transmission? For Indian made 80hp Suzuki Swift, automated manual transmission (slow and jerky unless gently accelerating) is 450$ premium, which matches the price difference of your truck.
For 80hp Suzuki Baleno and 110hp Yaris, CVT had $1200 premium, for 120hp Hyundai Verna, 6 sped TC had $1200 premium.
Ford more or less uses the same eCVT in the Maverick. They are the best hybrid transmission by far, but they still require a generator and high voltage battery.
For dirt-simple, low-fi reliability a manual transmission is still the guy for the job. I generally buy older cars and I almost never buy a car over 150k miles unless is has an manual.
> they still require a generator and high voltage battery.
They require two generators, but they also don't require a starter motor or an alternator. Meanwhile a non-plugin hybrid only needs around 2 kWh worth of battery, which has gotten so cheap you might prefer that to a clutch replacement.
> configure the benchmark to run for a long duration in a steady state: e.g., hours.
Benchmarking as a developer activity isn't going to cut it long term, that is at best a proposal to change.
If you do an active benchmark, which is basically running a workload when the SuT is tied up to a "data polygraph" of sorts, then you tend to accidentally build production observability tooling in the process.
The best kind of "benchmark" comes from a fleet in action, rather than something which was run in isolation. But those tend to be entirely a 1 of 1 test scenario rather than a real replicable result.
Most of my past performance work revolves around caches, those are exceedingly sensitive to workload patterns and what worked in May will fail in October.
With the whole cloud thing, the "workload pattern" might be someone else's.
My team put together a blog post last week which concluded that our coding models work better in PST afternoon rather than the mornings.
Bitmaps fell out of fashion after the branch predictor became the slowest part of vectorized execution.
Most columnar dbs use compact selection vectors and loop over the values within them, rather than an if() over each row.
Imagine you had 1024 rows in the col1 column and only 1 row is selected, then this optimization has to only do 3-4 branches before the vector is done vs a bit-map which has to always do 1024 checks.
This became relevant somewhere in the mid 2010s and is called out in the Hive design[1] docs from that era.
[1] - https://issues.apache.org/jira/secure/attachment/12594231/Hi...
reply