In Cargo-built Rust, a crate is costly in terms of abstraction because it needs its own directory, Cargo.toml, src/ subdir, etc. In the Google build systems I've worked with in the past (and it looks like Chromium follows) they instead put source files in the same directory mixing languages and with finer-grained libraries as they see fit.
Given that a crate is the unit of build parallelization in Rust, there's a benefit to more crates especially when they don't cost much cognitive overhead. You could imagine using crates in the way you'd typically use separate mods in a normal Rust project.
I dug around in their code search to find some examples but didn't see much Rust in there, seems still early. I think https://source.chromium.org/chromium/chromium/src/+/main:tes... has ~4 Rust files that end up separate crates though. (One of them defines a proc macro so even in a Cargo built Rust project it would need to be in its own separate crate, I think!)
Recently bought something online from Home Depot and it arrived damaged. When I asked them for help they said I had to drive it back to the store myself.
If I have to drive to the store to get it, I wouldn’t have shopped with them! It feels like every time I don’t order with Amazon the stores remind me why I ought to.
I moved from a rich city social life to a lonely suburban one (I have regrets, there are complicating factors). I met a fellow dad who mentioned a book as we started chatting and I was excited (someone who reads!).
As we talked about this very isolation problem he told me about his neighbors who kept inviting him over and offering to watch the kids for him and how he thought it was just weird and couldn’t trust them. It broke my heart — I could just imagine the neighbors trying to be good to their community and that very behavior is met with distrust.
(I read half the book he mentioned and asked him about it the next time I saw him. Turns out he hadn’t read the book, just heard about it on a podcast. My quest to meet someone who reads continues.)
A trick I learned from an east bay friend is that (when possible) the bus across the bridge is way nicer than the BART under the bay, especially around sunset.
A bus that gets stuck in traffic is by definition a failure. It will (again by definition) never be a viable alternative to driving, meaning a reinforcing loop where more driving means more traffic for the bus to be stuck on, meaning the bus is even slower and less attractive...
> It will (again by definition) never be a viable alternative to driving
I don’t follow the logic here. Why would a bus (periodically) getting stuck in traffic forever invalidate it in comparison to getting stuck in the same traffic in a car?
In aggregate, people go the fastest way - no way to stop this from happening. In places where everyone takes public transit, the transit is fast. To make more people take public transit, you make the transit fast.
Reserving lanes as bus lanes can actually speed up car traffic by making more people switch to bus.
If buses and cars participate in the same road traffic (i.e. the bus doesn't have dedicated lanes), then a bus trip will never be faster than a comparable car trip. This is because you have to walk to/from the bus stop at both ends of the journey, wait for a bus to show up, ride the bus and have it stop for other passengers, transfer between routes, the bus has slower acceleration and turning than a car, etc.
Think about even the most strawman trip possible: A bus rider and a car driver start exactly at the location of a bus stop, the bus shows up exactly at the start time, both contestants ride/drive straight for a few kilometres (which happens to follow exactly one bus route, without changing buses; but the bus might stop a few times along the way for other passengers), and end exactly at a bus stop. The car will win every single time.
It's good to have dedicated bus lanes, but the point of a bus isn't to get from point A to point B faster than a car.
It's a part of a transit ecosystem that makes it possible to survive without a car. It fills in the the gaps where it's not economical to add train/metro lines.
Caveat: the bus can be a little bit slower and still win, because you don't need to drive. Or the car can be a little bit slower and still win, because people are antisocial. When the difference is more than a little, then you're correct.
Except that many buses broadcast their location, and many cities prioritize green lights for buses over cars.
So the car might need to stop on red multiple times (unless following a bus), while the bus will have most lights automatically turn green in front of it.
Yeah, the car will be faster, unless there are accommodations that benefit the bus only (like dedicated lanes). But the bus doesn’t have to be _faster_ to be viable to many people: there are other things going for the bus like not having to park, often being cheaper especially if it allows you to not own a car, etc. etc.
A big one: no need to find parking! Or to go back to your parked car on the reverse trip. You can do a triangle trip A,B,C,A with public transit. With a car it has to be A,B,C,B,A as you left your car at B.
I do trips like A,B,C,A all the time in a car. Sometimes even A,B,C,D,E,F,G,H,I,J,K,L,M,N,O,P,Q,R,S,T,U,V,W,X,Y,Z,A trips in a car.
If you're sprinkling in little rules like "ah but you couldn't park at C", well, "the bus doesn't go to C" is also a valid restriction I run into often. So often when I take transit, its drive to train station -> take train someplace -> take bus someplace -> walk several blocks to destination -> walk several blocks back to bus stop -> ride bus to train station -> ride train back to train stop -> take car home so a A,B,C,D,C,B,A trip is actually quite common just to do the same as A,D,A with a car.
Bus lanes make a bus a very good alternative to driving, and they are much more efficient space-wise compared to cars (and god forbid, pick-up trucks), which means you can get more throughput...
I've heard multiple times Americans complaining about being on the bus.
Genuine question, what's up with being on the bus?
I lived in Vancouver BC, I can't see anything wrong with their bus. Not perfect, but yeah... It's a bus, goes from stop to stop. Anything special about bay area buses?
Service quality is just abysmal, extremely crowded at basically all times, delayed or skips your stop when your standing there, when it isn't crowded there's all sorts of other issues with the service, and this doesn't change the fact that it doesn't really take you where you want to go without transfers.
Truthfully, whenever I'm in the bay area I just find myself walking. There's a reason why e-scooters first found major adoption there.
Yeah. I took the bus in NY and SF and I’m also not sure why the complained. Maybe because I’m a tourist but especially in SF I had always nice interactions with the driver. Mind you I took the Bus in the City Centers. Is it that much worse in the outskirts or is it just the concept of taking a Bus with multiple other people the problem?
The buses in SF and NYC suck because of this bipartisan belief of public transit as social welfare rather than a means for the bulk of society to get to their destination. This leads to things like far too many bus stops because granny complained, lack of fare enforcement, high staffing costs, etc. To make the bus as nice as Vancouver, these cities would need to cut stops and do a draconian fare enforcement campaign. For some reason, Canadian bus riders are not ungovernable barbarians like Americans, despite being poorer on average and more diverse.
In SF it really depends on which line you're on, and a bit of luck. In my experience the 9 was the worst. I didn't ride that one often, but almost every time I encountered something ridiculous. One time a guy rolled a blunt and started smoking it on the bus. Another time a guy in a wheelchair spilled a bottle of booze that then ran up and down the floor of the bus. I got off early several times when an obviously crazy homeless person got on and started silently staring me down.
But even popular commuter routes like the 38 can get pretty dicey. One time a guy was blasting music out of a bluetooth speaker. He had headphones sitting on his neck, so I asked him to use his headphones instead. His response was to get in my face and start yelling, threatening to beat me up. Everyone around us stepped away and ignored us. Apparently the next stop was his, so he got off while threatening to jump me if he ever saw me again.
If you take public transit in the bay area regularly, you are almost guaranteed to have some very unpleasant experiences.
I don't think so. American buses are ridiculously outdated, while having to compete with some of the most well-equipped cars in the world. If Europe had such buses, they wouldn't be popular either.
The shitty metro buses in my 3rd rate "city" have air conditioning, wifi, live location tracking to your phone, wheel chair lifts, tap to pay, tap with app to pay, and run on propane because they are old enough to have "features" left over from the last oil crisis.
Can you tell me what a "Not outdated" bus looks like to you?
Europe isn't the richest place. Some of the western cities surely have nice buses, but there are plenty of old shabby buses there too and people still ride them. Might be that the routing and timing sucks, or the general cultural individualism.
We were discussing riding the bus to a place that depending on traffic is a 20 minute car ride from SF, or a 2 hour car ride in the worst case scenario traffic.
There are a few bridges that connect SF to other places across large bodies of water, the Bay Bridge and the Golden Gate Bridge. These bridges do not provide a dedicated lane for the buses. These bridges act as a choke point for traffic. As a result, the buses are not reliably on time, because the traffic is hard to predict and it depends on whether or not people have a car accident that evening.
As I mentioned in another comment, these buses do not have bathrooms inside. So you are spending an unpredictable amount of time standing at an unsheltered bus stop, often with no bench to rest on, followed by an unknown amount of time on the bus.
The buses come infrequently enough that if you go find a restroom after waiting a while at the stop, and the bus comes while you are gone, it will be a decent chunk of your life spent waiting for the next bus to come as well.
This might be fine if you're on vacation and you just wanted to explore the Bay Area, but if you live here and work here, this is just too much time!
If you have a car, there is essentially no upside to taking the bus in this situation, which means if you are greedy and choose to drive instead you contribute to traffic by taking up more space on the road.
In contrast, if you choose to take the train, you know exactly how long the trip is going to be, because there's no variable traffic disrupting the schedules. It's predictable. It's convenient. Sometimes the train might get stopped for one reason or another, but the trains come so frequently that this is rarely an issue.
Before your trip, you're also inside a train station, which is sheltered, has seating, and even often has bathrooms.
There's a sign telling you clearly how long until the next train arrives, you don't have to count on something on your phone to find out.
My understanding of the initial commend “But then you are on a bus” was a negative one to buses and or public transport in general.
I know the geology of SF and drove both bridges a few times myself but was always lucky to not end up in traffic. I would also always prefer the train over the bus. But currently where I live this also is put to the question. I live in Berlin and we have a great public transport network which at the moment struggles a bit on multiple fronts. This happens every few years at least in my experience.
I like to try out the public transport situation in different cities for my own comparison. Funny enough one of the best times I had was driving the metro in D.C. Met a very energetic metro employee who explained the machines and the system :).
American busses are infrequent and often don't follow the schedule. This means any trip where you have to be at certain time (work meeting? class? personal meeting? some event? etc...), you have to add 30-60 minutes buffer, and that's on top of bus being already slower than the car.
Buses are widely perceived as signaling low socioeconomic status in the US, to the extent that it's become a self-reinforcing negative spiral. (As soon as a system loses its higher income riders due to real or perceived dangers or inconveniences, political support for tax subsidies and infrastructure investments (bus lanes etc.) tanks, quality then actually declines, rinse and repeat.)
I've taken buses in the Bay Area and New York a bunch of times and personally haven't had negative experiences (at least not compared to rail-based US transit), other than that the comfort of the buses was generally a few decades behind those in Europe. On the other hand, many of them even have Wi-Fi!
Where I am, buses are heavily subsidised; cost is about $3 per passenger mile. Fares are much cheaper (so cheap nobody ever really thinks about it, other than worrying if they have quarters for exact change).
They are often empty or have 1 or 2 riders.
They are slow, often show up to a stop late, so you have to plan everything an hour early.
They don’t have amenities like bathrooms and are uncomfortable for long trips, even though that’s the only sensible way to use them.
Due to political pressures, they have many, many stops and few express buses, so they are much slower than a car. The stops usually have nobody at them but the driver still has to slow down and check (sometimes they will just skip a stop they know is almost always empty).
The bus depots are not particularly safe or pleasant after dark and have trouble with attracting antisocial people.
Meanwhile, it would be cheaper to just pay for Ubers for anyone who wants them, and just make the Uber show up randomly late to replicate the experience.
There is a market emerging here often branded as “hood Uber”, “Alicia’s Uber” etc of people who will drive you for $1 a mile (cheaper than Uber) negotiated by text message and paid via CashApp etc. It’s completely unregulated. Some are now offering regular service in a minivan on a fixed route taking people to/from employers every day.
As a human who sometimes needs to use the restroom, I like my transit rides to take a predictable amount of time.
Sunset on the Bay Bridge is prime rush hour traffic time, and the favorite pasttime of many Bay Area drivers is to get in a traffic accident after work.
As far as I can tell, these double decker buses are not equipped with a restroom for their riders. Even though I'm sure the view is amazing, the red line is less often obstructed by traffic.
No doubt they are possible. One factor that discourages their existence though is that the dregs of society mess them up or go in there to do drugs in private and then conk out. It's the number one pushback to providing restrooms for the public. It sucks, but it's how it is. Most likely the only way around this is to stop subsidizing transit and to be very careful to enforce fares. This is not popular among the people who love transit, who tend to also love the underdogs and root for them.
I'd be happy to pay bus fares that cover what a bus actually costs; needy people can apply for and receive assistance that is income-based (and get booted out of the programme if they engage in antisocial behaviour).
No bladder issue. Probabilistically, when you load a bus up completely with people (the ideal situation for transit) one of them will have to use the restroom over the course of an hour and a half.
The solution is to make changes so that the bus doesn't take an hour and a half for a short distance. Like a dedicated bus lane during rush hour on a bridge.
I think the Rust feature you’re looking for regarding recompiling the standard library is called “build-std”, that should be enough for you to search for it. (For similar reasons you also need that flag if you are trying to use Rust to build multithreaded wasm binaries, so it might come up for you!)
In my own journey of discovery I found https://cheats.rs/ very helpful, and in particular its "memory layout" section has visualizations. (No affiliation with the site, just a happy reader!)
[Ninja author here] Nice post, cool to see the deep dive! I also appreciate the details on how they produced their numbers.
As they observe, Ninja gets to be fast mostly by cheating: it avoids a lot of work by saying many things are just out of scope for Ninja to do, and that means it is a useful a target to race against. (Funny thing: when I wrote Ninja I was misremembering how fast an earlier build system was so I kept trying to make it faster. So don't treat it as a lower bound, I just made it up!)
I comment here to say I find the explanation for 'why' in this post unsatisfying. They mention three design decisions.
The first one is a criticism of CMake, not Ninja (?), so I don't think it can be why. I might have misunderstood?
The second reason given is doing some work like header dependencies in multiple threads. This is the most plausible reason to me but it still feels unlikely. It's a very small amount of work: the post mentions 300 compiles, so maybe parsing 300 small text files?
The third is that they run the compiler up front an additional time to gather headers, which is strictly more work than Ninja. There is some hand waving about file access patterns but I am skeptical; if the end-to-end build time is 3 seconds then the project is small enough to all fit in kernel caches. They also mention doing other things like invoking the compiler to get version information. This seems like it would dwarf any performance gain from number 2.
Maybe it's just my own curiosity, I think this post would be better if it had a better explanation for the reason. I'm not disputing the result, I just think the result should make you suspicious that something else is going on, and you might learn something from that! You could for example explore whether it's the header dependency thing by profiling the Ninja invocation and seeing if it's waiting for CPU or waiting for tasks to execute.
(If I had to guess without looking at any of the involved code, I would predict it's something about how CMake generates the build, like it introduces serialization in a place where build2 is parallel, or it adds some extra build steps like gathering the current git hash into a header file or something.)
[build2 author here] Thanks for the feedback! Some additional details:
> The first one is a criticism of CMake, not Ninja (?), so I don't think it can be why.
Fair enough. The point I was making is that if you want to compete with Ninja, you cannot leave any potential performance gains on the table.
> The second reason given is doing some work like header dependencies in multiple threads. This is the most plausible reason to me but it still feels unlikely.
We are talking about ~2% performance difference here. Parallelizing even a small amount of work across 24 threads rather that doing it serially saving a percent or two feels plausible to me.
> There is some hand waving about file access patterns but I am skeptical; if the end-to-end build time is 3 seconds then the project is small enough to all fit in kernel caches.
It fits into the system's file cache unless there is memory pressure, like one would expect from having 24 C++ compiler jobs running in parallel. We actually measured this in isolation (with more detailed results in the linked article) and it has a measurable effect.
> They also mention doing other things like invoking the compiler to get version information. This seems like it would dwarf any performance gain from number 2.
I measured this, it costs 70ms or ~2% of the overall time.
> It fits into the system's file cache unless there is memory pressure, like one would expect from having 24 C++ compiler jobs running in parallel. We actually measured this in isolation (with more detailed results in the linked article) and it has a measurable effect.
300 TUs is not much. If they build in 3 seconds then they are trivial (small). If the machine is 24-thread (I assume some sort of heterogeneous 12-core), how little RAM does the machine have for the kernel to start evicting page cache during the build?
> > They also mention doing other things like invoking the compiler to get version information. This seems like it would dwarf any performance gain from number 2.
I measured this, it costs 70ms or ~2% of the overall time.
Couldn’t you amortize this to 0 by just caching the result and only changing it if the binary timestamp changed?
> Couldn’t you amortize this to 0 by just caching the result and only changing it if the binary timestamp changed?
Yes, that would be nice, but the tricky question is can any of this information change without the compiler binary mtime changing? First off, GCC's gcc/g++ binaries are drivers and are not what does the actual compilation, it's private cc1/cc1plus binaries that do the job. Can one of these change but not the driver? I think it's plausible (some package manager optimization where the file is not touched if it hasn't changed). So at a minimum we would need to discover where those are located (probably by invoking gcc/g++) and checking them as well. Could there be something else? Who knows. We value speed very much but we value correctness even more.
I think a more fruitful direction to explore is to improve GCC itself to dump all this information in a single invocation and in a machine-readable format (JSON). I think if we go from 70ms to 14ms (and perhaps even lower because this special GCC mode could conceivably do things faster than how we do it now), it would be good enough.
Simple CMake projects using the Ninja generator are very efficient, unless you create generated files.
And if they are in their own targets, they are not really an issue (they would serialize everything that depends on them as you'd expect), but if you have them in a library grouped with other files to compile, then the whole library compilation is serialized.
And obviously worse if you also have to build the generator for the generated files, but that's not a big surprise, you can observe that in full builds of Chromium or its libraries too waiting for protoc if you crank the parallelization a lot.
Yes, I don’t remember the details but vaguely remember that CMake tends to group things together that could in principle be made more parallel, as you mention with generates files in a library. On the other hand if build2 makes it easier for authors to express these kinds of patterns without the serialization then I count that as a win for build2!
I feel like make/ninja suffer from the same output modstamp > input modstamps to rebuild. Really a build system should track modstmaps on all inputs (apps included), and the rebuild on not-equal.
Reading about manifests immediately reminded be of Clearcase/Clearmake. My memory may be not correct after all the years passed, but configuration records for derived objects contained the same information as proposed manifests + some more metadata about source (versions, at least).
If I remember correctly SCons allows to implement a custom "checker" (or whatever it is called there) to compare files based on any criteria that are useful, not just timestamps. But this is about the only visible advantage of SCons over Make, at least for our case.
Why not just embrace Content Based Addressing and the Bazel action cache? Not necessarily adopt the RBE protos, but you could have on-disk cache that are similar or even identical.
Nowadays, I think a build tool that doesn't natively support distributed caching (and possibly remote execution) is a weird choice. And I don't think that spawning layers of processes allows for good parallelization as you don't know if an action is going to be network bound or compute bound and the job slot is then spent. So you either oversubscribe or undersubscribe.
Sadly, I think things like distcc or ccache have mostly gone out of mode. Vertically scaling (multi core, multi thread, multi TB RAM build machines) and simple artifact caching have won for now. Artifact caching with dependency tracking in the midst of job scheduling was solved “good enough” by OS based mechanisms. Having used and managed the basics of a build farm, removing the need for networking, build farm management and build job coordination is a huge burden relief when your source tree doesn’t require it.
Part of the motivation to use processes is because their structure helps to keep the job generic, uni
I stumbled upon the very experimental nix-ninja [0] recently.
They seem to create a dynamic nix derivation per compilation unit, which would be very similar to what you describe as manifests in your post as it also creates a hash of all inputs.
Would be interesting to here your opinion on that approach
I really like the ideas you have in n2. Hashing the entire build string is a great idea. And it really is as simple as just having all the timestamps of the inputs in the hash. And you have the output timestamp as a quick test.
Always glad to see authors on here. Thanks for your insight!
Since they had to rewrite the build file for their program I also assume that something is missing. Didn’t see any mention of verifying that.
I also really didn’t like their denigrating tone. It totally turns me off trying build2, because it seems they don’t understand the point of separating build stages like environment setup (getting dependencies), configure, native build, cross build, packaging. I am a very happy ninja user instead of a batteries-included solution because it does its one job well and can be used very flexibly. I personally detest cmake, so I use nix + own configure script + ninja.
The blog is also wrong about ninja being unable to call configure, but I intentionally don’t want that (I want the build stages to communicate in one direction for sanity).
> I want the build stages to communicate in one direction for sanity
That's easy to do if you control the whole pipeline and can integrate all the features together, not so much with the CMake model unfortunately. I think it would be nice if CMake had Ninja integrated as a library, it could lead to some nice optimizations later.
Given that a crate is the unit of build parallelization in Rust, there's a benefit to more crates especially when they don't cost much cognitive overhead. You could imagine using crates in the way you'd typically use separate mods in a normal Rust project.
I dug around in their code search to find some examples but didn't see much Rust in there, seems still early. I think https://source.chromium.org/chromium/chromium/src/+/main:tes... has ~4 Rust files that end up separate crates though. (One of them defines a proc macro so even in a Cargo built Rust project it would need to be in its own separate crate, I think!)
reply