This is so cool. I spent years at Pachyderm (YC...something) banging our head against:
- we're "Bazel for data"
- but...we have no Starlark. We're trying to port Airflow but it's not playing well with the way we cache partial results.
Amazing to me that I'm only now seeing the architecture pattern of "pure coordination language connecting sandboxed imperative components", which is used by Nix and Bazel, applied to distributed workflows. If somebody applied this to batch data processing you might have a Pachyderm that actually works better than ours did.
Cool thought! We have an internal version of map reduce implemented in waymark that worked pretty well (though uses Redis for its intermediate values so it's technically not an end-to-end integrated solution).
In principle I would love to have something like this working for arbitrarily large ETL flows; ie. write the logic you want and it tries to figure out a good distributed algorithm to get the outputs. At the moment flowing data through the cluster is the biggest bottleneck. We generally assume that action inputs & outputs themselves are pretty small at the boundary even if their containerized context is quite large.
Well, it was just that the models didn't really line up.
To give a somewhat longer description of what Pachyderm does/did:
1. Connect data-scale version control with pipeline orchestrator.
2. Version control tells orchestrator which shards have changed since the last run and which ones haven't.
3. Orchestrator only processes changed shards from one run to the next, instead of all of them.
4. Profit.
That's fine for e.g. one mapreduce, but,
1. If an action's behavior changes, you can't re-use its prior results (usually—sometimes you're just adding print statements to debug a failing shard)
2. If the overall pipeline changes, you can't re-use results from the first change onward
3. Figuring out if the overall pipeline changed is not necessarily easy, and if you're always throwing out everything just in case, your product is not helping people.
So, the way Waymark uses a compiler for the orchestration layer would've worked so well with our model.
It's also just satisfying—once I learned in detail how Bazel/Nix work, I see a fit for that architecture pattern in anything remotely pipeline-y, and it drove me nuts that Airflow/Beam/Dataflow weren't really using it. So I'm very happy to see this for that reason as well. And, as if that weren't enough, it's so cool that it introduces a pure language at the coordination layer (which this type of architecture basically needs, hence the Nix language/Starlark) without actually requiring a new language, or even a new file. Amazing!
Yeah, that's an interesting use case. The problem for you with a compiler would be how to get it to work with all of the efficient python modules like pandas, script, numpy, pytorch etc that all implement their backend in C with intrinsics/GPU usage.
Well, this is something we've been, kind of, getting ready to tackle with some of the aspects of the internal design of the VM. In short, it would be pretty easy for us to just add an "instruction (sub)set" to work with GPU kernels or pytorch or etc - and then to implement it via whatever means fir best (a third-party crate, a GPU offload via vulkan or sth, heck - even spawning a python interpreter to handle some big operation where the overhead of python bringup is negligible in on the table).
The main issue so far is that our VM that does the workflow orchestration and the actions runtime are living in separate processes - but we even then, we could do tricks to share memory between them for more efficient / zero-copy data passing... That's not something we have right now though.
With Pachyderm, part of our model was that the "actions" were arbitrary, user-provided containers[^1], and the inputs/outputs were mounted filesystems (`/pfs/my_first_input` was a directory containing the first input dataset, or one shard of it, etc, and `/pfs/out` was a writeable directory where the output for that shard would go[^2]). We built the (versioned) filesystem that data in /pfs came from/went to, but you could probably use ceph or something if you're not trying to do versioning. That's how we dealt with big inputs/outputs.
We never tried to optimize the single-machine case, for better or worse.
[^1]: Using arbitrary containers was a good choice for us, though, because we were marketing to spark users and a lot of them were frustrated with how hard worker-dependency management is in spark. So we got to say "put perl, weird sequence analysis libraries, whatever you want, in a container, and if it can read and write files, we'll mapreduce it over your data".
[^2]: `/pfs` was a terrible choice because it's nonstandard. Really wish we'd used `/mnt` or `/mnt/pfs` instead)
In Seattle, local residential streets are often designated as "stay healthy" streets, where bikes and pedestrians have right-of-way. You can drive a car there, but it will be slow (intersections are narrowed to discourage fast driving). Cars take parallel streets if they're going more than a block or two.
I haven't checked the statistics but it seems to work. There's one near me that's packed with bikes during coomute hours, and I think the cost to the city was negligible.
I wouldn't paint Seattle as a picture of success when it comes to bikes. We've spent untold millions on bike lanes, reduced vehicle capacity with road diets, reduced parking and access to small businesses and made life more difficult for delivery drivers and ADA access.
After all that, the mode share of biking has actually gone down in Seattle over the last decade, to under 3%. It turns out if your city has hills, rain and darkness much of the year, there is no amount of money that will convince people to ride a bike. We've effectively subsidized a recreational interest for a very small population, a group so small that the top profession of the ~12k who do cycle is "college professor". And as a pedestrian who has to share with cyclists almost every day, it's been a net negative. In my own neighborhood they've made sidewalks unusable for pedestrians in the quest to encourage more cyclists.
This was in another post but I'm likewise skeptical that "commuting to work" gives a holistic picture of bike usage these days. I live in North Beacon hill, and what I've observed:
- Limes going up and down Beacon Ave constantly. I think a lot of it is young people, I doubt most of them are going to work.
- Healthy amount of bike traffic on 18th Ave S in the morning and afternoon (a stay healthy street. This is my commute)
- The bike garage at Judkins Park station is packed by ~9:30am every day. People lock their bikes/scooters to whatever they can (mostly e-bikes, interestingly).
True, poor choice of words. But it's one of three bridges between the downtown core and several suburbs, and it's hardly a broad indicator of cycle traffic in the region.
It doesn't change the fact that the survey data shows a smaller percentage of people are cycling than in 2015 in Seattle, King County and WA.
"After all that, the mode share of biking has actually gone down in Seattle over the last decade, to under 3%" <-- your article is from 2018 and that is like 8 years old, an eternity ago.
There is significantly more bike infrastructure now then in 2018. Plus the lime scooters and bikes have radically shifted the balance of bikes vs cars to the point where I'd say that bikes now rule the street in most of the dense parts of Seattle.
I would imagine there are at least 10x more people using those lanes than there was before affordable e-bikes and "bike shares" showed up. You've got people who would never, ever ride in a million years be caught dead on a bike using those bike lanes to run errands, go to visit people, commute to work, etc. Just sit down anywhere in the dense parts of the city and you'l see them go by constantly!
I doubt the people who put those bike lanes in back in 2018 and before ever could have imagined how popular they are now thanks to lime bikes, electric scooters and such. Its honestly pretty remarkable and thank god somebody had the thought to build up that stuff back then!
Stockholm is colder, further north, (though admittedly less hilly).
I can't argue that the bike infra improvements in Seattle haven't made a huge difference, but I'm not at all convinced this is because it is intrinsically unlikeable.
For what it's worth, the environmentally incredibly bikeable SD/LA has such bad infra that they have both fewer bikers and far more deaths.
This is more in the direction of what I'm talking about. I think it's a pretty good model and affirms not contradticts what I'm saying because you do have to sacrifice drivability (unless you can just add streets like this - but I assume not).
Well, I was basically agreeing with the idea that bikes and cars should ideally be separated. I don't know if the laws are even formally different on these streets, but it kind of communicates to everyone what they should expect and helps cars and bikes stay out of each others' way, which they both want.
I also think that residential streets are often under-utilized, though (mostly existing to connect houses to thoroughfares), and I'm skeptical that drivability is being sacrificed all that much—I still don't have statistics but IMO there are plenty of places to drive and I don't find my neighborhood hard to navigate by car. Just one person's experience, but I think it's a nice, cost-effective solution that I hadn't seen before I moved here.
(I have a bike and commute with it most days, but I also have kids, so weekends are all driving, and I sometimes need to drive to work too, for one reason or another).
I don't know if the laws are even formally different on these streets, but it kind of communicates to everyone what they should expect and how they should plan their trips (though they do have these concrete bollard-looking things in the intersections that narrow them to one lane, presumably to slow down car traffic).
IME it just makes it easier for cars and bikes to stay out of each others' way, which I think they both basically want.
Nobody pretended they're a car. But yes, people on/in vehicles use the road that's dedicated to vehicle traffic instead of narrow sidewalks where they harass pedestrians and little kids.
I think this is what Baumol’s cost disease is about (productivity growth on one sector drives up labor costs and ultimately prices in other sectors, such as healthcare, construction). Basically, when software or AI is super lucrative,
1. students go into new lucrative thing
2. supply of doctors goes down
3. doctor wages go up
4. hospital margins go down
5. Hospitals raise their prices to stay in business
6. Wealthy engineers are able to afford higher prices
7. Services that are transitively consumed by the high-productivity sector stabilize at a higher price. Services not transitively consumed by the high-productivity sector go out of business.
- I remember visiting a developing country about 15 years ago and seeing people getting paid to stand there and hold a rope, because I guess their time was cheaper than cones, which I think is the inverse of this.
- I also think that, in the limit, society consists of a handful of economically viable occupations and infinite cheap stuff. Nothing else is worth the prices that a seller would have to charge to afford employees.
- I also think that we’re entering a stabilization phase.
>I think this is what Baumol’s cost disease is about (productivity growth on one sector drives up labor costs and ultimately prices in other sectors,
This is a clear and accurate description, but your explanation of the underlying reasons isn't really as clear.
>1. students go into new lucrative thing
Part of the Baumol effect is that the share of total employment in the sectors with high productivity growth decreases, while that of low productivity sector increases. This is sort of the opposite of what you are saying.
>2. supply of doctors goes down
The Baumol effect by itself has no effect on the number of Doctors. There are always more qualified people who want to be Doctors than are allowed to be Doctors for a great number of reasons. It isn't because not enough people want to be Doctors.
3, 4, 5
Yes those look like the Baumol effect.
6, 7
Not really what the Baumol effect is talking about.
Baumol illustrated his point in a simple manner. In the 1700's a string quartet required four musicians, and today a string quartet still requires four musicians (no productivity gain). However, those musicians today expect 21st-century salaries to match the rest of the economy, making the cost of hiring a quartet vastly more expensive relative to goods like mass-produced items.
TIL! I’m entirely inexpert in the topic and posted as a curious layperson.
That said, some of what’s in the article only makes sense to me in the context of a system that’s at least somewhat like what I tried to describe. Maybe you can enlighten me?
> This effect is an example of cross elasticity of demand.
Is the cross-elasticity of demand here not referring specifically to the cross-elasticity of different forms of labor, which exists due to the many common inputs of different forms of labor? Eg land, healthcare, unskilled labor (as an input, in the case of skilled labor as the output—doctors and engineers in my comment)?
In the case of a quartet, if musicians charged what they did in 1700, they would quickly become homeless, and then quickly get other jobs.
So people are either leaving the profession or raising their prices, and either firms pay, high-wage-earners pay, or they get no customers and again get other jobs (like the rope-holders). Is that not the effect?
>...Is the cross-elasticity of demand here not referring specifically to the cross-elasticity of different forms of labor, which exists due to the many common inputs of different forms of labor?
This isn't necessary to explain Baumol cost disease.
Different kinds of workers may also be substitutes for one another, or complements. However, the quartet doesn't need to share a particular set of inputs with car manufacturing for Baumol's mechanism to operate.
>...In the case of a quartet, if musicians charged what they did in 1700, they would quickly become homeless, and then quickly get other jobs.
Yes. Baumol cost disease is a way to explain the upward pressure on the cost of lower productivity labor-intensive services. It isn't an economic law, it is a tendency. It doesn't guarantee that every business will be able to pass those costs on to customers, or that every business will survive or that every worker will receive the same wage increase, etc.
Ok, I did more reading. I was actually super wrong about this, but it took me a while to get there. Oops! But very interesting!
Basically, people and firms are little black holes of unmet demand for all sorts of things. When the productivity shock happens, demand for stuff in the now-high-productivity sector(s) is now (mostly) met, freeing up money (that people and firms in the high-productivity sectors have more money to meet their own latent demand is also true, but somewhat incidental). That money goes to meeting other latent demand, some of which is for stuff in low-productivity sectors (housing, childcare, education, healthcare, construction), and that’s what drives up prices until/as people eventually move into those sectors. Even if housing specifically isn’t constrained, it still happens.
And I guess…as wages go up, some forms of automation also become economically sensible, which is what changes the balance of low/high productivity sectors. Latent demand for rope-holding is met with cones, latent demand for maids is (mostly) met with washing machines and dishwashers, latent demand for secretaries is (mostly) met with personal computers, etc.
So with AI, it sounds like people will move out of software into construction, healthcare, etc (only specialized work where the automation isn’t “good enough” stays). All the “teaching kids to code” that schools are doing now, eventually turns into people dealing with their own AI output as adults, where previously they would’ve hired a developer (like how managers who used to have a secretary now get a computer and type for themselves). Wages go up, education, healthcare, and housing get more expensive.
The Wikipedia entry actually says 2, or something very similar:
> The rise of wages in jobs without productivity gains results from the need to compete for workers with jobs that have experienced productivity gains and so can naturally pay higher wages.
Supply of doctors doesn't go down only because hospital raise wages to compete with the new lucrative thing.
There are plenty of people that want to become doctors, but only so many of them are people that you want to be doctors, and if you don't pay enough they'll do the lucrative thing.
Doctors are just not a good example to use though as an example here. Healthcare costs have risen as one would expect due to Baumol’s cost disease, but Dr salaries have generally not gone up as much as one would have expected. Medicine is such a tightly controlled market that lots of things are distorted - Medicare decides how much they will compensate, etc. Many more qualified people would like to become Drs but are not allowed too (From 1980 to 2005, the number of medical students stayed flat despite population increases - this is slowly been increasing the last 20 years along with the number of residency positions.)
Yeah, and it's similar reasoning as to why gentrification tends to happen when the wealthy move into an area, and why prices for things typically decrease when the wealthy leave.
The argument works with any service business that is shared by high income and low income earners: restaurants, transportation, clothing, housing, groceries, etc.
So is the orthodox economist solution: government, the biggest employer and biggest debtor in the whole economy (by far) starts printing money to pay its employees more, thereby using inflation to redivide the debt to the government, and then to its employees which then conveniently also forces every employer to increase pay.
Which means the productive economy gets a greater share of the wealth, by taking it from the rent-seeking part, whatever that may be.
Of course, billionnaires hate this, since their wealth is pretty much some percentage of that debt.
But pensions go down as well, because that's how the system was redesigned by large banks after the last time governments really did this. This was done so it would be as unpopular as possible. Of course, the whole "so what do we do instead" was left unanswered ...
And instead banks decided to lend the government money (QE, covid, ...) which is why governments are now in trouble. In the US, despite what you may have heard, just lending more to the government is quite possible in the US (because the government would still get substantial income for a substantial amount of time by lending more). However for a number of EU countries (reporting focuses on France but if you allow me to be 5% inaccurate let's just summarize that all EU countries except Germany cannot lend more as it would "snowball" in 2029 or 2030, which is within the next government mandate for those countries)
Yeah, I forget the wording but there’s a notification like “so-and-so’s AirTag is near you and they can see where you are.” My spouse put AirTags in our baby stuff and I get them whenever I’m using it.
It's also other devices. I used to get a notification that my wife's AirPods would be following me. It only got silenced when we formed an Apple family.
For anyone who hasn’t read it, I highly recommend the book “Ultraprocessed People.” It's a scientifically literate and yet quite readable overview of research into ultraprocessed foods.
FWIW there are four categories:
1 is whole foods (eg fruits, nuts)
2 is stuff that is processed but still basically an ingredient (eg flour, maple syrup)
3 is combinations of 1 and 2 (most restaurants are here)
4 is ultraprocessed food (UPF): things that were not food that have been turned into food (e.g. modified starches, artificial flavors). Also, anything containing UPFs is a UPF, so e.g. most breakfast cereal (food coloring, flavors)
The basic thesis (which this research challenges, interestingly) is that UPFs aren’t inherently bad, but food scientists are shockingly good at creating highly specific and desirable flavors and textures (the book gives the example of a food company creating "ice cream" out of various modified starches and flavors so that it doesn't melt during shipping), and foods that come out of this process have generally been engineered to overwhelm our natural feedback mechanisms against overeating in order to sell more product.
My reading of Ultraprocessed People was more that the author was very careful to only pin on UPF's what could be proved to be true. The between the lines read certainly felt like "you should probably not eat UPFs".
I would humbly add that any of Michael Pollan's books, especially "In Defense of Food" are a good answer to how we should eat if we can't eat McDonald's. I have certainly found restarting the dining table tradition to have incalculable benefits for my family.
> I have certainly found restarting the dining table tradition to have incalculable benefits for my family.
I would love to hear more of your experiences here if you would care to share? It is something I have been thinking about a lot recently (the slow erosion of this practice and its benefits that we're giving up)
We have a small child so eating at the table is almost a default for some meals. But we went from basically never eating together at home to deliberately sitting and talking while eating for breakfast and dinner. There's no rules around not having devices available and we don't force the conversation. I think being relaxed about it has helped make the practice stick. Mostly its just semi-guaranteed time to talk and do something together every day.
Most of the “chemicals” in so-called ultra-processed food aren’t added so customers overeat, they’re added to improve things like shelf-life or product quality (modified starch thickeners).
I’d find it hard to believe there are evil food scientists in big food companies who are nefariously designing food for overeating.
What is more likely is they are creating food that taste good, which shouldn’t surprise anyone. Now if the claim is they work towards this goal without much concern about things like sugar content or glycemic index, I could believe that.
But at the same time nobody thinks eating a whole bag of five servings of Doritos is a healthy food choice. If you’re designing good tasting food and assuming people will eat one serving as a part of a healthy diet, then no nefarious goals are required.
Food scientists are very much devising addictive food:
"The difference between a watermelon and something processed by a large company is that, for the latter, those foods are altered in ways to make them literally irresistible. This happens when companies use a combination of salt, sugar and fat to create foods that overstimulate the taste buds — and yet are designed to never quite leave you feeling satisfied. Even though saturating foods in salt, sugar and fat fuels the obesity epidemic — and does not necessarily provide consumers with the best culinary experience — it guarantees that customers will keep coming back for more by overstimulating one's feeling of taste pleasure in exactly the right ways. Since the companies' view their foremost responsibility as being to their shareholders, that is in itself a good enough reason to continue preparing foods with excessive salt, sugar and fat."
They came out with a new coke can for the first time in my life some years ago. Its smaller than the old cans, not bigger. How does that fit the thesis of these companies fattening us up?
GP missed the major point. Its not even about engineering tastier food. Its about shelf stable food that isn’t going to give you botulism in 3 weeks.
But food sciences are, other than maximizing the utility of food, also largely focused on finding the point where food tantalizes but doesn't satisfy so that at a very base level people who eat a processed food are physiologically induced to eat more of it without ever getting enough.
"The difference between a watermelon and something processed by a large company is that, for the latter, those foods are altered in ways to make them literally irresistible. This happens when companies use a combination of salt, sugar and fat to create foods that overstimulate the taste buds — and yet are designed to never quite leave you feeling satisfied. Even though saturating foods in salt, sugar and fat fuels the obesity epidemic — and does not necessarily provide consumers with the best culinary experience — it guarantees that customers will keep coming back for more by overstimulating one's feeling of taste pleasure in exactly the right ways. Since the companies' view their foremost responsibility as being to their shareholders, that is in itself a good enough reason to continue preparing foods with excessive salt, sugar and fat."
Lol the entire junk food industry would like to hire you for PR
You know cigarette companies have existed for 100 years and bought food companies right? You know their stuff is blatantly toxic and kills millions of people a year and they advocated ads and stuff and paid doctors to say it's not harmful right?
Like what world do you live in?
You really think hanlon's razor or any other thing like that remotely applies here?
For one, for achieving fat loss, it's surely simpler to focus on factors like calories & how full the food makes you. -- With that perspective, yeah it's obvious that fried chicken and sugary chocolate aren't great for your health when eaten frequently.
For another, e.g. it seems that nutritionally, something like "diet coke" is strictly better than the sugar drinks it replaces: you get the same taste, but zero calories. -- I mean, zero-calorie drink is clearly UPF but also nutritionally beneficial. I don't see "UPF" as a practical way to look at it.
The book discusses this a bit, actually. It basically credits the UPF theory to one Brazilian researcher, Carlos Monteiro, and discusses how he developed the theory by applying epidemiological techniques to the obesity epidemic in Brazil. It gives the specific example that, in Monteiro's research, if a family bought sugar at the grocery store, it actually reduced the likelihood that they were obese because, even though sugar is not healthy, it meant they were still cooking at home.
My understanding is that he would probably make similar predictions about your examples: people who eat fried chicken and chocolate as part of a mostly home-cooked diet are less likely to be obese than people who drink diet coke as part of a diet that mostly comes out of wrappers. He would probably also say that there's nothing inherently bad about diet coke, but there is something bad about getting all your food from people trying to make money instead of from people who love you.
I'd say "calorie dense food that's highly palatable makes it easy to consume more calories than you expend, and that makes you fat". -- I mean, that's a mechanical model explaining how you get fat.
I think that UPF mostly aligns with that in that the food that's "UPF" is generally calorie dense, highly palatable, low satiation.
But...
> He would probably also say that there's nothing inherently bad about diet coke, but there is something bad about getting all your food from people trying to make money instead of from people who love you.
At this point, it seems more a discussion of morals and social values rather than a scientific "how to lose fat".
> At this point, it seems more a discussion of morals and social values rather than a scientific "how to lose fat".
He uses language like that in the book (and I was speculating about “what he would say”).
I think the reason is that when people hear the theory “UPFs caused the obesity epidemic,” their brain sometimes goes to “well what is in UPFs that causes obesity?” His answer is “nothing.” He’s trying to emphasize that the issue isn’t some toxic ingredient, but that UPFs are a technology that are being used in a way that leads to this outcome.
(though the research in TFA might actually challenge the “nothing,” by showing that there is some kind of toxicity)
I thought that aside from the social context above, there is a theory that artificial sweeteners actually cause paradoxical effects in the metabolism because of triggers associated with the sweetness not the glucose/sucrose content. Ie specifically "diet" soda aside from eating out a Lot, may not be helping.
Zero calories doesn’t mean it’s healthy, as all those drinks are loaded with artificial sweeteners some of which have a strong insulin response and are also associated with cancer.
1. Sucralose increases peak insulin response when co-ingested with carbs:
"Sucralose Affects Glycemic and Hormonal Responses to an Oral Glucose Load" (Pepino et al., 2013, Diabetes Care)
https://pubmed.ncbi.nlm.nih.gov/23633524/
Finding: Consuming sucralose 10 minutes prior to an oral glucose tolerance test increased peak plasma insulin levels by ~20% and decreased insulin clearance compared to water.
2. Microbiome-driven glycemic & insulin dysregulation:
"Personalized microbiome-driven effects of non-nutritive sweeteners on human glucose tolerance" (Suez et al., 2022, Cell)
https://pubmed.ncbi.nlm.nih.gov/35987213/
Finding: A randomized trial showing saccharin and sucralose alter gut microbiota composition, leading to personalized impairments in glucose tolerance and postprandial insulin responses.
3. Reduced insulin sensitivity from daily intake:
"Sucralose decreases insulin sensitivity in healthy subjects: a randomized controlled trial" (Romo-Romo et al., 2018, AJCN)
https://pubmed.ncbi.nlm.nih.gov/30535090/
Finding: 14 days of daily sucralose intake significantly decreased insulin sensitivity in healthy young adults compared to placebo.
4. Direct pancreatic beta-cell mechanism:
"Sweet Taste Receptor Expressed in Pancreatic β-Cells Activates the Calcium and Cyclic AMP Signaling Systems and Stimulates Insulin Secretion" (Nakagawa et al., 2009, PLoS ONE)
https://pubmed.ncbi.nlm.nih.gov/19352508/
Finding: Sweet-taste receptors (T1R2/T1R3) expressed on pancreatic beta-cells bind sweeteners (sucralose, Ace-K, saccharin), mobilizing intracellular cAMP and Ca2+ to stimulate insulin secretion.
Zero-calorie drinks like coke zero provide a no-calorie alternative to the full sugar drinks.
If you habitually drink 3 cans of coke a day, and want to lose weight, then switching to coke zero is practically the same benefit, but consumes 430kcal less. Since coke zero is like a zero sugar replacement for typical coke, it makes sense to use "compared to coke" as the baseline.
That's what I meant by "clearly .. nutritionally beneficial". If you'd prefer a different term, then sure.
(I think the proposed alternative of "just eat whole foods and water" is surely going to be even more beneficial, but also requires a change in habits. If "change habits" was easy, I don't think there'd be any discussion around "Ultra Processed Food")
>foods that come out of this process have generally been engineered to overwhelm our natural feedback mechanisms against overeating in order to sell more product.
That's the point at which it becomes highly speculative.
It seems to invoke an assumed knowledge of human biology that either does not exist or is known by some select few in some form of conspiracy (somewhat unlikely)
It reminds me of raw foodists that argue that cooking is bad because humans harnessed fire only after we'd mostly evolved. Or a magnet bracelet peddler claiming that since we started wearing shoes, that separation from the earth's magnetic field means we need the bracelets now.
> That's the point at which it becomes highly speculative.
The “…in order to sell more product” part is scientifically speculative, I agree—the book makes the claim, but the scientific research it references does not, IIRC. The book’s author imputes it from interviews and the like.
The “overwhelm…feedback mechanisms” part is mainly substantiated by a 2019 study by Kevin Hall where, similar to TFA, they had people essentially live inside a calorimeter and eat calorie- and nutrient-matched meals that were UPF or not. They found that when given UPF, the same people ate more and faster, and gained weight at a pace that surprised the study authors (who were initially skeptical of the theory and had constructed the study to try and disprove it. The study author is interviewed and explains why he found it compelling).
Whether you find the evidence compelling is, of course, up to you :-), but I felt the book did a good job interviewing members of the obesity research community and tracing the history of the theory.
It didn’t take an advanced understanding of reward pathways to realize that things like nicotine and cocaine were highly addictive and that aspect could be concentrated (e.g. crack).
I wrote a blog post making some very similar arguments earlier this week in response to some mathematicians’ takes—just before the OpenAI paper drop a few days ago (https://creating.software/essays/ai-and-math/).
I feel we’re all realizing, at the same time, that (1) intellectual activity is communal, and yet (2) those communities are sustained by recognition of individual achievement. What a way to find this out.
I claim (here and in my post) that humanity actually can’t advance without these human communities. It’s critical that people not give up learning things that AI knows better than they do, because they play an important role in these communities, and the community is critical.
My (maybe naive) hope is still that people find ways to keep doing this stuff for fun, like chess (competitions, awards, coaches, I don’t know). We still have chess competitions and the like, even though computers are far better at chess and have been for a while. The community and the recognition of individual achievement are both there. And human chess and chess engines have gotten vastly better, so computer dominance failed to kill progress in chess (today’s stockfish would’ve beaten Kasparov in 1997 running on a desktop from 1994: https://www.lesswrong.com/posts/4MLBK7iCW3vYd93Mn/a-closer-l...)
Maybe math olympiads and programming contests live on well past the end of their practical utility—I always liked them.
My understanding was that they take issue with the word "attributable". They don't attempt to propose a mechanism by which living near Costco causes cancer (the most strongly associated of the landmarks they measured). Rather, the implication is that it doesn't. Like I think it's a lot of words arguing that "correlation doesn't prove causation" could be restated "association doesn't prove attribution."
Heterodox argument: a big part of the reason JavaScript has been so successful is that interpreted languages actually can be extremely fast—competitive with, and in many cases, faster than compiled languages, with V8 arguably the fastest interpreter in existence.
See this post by Mike Pall, author of LuaJIT (which is comparable to or faster than V8 performance-wise, despite being basically a single-developer project), which explains why: https://web.archive.org/web/20180603053407/http://article.gm.... Basically, it’s much easier to add high-quality runtime-trace-aware recompilation to a JIT interpreter, which a non-trace-aware compiled language will often not be able to beat.
Javascript only becomes fast because most of the CPU time is spent in native code rather than actual compiled or interpreted javascript.
The big diff between scripted and compiled languages is the former dispatches by name and the later dispatches by index. No surprises which is faster and which can be compiled in the faster native code.
JavaScript only becomes fast because it continually is making tradeoffs to decide what level of work to put into running any given bit of code. For a while this was three or four different targets on V8 (which I think maybe is down to just 2 now again).
Compiled vs interpretted is viewing this through the compiled language lens. I think the interesting part is that some runtimes are dynamically deciding how to run the code. The contrast versus those folks who decide and bake every single decision in ahead of time, where the only thing that happens is totally pre-scripted, is strong.
Stop Writing Dead Programs is such a lovely talk that hopefully can shake a couple of these sleepers awake, get us to see how absurd compiled code really is. And is quite fun and funnily said. https://news.ycombinator.com/item?id=33270235
It's mostly fast enough for most apps and that definitely helps. It's not as consistently fast as some languages that don't rely on an JIT. That matters more for tooling than your average website.
There's no need to even contextualize your argument, it is the reality. The whole point of spending such heroic efforts on a fast VM implementation is so that even faster peformance becomes a niche use cases.
But alas, many unskilled programmers use Javascript to make useful things - sometimes beyond their ability. Even a state-of-the-art virtual machine like V8 cannot hope to fix that class of peformance problems.
I will withhold my opinion on the "faster than compiled languages" part though. ;)
> I'm glad you're having fun vibecoding, and I like that you're interested in this area of research/engineering
These impute both his motives ("fun") and particularly his level of seriousness ("random project" and "I like that you're interested"—imputing passivity, as opposed to "are studying" or "are researching," which would be more appropriate given the amount of time invested). They're all dismissive and patronizing.
I would actually regard this as bullying. Some feedback.
(I suspect you're an academic, either a researcher or student. I know from my own experience that bullying is endemic in many academic research environments, so if you find the negativity you're receiving "strange," I suggest finding a therapist, who may help you understand how your communication habits could be negatively affecting other people and unintentionally damaging your relationships.)
In my culture it was ad personam, not even trying, and it’s fantastic that it’s been pointed out in a respectful way and reacted to with calm by all participants.
I don't think stschaef is a bully; in fact, if my guess that he's a researcher is correct, I think he's likely highly altruistic (I've never met him, but categorically, researchers are people who chose a difficult, low-paying job doing work of great societal value).
However (again if my guess is right), I think he could easily be in an environment where narratives about people's work being worthless, about them being stupid or unserious or otherwise beneath consideration, are common. It's a reaction to the fact that in any field, the amount of research produced is overwhelmingly more than anyone can digest. There's a lot of unstated anxiety and guilt about that, on the side of both writers (who worry no one will read their research) and readers (who feel obligated to try to read everything and are eventually, inevitably overwhelmed), which IMO is itself a product of the basically altruistic nature of most researchers.
The overwhelming reality of being a researcher is an inescapable, empirical fact, but the narratives people create around that reality, about peoples' work and its worth, are not. People are hard-wired to be sensitive to rejection, because humans are a cooperative species and social acceptance is existential to each of us, and the problem is that a lot of researchers, who are steeped in these narratives, are trapped in a self-reinforcing cycle of community attachment threat: their research sucks (or could start to suck if they ever went on vacation), and the research of most of the people who are evaluating them sucks too.
What if these stories about the worth of people and their work aren't true? Guess what: one can do amazing research and it still goes nowhere, because ultimately it's not possible to control other peoples' behavior. If one's goal in research is acceptance and respect from the community, they should consider that they're gambling their time, energy and youth on an outcome they can't control. The research community can be a fine place, but it's not special—one does not have to be a researcher, and if the experience of being a researcher sucks for them, they should quit, because living a good life is their responsibility.
I grew up around this attitude, and children are particularly sensitive to attachment threat. It's bad enough that researchers tell these stories about each other, but once these narratives and communication patterns about peoples' work and its worth leak outside the context of the research community, they run right into the reality of human attachment and the expectations of communities that aren't the research community. I'll say about my own family: I think they were good people who learned an unhealthy, judgmental attitude (towards themselves as well as others). I think they immiserated themselves (and inevitably the people around them), because perpetual attachment threat had traumatized them into false sense of obligation, and I wish they had quit.
Even if one stays, they should understand the mismatch between the narratives and culture of the research community, and they needs and expectations of most people outside of it. That was the context of my comment. I think it's fine if stschaef doesn't like TFA and doesn't find Bend novel or interesting, but my analysis of the sentences quoted reflected my organic reaction to them, and I stand behind it and my other feedback.
I haven’t played The Store is Closed, but to everyone saying IKEA had some obligation to shut down a parody…videogames parody real brands all the time, including their colors? Why was this different?
Some examples offhand:
- Fallout’s Nuka Cola is an obvious parody of Coca Cola (also same colors: https://en.wikipedia.org/wiki/Nuka-Cola) and Cram is an obvious parody of Spam (same colors)
- GTA is full of these: Sprunk (an obvious parody of Sprite, again same colors), Fruit (obviously Apple—same colors, same branding style, same use of iPrefix and “think” marketing), definitely others
- Saints Row “Freckle Bitches” was a fairly obvious reference to Whendy’s (“I’m all grown up”)
All of these games were bigger, not Indy games at all, and none of the parodied brands saw any obligation to sue AFAIK. Would IKEA have gone after Rockstar, given that they’re much bigger and very popular? I’m suspicious that this was a case of “small target” rather than “big problem”.
(Ok, I clicked through the article, it looks like it was just a demand letter? My impression is that demand letters are nearly meaningless. No judge has to approve them, and they cost an hour or two of lawyer time.)
> it looks like it was just a demand letter? My impression is that demand letters are nearly meaningless. No judge has to approve them, and they cost an hour or two of lawyer time.
A solo indie developer who is busy building their debut title is no match for the legal team of a global multi-billion corporation. The developer would run out of money way before the parties could reach any kind of settlement.
- we're "Bazel for data"
- but...we have no Starlark. We're trying to port Airflow but it's not playing well with the way we cache partial results.
Amazing to me that I'm only now seeing the architecture pattern of "pure coordination language connecting sandboxed imperative components", which is used by Nix and Bazel, applied to distributed workflows. If somebody applied this to batch data processing you might have a Pachyderm that actually works better than ours did.
reply