Having written an open source Fortran library (for x ray scattering and absorption in a particular regime so this is fairly “niche” shall we say), genuinely curious: what if any advantages or strengths does Fortran have these days over other language choices? Are there circumstances where it’s arguably the “right” choice?
A lot of the early, absolute advantages of Fortran (numerical precision, non-aliasing) has been adopted by C.
However what still makes it a great choice is that it still a relatively simple language compared to C, and maps mathematical formulas to code very well. Inelegant code will still run reasonably fast (which is not the true with C) - so all of these points make it actually a good language when the program is being written by domain experts (i.e., scientists) rather than professional programmers or software engineers.
As @jleyank pointed out, the more restrictive semantics of Fortran prevent expressing things the compiler can't optimize. A Fortran compiler can make assumptions about code flow and memory accesses a C compiler can't, and, therefore, can optimize the code further than a naive C implementation could be optimized.
Basically you don't need to think much about the computer itself, unlike C or C++. Just pass in your arrays or pointers (everything is passed by reference anyways), crunch numbers, and it'll be about as optimized as carefully __restrict__ laden C code.
There's also fewer traps to fall in than C++ which is elephant sized in scope to Fortran's mouse.
May I ask: why did you write that library in Fortran? That is, what was your answer to your question at the time you wrote it?
I have heard that Fortran is very very good at optimizing array processing due to lack of aliases. That may be one answer today. There’s also value in continuity: in using a language for your task because it can build on what others have built. That applies to any established language, of course… but I could see it reasonably applied if someone wrote some kind of numerical processing library today. They’d write it in Fortran and call it from Python!
The answer was “my adviser insisted because that’s the language he used”. I had to fight to use Fortran 95 and not 77. So I think like you say value was building on an existing community’s codebase in a language they were familiar with and comfortable with.
Source code is meant to be read, and, occasionally, compiled.
If nobody with domain knowledge can understand the program because it's written in a language unfamiliar to them (such as C) with tons of traps and pitfalls (such as C), the value of its contribution to the corpus of knowledge is diminished.
I agree with this and think this is the only argument I’m aware of for real decisions to use of Fortran, which I suppose means its future is doomed with human turnover + ease of porting in these AI days…
Do you realize F95 is over 30 years old? The latest standard is 2023. When people talk about modern Fortran, they usually mean Fortran 2008, 2018, or 2023, or, at worst, Fortran 2003.
I mean, I think it's best in class in its niche: writing numerical algorithms that compile to highly optimized machine code, while being reasonably readable and intuitive (at least from F90 on). It's arguably still the easiest way to write compute-intensive numerical extensions for python (well, if it's single-threaded, and if numpy can't do it, and if you're not using GPUs ...).
I think they mean improve our understanding of physics with new theoretical results or paradigms. Like if it’s 1899, would Astra develop General and Special relativity on its own?
This might just not be possible at the current time. In 1899, there was an "experimental overhang" in physics -- results that could not be explained theoretically (Michelson-Morley, but also lots and lots of empirical material/spectroscopic properties that we could today calculate using quantum mechanics). The big problem in theoretical physics today is that unifying general relativity and quantum theory has no experimental results you could get at our technological level.
I think you make a fair point, but also remember: new paradigms don't necessarily require confusing / contradictory observations. You could have the simple idea of "what if gravity is an inertial force?" at any period in time and work out the mathematics of this. It would make theoretical predictions which could then be falsified, but then again, who would take it seriously enough to test it if it was maybe say 1850 and not 1899.
A better example is maybe Maxwell's laws. Maxwell wasn't inventing a theory to try and explain confusing results, he was unifying a chaotic, empirical laws from existing experiments. That may be a cleaner example. That knowledge compression into satisfying theoretical framework is likely what is attractive.
You can potentially ask the same thing about like you say -- general relativity / quantum gravity but also likely plenty of other areas that may be like this today. Again going outside my particular area of expertise: standard model physics is in a large important sense empirical; lots of values and numbers that are simply unmotivated by theory or where we don't have a good way to make a principled theoretical choice. That could be a place where these models are able to help.
But right now: I doubt it. This is what everyone is working furiously on right now. How do you close a "science" verification loop? In principle this should be easy right: you have ideation (exploration, sampling with ~high temperature maybe as an analogue) and you have verification (which of these ideas are good) which amounts to rejection sampling in idea space. You have to have a sampler that is good at picking _good_ ideas for efficiency sake and you need a relatively fast and reliable verification step of "is this idea good and worth continuing to explore". But I may oversimplify
This is as good a time as any to note that we might be closing in on a new conceptual revolution in our own time as it relates to holography and an information centric approach to spacetime. Obviously It's the furthest possible thing from a guarantee, but it has much of the enthusiasm and motivation that string theory had previously enjoyed in prior decades.
So it could be a natural experiment for whether AI can contribute to novel physics. Specifically, there's a big question about weather. Something like our informational understanding of black holes where information inside it is equivalent to information on its boundary (which I'm sure I'm not saying correctly), might be generalized to regular space-time. More people should be freaking out with excitement about this and perhaps it's something to which AI can contribute.
Honestly I don't think I have single great article, though some Quanta ones are ok, and the Wikipedia article is okay.
The best thing I can recommend is what I did, which is ask Claude about the significance of (1) quantum computing error correction, and (2) error correction in black hole holography and research convergence between the two.
I have no idea what you're hoping the contribution would be. The AdS/CFT correspondence is 29 years old by now and it doesn't seem to apply to our spacetime, where the cosmological constant seems to be positive rather than negative. There are some puzzling consequences of the holographic principle in that scenario as well (https://arxiv.org/pdf/hep-th/0208013), but the linked articles don't talk about them?
Quanta articles are written for people with no background whatsoever, which makes them impenetrable if you have a bit of background and are trying to figure out what they're about. I don't know how good Claude is compared to that -- whenever I try asking any LLM about something I don't understand, it produces a wall of text, I have no idea whether it's correct or relevant, and I look for a textbook or review paper instead.
The wiki article has a section on 'Energy, matter, and information equivalence', the first Quanta article is almost entirely about the 'deep connection between quantum error correction and the nature of space, time and gravity' and about bringing the same information centric approach from AdS to our spacetime. The second Quanta article is explicitly about about bringing a holographic approach to our non AdS spacetime and cites an Ed Witten paper as the cornerstone of that approach (which one perhaps overly excited MIT physicist describes as 'revolutionary').
AdS is 'old' but the articles aren't suggesting it is new, and our spacetime is not AdS and the articles don't suggest otherwise. The point is that there's a search for a way to fit the holographic approach to our spacetime that's inspired by how AdS helps make sense of black holes. Quanta writing being directed at a lay audience ought to be a good thing, not a bad thing and they do link to papers if that's your jam.
Whether or not your LLM of choice produces indecipherable walls of text, and whether it ties those to sufficiently satisfying citations, I think is just a matter of how you go about the prompting.
You're right, I'm prejudiced against Quanta (often IMO they look for a clean narrative to the point of misleading and/or rely too much on metaphors) and was probably too harsh here. Sorry!
That said, I don't like them because their articles never leave me feeling like I understood something. They never go in an order of simple to complex and constantly try to hook you. (A positive example to contrast would be 3blue1brown, who manages to both hook you and make you understand, even with rather little background.)
Could you share a link to your Claude conversation? If this is a prompting issue, I would be interested in seeing what's possible. Thanks!
There was symbolic AI programs in the 1980’s that “discovered” Kepler’s laws and the resulting solar system model from just tycho brache’s astronomical observations. That was the the very first “new physics” ever.
Do you mean after? People do this!! But I think it’s a bit different. It won’t be apples to apples because the data volume I think is just so much different. Maybe there are good experiments for something like this.
My job has gone from coding, plotting, writing to solving the riddle of what Opus 5 is saying and figuring out what is bullshit, what is valuable, what is a total divergence from what I asked it to do. Then eventually at token like 300k starting to shout at it in all caps with obscenities.
Things were slower and harder before but the baseline of frustration / rage was never this high, even as the models have gotten objectively better in many or most respects.
I don't think it's a stall, two ways I would believe there is a stall:
- Does the epoch capability index progress show signs of plateauing? I consider this a good aggregate measure of diverse benchmarks into a single capability index. If we see things slowing down here thats a pretty direct and convincing piece of evidence for a stall.
- Do we see any signs that scaling laws are beginning to fail? That would be by far the most alarming to me, since I would interpret that to mean that the entire premise of this unprecedented capital allocation tsunami is broken.
Neither of these are true (for now). Progress is marching the same as it has for 4+ years now. It's still the same time to get a generation leap (I think like ~16-18 mo? Epoch has it) like GPT4->5. My theory is that people interpret plateauing because the releases are far more frequent now than they were in the past.
I’ve noticed this too but it hasn’t been obvious to me that this style of search is not a learned behavior. Tool calling is very much part of the post training phase, I would expect that these style searches just naturally emerge during training. This is just my prior though.
Im curious if you find this to be a parody in a bad way or simply a “the state of the art in math research right now is telling a machine to believe in itself”. I am in the latter camp…
The former, because it's anthropomorphizing a model.
Anthropic is especially guilty of this. They have been using such language for a while, like when they analyze model weights for mechanistic interpretability and call it the model's "biology".
> The former, because it's anthropomorphizing a model.
Not really. The input and output is already natural language. That is already "anthropomorphizing".
That is, if this is the bar for anthropomorphization its already happened.
Telling the model to "believe in itself" is just stochastic manipulation that has shown enough reliability to be a recipe to make it keep going.
It's only actually anthropomorphizing if you forget it's a trick and think it's a real person.
There is nothing distasteful about it. If people get confused that's on them. They wouldn't be very useful if you couldn't just talk to them. That's kind of the whole point. Otherwise you can just go back to coding by hand. Telling it to believe itself is just input that happens to work. This probably tells us more about human nature than you realize given the corpus on which it is trained. It obviously doesn't mean anyone actually thinks it's a person.
It's obvious that you don't get it but I will try my best to explain why so at least you can form an idea about how others feel.
It's about what makes humans unique. The LLM does not experience reality, it just merely pretends it does, and even that, it does in a shitty way. I think disgusting is a very adequate adjective. The reason why it is disgusting is because you are devaluing a divine experience to the realm of the common and the vulgar, a cheap substitute being valued as equal (or even on the same scale) as the most important experience we could go through.
To give you an example that might land in a more familiar context, think of that one guy who takes his plastic doll everywhere and pretends it's his wife and gets upset when others don't acknowledge "her" as a person.
Who said that it did? The comment you're replying to literally states "It's only actually anthropomorphizing if you forget it's a trick and think it's a real person".
Telling it to believe in itself is no more pretending than telling it anything else in natural language. Why are you speaking to it at all if it's not a person? Why write in higher level languages even? It's just a machine let's all go back and code in 1s and 0s.
No one is calling it a person except mental health patients and straw man detractors.
The biology example was even weaker. Saying it has a "biology" is about as distasteful as the term "neural net" or calling an input device a "mouse". Is it animal abuse to click on something all day? Language is inherently anthropomorphizing.
No one is calling it human. The fact you are so easily threatened is far more suggestive of your own poverty of understanding of not only the machine, but yourself. If humans are so special the threat posed by this should be self evidently non existent.
Is it though? There's a perfectly "technical" reason why this strategy should work, without any sort of anthropomorphising:
Assume models are trained on vast amounts of data. Assume that the model is asked to solve something that the literature says it's impossible. It will start generating tokens towards that "this is a famous conjecture, it's not possible to prove it, blah blah". Assume the model was also trained on books/novels/etc. Assume the model was also also trained on "solving" many math problems. Now, you can make an argument that just placing "you can do it" in the context will "steer" the model towards generating "moving forward" tokens. Take ideas, generate tokens, go towards negative. "You can do it". Model starts generating tokens again, more ideas, more "exploration". More negativity. "I believe in you keep going". The two (book tropes + math CoT) mix together in the context. The model keeps on "pushing" and "vibing" between the two. Ta dah, it works.
For me, personally, it's that the Bun guy - specifically him, not a mathematician - indirectly progressed the Riemann hypothesis by repeatedly telling a model to ganbatte!
But the words don't really matter, do they? The model thought "user said believe in yourself, it means they want me to continue"..
AI did a fixed amount of guesses, didn't yield anything. It probably documented the tries, outcome, and some numbers hinting at why they failed. So the user could have prompted "continue", or "try again with previous outcome in mind, generate new ideas and test them" and it would probably yield the same result.
As a data scientist it is very frustrating to see a consistent lack of error bars on these numbers or any discussion about the magnitude of uncertainty around them by all news reports every time these numbers get released
This should have nothing to do with data science but basic math literacy. Quantities without context are meaningless. Reporters should know better but if when headlines have to catchy day in day out … shrugs. Stock market reporters are the worst: a 1000 point drop in the Dow meant a lot more when it was 7000 than it does at 40,000.
If honest, the error bars would be so large as to make drawing conclusions from these numbers seem pointless. Certainly, both jobs growth and jobs loss are within the CI.
What's actually being tracked by people using this data is the delta from previous years -- which, conditioning on the same collection methodology etc -- is more reliable.
Something the best data scientist I ever worked with was adamant about... there is almost never an inherent "speed bump" at 0. If error bounds expand on both sides of 0, we have no idea if the true sign is positive or negative. There is no "directional" read in that case. There are some areas where 0 has a special meaning, but that's rare and the cause of that needs to be understood before assuming that's the case. Everywhere else, 0 is just another point on the number line.
> the error bars would be so large as to make drawing conclusions from these numbers seem pointless
That is often the case, and being truly data based requires acknowledging that. We can then act based on other data, or admit that we're falling back to intuition and abandoning data based decision making for now.
As a physicist, I always shed a tear when looking at data without error bars.
As a data scientist/generic programmer now, I don't really see them anywhere. I remember a colleague during my postdoc saying 'don't worry, no error bars in industry' when I asked about regression with ML lol
Our fake economy needs even more fake certainty now. They are all hiding from the obvious issue of stagflation because it is the most difficult economic problem to solve. The Fed know this which is why they did nothing with rates.
Not in their primary news release. A footnote says it is "generally conducted at the 90% interval", but the individual numbers do not have their actual computed error bars.
Yes, they do publish them, but it is disappointing their primary releases do not have them.
It’s common in scientific/stats publishing to have the headline state the avg/median, the main text plot the errors, and the supplementary give the explicit numbers. They have a big page about std errors.
Yes! I am aware of this and they are really great. Problem is nobody will ever read that, but they WILL watch Fox News, CNBC, etc. and those outlets never even point out that there are errorbars/uncertainty. Then I hear things like outrage when BLS revises numbers up or down (facepalm).
The numbers are as good as they can get with hard manual work that goes into compiling them.
The problem is that small businesses lag by 1-2 months reporting them, so the initial number being reported is almost always irrelevant, as it mostly applies to numbers coming out of larger corporations.
I would say the initial numbers should be embargoed until the number is reliable. People don’t understand error bars and reporting can more easily manipulate the current situation without outright lying.
Error bars would be misleading, because this isn't a sampling error. These start with actual counts. The more important errors are systematic: the definitions of terms, the reporting mechanisms, etc.
Except that those things are constant from month to month, so deltas largely cancel them out.
The problem isn't in the size of the possible errors, but in understanding what those numbers mean and how they are gathered. That requires far more background reading.
That’s what I meant. There should be error bars to reflect uncertainty of the inherent data, but they should wait until all the data that could change things comes in rather than using projections which always drastically change
What decisions are so important and time sensitive they depend on these numbers right after a month is over? And also we at one point were making decisions without these numbers yet the quality of the decision making isn’t clearly improved.
A few months of extra no data won’t impact the stock investments if no one has that data. Startup funding is similarly not so time sensitive.
As for businesses hiring, they should hire based on their own growth. Using an external signal that itself gets adjusted drastically is similarly problematic; I don’t hire because I think no one else is hiring and the numbers get majorly adjusted up. Similarly I hire a bunch of people and it turns out no one else hired.
No one benefits from faster numbers that are majorly shifted in a few months anyway. You wouldn’t make any different decision if no one had the numbers. The only thing you’re doing differently is when the numbers come out your gambling whether or not you think the true number will be revised in some major way in a few months. That’s not helpful for the economy writ large.
> If the error bars are large, that is important information to know.
The data is available. If it were important to the causal reader, however, then those casual readers would have already demanded its presence. But they don't care. What difference does it make to Hilary Homemaker? She just wants to get to say "Did you hear about the bad economic news?" when gossiping with her friends.
Those who need to understand the uncertainty will already be looking at more than a news headline anyway.
Ah yes, lie to them and withold information because the public is too stupid to understand. Then, when they don't trust you anymore, throw up your hands and say it's not your fault.
That's true for the establishment survey, but the population survey, where they contact individuals, is what is actually used to calculate these numbers.
It is literally calling/writing 60,000 people and asking "are you employed? did you look for a job? Etc."
"How the government calculates unemployment" [0]
| Does anyone in this household have a business or a farm?
| Last week, did you do any work for (either) pay (or profit)?
| If the answer to question 1 is "yes" and the answer to question 2 is "no," the next question is:
| Last week, did you do any unpaid work in the family business or farm?
| For those who reply "no" to both questions 2 and 3, the next key questions used to determine employment status are:
| Last week, (in addition to the business) did you have a job, either full or part time? Include any job from which you were temporarily absent.
| Last week, were you on layoff from a job?
| What was the main reason you were absent from work last week?
| For those who respond "yes" to question 5 about being on layoff, the following questions are asked:
You have to wonder, since they are doing two different surveys trying to measure the same reality, which one more accurately measures the facts on the ground? Asking businesses how many people they are employing, or asking households whether or not their folks are working? The numbers don't really match up between the two.
GDP too. That's how you can get more accurate information, and you have actual numbers on the discrepencies.
They rely on the household data more because it doesn't have as much systemic error, even though it does have more sampling error. Asking businesses gives you a more complete picture, but it has much more bias.
Hell, this is why even in accounting, you have double-entry bookkeeping, and even there, the numbers don't always match up.
Or "my bank account doesn't match up with my cashflow statement".
It's not that one is better and you should use that, calculating multiple ways is central to accounting so you can detect errors.
What makes you think it's small businesses? I would assume most businesses doing any reporting at all are using a payroll system. Mine handles the reporting automatically. The kinds of payroll systems used by small businesses can be set up in a day. If anything, small businesses that make even a small effort to stay on top of things have an easier time than big ones with complicated systems and processes.
I think you say “as good as they can get with [the] hard manual work that goes into compiling them” as a way to respect the challenge of the BLS and say they’re doing the best with their resources, but it’s worth pointing out they could be better with more resources.
DOGE hurt them pretty hard: "The Bureau of Labor Statistics posted a notice on Wednesday stating that it stopped collecting data in three not-so-small cities (Lincoln, Nebraska; Buffalo, New York; and Provo, Utah) and increased “imputations” for certain items"
Oh I’m well aware the BLS knows what they’re doing and publish error bars, they are great. The problem is news articles, no one ever reads BLS reports but they do watch Fox News, CNBC, etc. and those outlets never ever get into uncertainty.
The actual source this comes from has a 90% confidence interval in the published number. When people talk about the economy gaining or losing jobs they are talking about the bureau of labor statistics "nonfarm payroll" number. The methodology and a lot more are on the bls website
Exactly. I'm somewhat surprised someone's give-a-damn wasn't broken enough to do a cursory filter. Not like it isn't plainly obvious in other statistics, however.
I agree and that’s why we need and indeed have an ever evolving landscape of benchmarks
> Private, refreshed test sets attack the mechanism itself, and in my view they are the only intervention that does. If the questions have never touched the public Web, they can’t be in the training data; if they rotate, memorizing this year’s set doesn’t help next year.
That’s what we have. A fresh public benchmark is also good, and teams do make efforts to decontaminate training data but there’s likely just no great way around leakage.
Btw, lots more issues in benchmarks than the ones discussed; for instance you can leak answers from the questions themselves or in the case of e.g. multiple choice formats in the actual answers. You just pass the MCQ choices themselves to the model and it may be able to guess way above chance. Coding agent benchmarks sometimes forget to delete .git. They mention e.g. a 6.9% error rate in one of the benchmark items, this seems pretty typical and I would actually be fine shipping that.
Benchmarks are very ugly, but if they didn’t exist we would need to invent them. All of the problems above and more do not explain the progress we see. There are probably 50,000 benchmarks in the literature and new ones get created frequently with varying levels of quality and usefulness.
I don’t think that’s fair to say, there are two principle sources of data that paint a fairly consistent picture
- one is scaling laws, where we found years ago that pretraining validation loss scales in an almost miraculously predictable way with data volume and compute. There are apparently theoretical bases for this that I don’t quite understand but this property alone is holding at every scale we’ve ever tested. There is not just “one” scaling law but the point is there are scaling laws and they continue to faithfully predict the performance gains we see
- one is benchmarks, which I always point to epoch capability index as a good summary of them in aggregate which makes it nice to plot on one curve the capability improvement over time
To me either one without the other is substantially weaker, the fact that theory and empirical measurements give you a very good scaling law on a more unintuitive quantity (pretraining validation loss) that’s only indirectly related to the downstream performance you care about, benchmarks (in aggregate) are more direct measures of downstream performance but are harder to nail down clean and well motivated “laws” from theory (as far as I can tell). Nevertheless we do in fact see a clear trend that is not slowing.
That doesn’t mean there aren’t a whole host of benchmark problems that don’t impact the numbers involved here (leakage from training data, fundamental flaws in the design, benchmaxxing) but they don’t change the larger story. These problems don’t plausibly explain the clean trends we see.
