Curl UI wrappers are just the weirdest thing. They’re so popular, they offer so little (compared to other paid services), and they all have the same evolution of simple -> require account -> sell data.
They just all seem like things that developers would easily ditch. But up until now, I was also still stuck on postman because my team/org uses it as a knowledge repository. It’s just classic enshittification. They’re using high switching costs to make as much money as possible without everybody leaving. Who
Yep I’m finally switching off of postman today. I’ve wanted to for a while, but my team at work uses it, and all of our shared stuff is in there.
I’ve been thinking of replacing postman with a clojure project, where you just send stuff to the REPL to call it. You could even set up a little dependency map, so your calls would have built-in behavior like “if the token is expired, grab a new one with user/pass”. And it would be trivial to script endpoint calls together.
If anybody has done something similar, I’d love some tips or ideas
Right! I personally think that characterization is crazy, but it's out there. This twitter thread started me down the rabbit hole, and, maddeningly, it's very coy and vague.
What do people think of property based testing? I’ve set up a few property based tests and it feels almost like formal verification.
I know it’s not the same thing, but PBT’s can catch all sorts of absolutely crazy bugs that I never would think to check for. Stateful stuff, cross-service stuff. My only gripe with PBTs has always been the tedious setup, but LLMs solve that.
I haven’t seen much discussion around PBTs lately, but you get a ton of bang for your buck in the LLM age
PBT testing and fuzzing are basically the same thing in my mind.
What's funny though is that the reasoning needed to say some sampling is comprehensive....makes it closer fo proofs! AFL — coverage-based fuzzing — was the original innovator here, but imagine an interpreter — code coverage in the interpreter itself isn't so good, you also want to explore the interpreted program!
This is why "is my testing comprehensive?" is actually an open-ended problem that, in the limit, is just as hard as formal verification.
I suspect we're basically going to see a convergence here. Randomized testing can be called "Monte Carlo formal verification" hehe.
> This is why "is my testing comprehensive?" is actually an open-ended problem that, in the limit, is just as hard as formal verification.
Well said! I learned about PBT in the first place because I watched a talk where the speaker referenced the story of foundationDB. Super interesting stuff. If anybody wants a link I can dig it up
But they are very different IMO at what they focus on and so they occupy different places when talking about does a piece of software does what it should do and withing the state/parameters specified:
- Property based testing focuses on finding counter examples
- Formal verification focuses to prove that a property always holds
More than this I think in all these discussions we are missing the most important piece: validation.
While verification means to prove or test or verify that the software does what is should do, validation means to prove or test or verify that the software is solving user needs.
I think there's some bad nomenclature here. "Property based testing" literally means you're writing properties, which can be verified formally, or randomly (fuzzing / constrained random testing).
For example in SystemVerilog you can write properties (literally with the `property` keyword), and then they can be checked in simulation, and with formal tools.
TLA+ can do the same I believe - there are tools to verify a model randomly or formally.
works really well for simulators, where you usually don't have an oracle but you do have invariants. e.g. in SPICE the low-freq AC gain has to match the slope of the DC sweep. it's cheap to check and it catches sign errors that look totally fine on a plot
So actually I've been testing this — it seems to be useful in limited cases. LLMs have gotten so good now that a properly defined spec is no worse than a spec + property based testing.
I still found it useful for
1. trusting a bit of code is correct
2. rule-dense bits of code (eg something like a date parser)
The real issue so far is that I'm having trouble finding tasks where an LLM can't correctly code up an exhaustively defined spec.
I was 10 when it came out. I remember the hype being so crazy. It’s all anybody talked about at school. Wii was going to bring us into the future and make all of our dreams come true.
It was totally awesome. But the advertised promise of motion controls didn’t really land until Wii motion+ controllers. When I played skyward sword I thought “ok THIS looks like the ads”
My kids' mother (who lived overseas) won a contest. She didn't win a Wii, she one the opportunity to buy a Wii. She sent it to them for Christmas, and twenty years later I don't think I have ever seen them so excited.
For about a year, we played Rockband (especially the Beatles version) for an hour every day. We had the Rockband 3 keyboard, and one of the kids decided that he'd like to learn the piano. Twenty years later and a B.Mus. and M.Mus. later, we just attended the premiere of his musical (not, not a big one, but still...) in Toronto.
His career literally wouldn't have happened without the Wii.
They just all seem like things that developers would easily ditch. But up until now, I was also still stuck on postman because my team/org uses it as a knowledge repository. It’s just classic enshittification. They’re using high switching costs to make as much money as possible without everybody leaving. Who
reply