It's hard to try and compare these benchmarks against real world use cases. As an experiment I let Meta Muse make all major decisions on a weekend trip to Seattle ~2 weeks ago. I think this would have been using Muse Spark 1.3, which is equivalent to GPT 6 - Sol.
On the plus side it told me taking the light rail from SEA into the University district was faster and cheaper than Uber.
On the downside it would recommend restaurants that were out of business, checking out Salmon swimming through locks out of season, got the opening of Beecher's Cheese in Pikes Place wrong, and fumble it's way through most simple planning decisions. I think my favorite was when it asked me what flight I have tomorrow so it could help plan when to head to SEA. It sent me a very panicked message that the flight was leaving today! Very soon! I had to remind it this is a daily flight and I am booked on the next day before it checked and apologized.
For things that are verifiable, and code tends to be that way, these models can do all sorts of things. But I find it hard to reconcile Meta's goal of having a personal assistant bot with something that is unable to do the simplest tasks.
I run by the locks a few times a week and while I don't usually see fish, the presence of sea lions the last few weeks tells me that at least some salmon are running right now.
There were a number of sea lions hanging out there! It's entirely possible they were snacking on whatever salmon were going by. Not all was lost though, I got to enjoy watching some boats go through the locks.
I'm assuming it was out of season because in talking with the Uber driver afterwards he said we'd missed the end of the season. I'm assuming his "local model" was up to date and accurate.
Elon fucking Musk has literally every single piece of personal information of every person in the country. It’s so far past too late for any of this to matter.
I’m not sure how we start over but this data plus LLMs is gonna make it a full time job to keep your parents from sending every penny to a scammer.
I was given a picture cube, which is like a Rubik's cube but every side is a unique picture. It came scrambled and I don't have an original reference image. I like to take videos of it and give it to llms to solve. I call it my agi test because it hasn't been solved yet
On the plus side it told me taking the light rail from SEA into the University district was faster and cheaper than Uber.
On the downside it would recommend restaurants that were out of business, checking out Salmon swimming through locks out of season, got the opening of Beecher's Cheese in Pikes Place wrong, and fumble it's way through most simple planning decisions. I think my favorite was when it asked me what flight I have tomorrow so it could help plan when to head to SEA. It sent me a very panicked message that the flight was leaving today! Very soon! I had to remind it this is a daily flight and I am booked on the next day before it checked and apologized.
For things that are verifiable, and code tends to be that way, these models can do all sorts of things. But I find it hard to reconcile Meta's goal of having a personal assistant bot with something that is unable to do the simplest tasks.
reply