We need this benchmark for non-coding harnesses. Harnesses with custom knowledge, custom tools, specific guidance, etc. A ton of the task-specific performance comes down to the skills/tool-set.
I've been building a tuner that tunes the harness (prompts, skills, tools, etc) instead of the weights of models: https://github.com/Kiln-AI/harness-tuner -- Might try to publish some "untuned" vs "tuned" comparisons, the numbers are dramatic.
Have you looked through this repo? I'm not seeing enough here to really answer the question posed. Like some of this repo, I know came from correlation of information not connected before (like the quasi-Riemann result). Some of it is just an extension of ground work already done, not necessary new in a sense that should be interesting. We know word calculator can calculate. There's probably a lot of human credit missing here.
I fully stick to ZigBee and ZWave devices wherever possible, the reliability is just much better in my experience + they don't clog up my wifi. Bonus aspect of replacing all my lightswitches with ZigBee smart switches is they all act as mesh routers for the network, so signal is good through the entire house.
Zigbee: an open spec, no licensing fees, cheaper devices. Can be locked down by OEMs to their hub (which can then demand an internet connection). In recent memory Hue attempted this, and only walked back after huge backlash. I couldn't find a thermostat back when I did my home without hub lock.
ZWave: the consortium demands interoperability. Testing and licensing fees means more expensive devices. Things will work together, though sometimes you will need to figure out what parameters actually do.
Use both. Home assistant is perfectly happy with two dongles plugged in.
In practice the answer ends up being both. There are some devices that only exist in one protocol (sometimes for technical reasons) and as a sibling noted, the antennas are cheap and home assistant exists.
ZWave stuff tends to be rock solid (there is expensive certification involved) but that means the devices are more expensive and the selection can be limited because the volumes aren't always there to make sense to get certified.
The only real problem I've found with the "all of the above" situation is that device bindings only work within one protocol. If you want to bind a lightbulb to a wall switch such that they don't require a trip through the hub they need to both be zigbee, zwave, or matter. This is unfortunate because there are a lot of really nice ZWave wall switches but not a lot of ZWave lightbulbs.
Devices using that band are like devices that don't have USB-C. There are plenty of useful ones in my home, but they are targets for eventual replacement and any new product still using the older standard should not be brought into my home if it can be avoided at all. A micro USB port on a new device, is a reason to return the product
Zigbee is definitely at 2.4 GHz. Zwave is 900 MHz only IIRC. I can't recall if Zigbee is at 900 MHz although some form of it (not called Zigbee) likely is.
This is true theoretically, but in practice, out of the tens of smart devices i've set up, the Zigbee devices have always been more reliable than the Matter over Thread ones. Also easier to set up. Sure it's an older standard but it seems like its simplicity makes it work much better.
I have had the exact same experience. Zigbee has been working for years (in HomeAssistant) without any problems.
With Matter/Thread though, i frequently run into issues. Particularly these were power switches from Ikea by the way, but i don't thing that's relevant.
Also the range of Zigbee devices is much better / bigger, since each device "expands" the network range so to speak.
Matter over Thread is a bit more complicated just because there are typically two parts: a Thread border router and a Matter server, compared to ZigBee where you might just use something like Zigbee2MQTT which does both functions.
That's also part of it's advantage though: I have a Mikrotik router which has a Thread radio, so I run the router software on that, and Matter server on another machine - so no USB dongle needed in my server rack where the signal would be poor.
Also Matter over Thread supports multiple Thread routers, multiple Matter servers, and devices being paired to multiples of each - maybe that doesn't help with the complicated part, but it opens it up for commercial deployments as you can have redundancy. I would not be surprised if it replaces DALI in some instances in the not too distant future.
There are still some teething issues, which is because it's still kind of early in its adoption (ZigBee felt the same before Zigbee2MQTT). Effectively it's the same, just packaged different, there's no reason why you couldn't plug in a USB dongle and run a "Thread2MQTT" container and be done with it - the pairing doesn't need an app, it can be done through a browser with Bluetooth.
Personally I feel it's the way everything will go simply because it is more interoperable. Already today you can get routers, TVs and fridges (Samsung) with Thread hardware, so it's going to become mainstream as regular people will be able to just buy a smart bulb, pair it, and control it with their phone without buying or setting up anything else. Theres also Matter over WiFi, so it could replace proprietary platforms like Tuya/SmartLife.
IKEA released a bunch of new smart home devices that support Matter over Thread and ZigBee this year if you want to try it out.
Yeah, I do have the IKEA Matter-over-Thread air sensor. Whenever the electricity goes down, it refuses to come back up until I restart it multiple times. I never saw this issue with any Zigbee device. I also had some dimmers that operated both on MoT and Zigbee, and ended up switching them all to Zigbee to resolve this exact issue.
I get that MoT has more features, but it seems like it makes the whole thing less reliable. I also hate the pairing process with Matter-over-Thread. Enable IPv6 in your home network, scan the QR code with your phone... What if the QR sticker falls in a few years? With Zigbee I just press the reset button and I see it directly in the HA web interface. No need for a phone, no need for camera, and doesn't touch your router. Less features, but my devices have been working for years without a single failure.
FAFAIK neither Zigbee nor ZWave uses IP networking in their protocol stack which makes them infinitely better than Matter, especially given the context of this article.
Matter-over-Thread also keeps devices away from the Internet if you run your own OTBR (e.g. via Home Assistant). With off-the-shelf border routers you can't be sure.
Matter over Thread is such an insane protocol. Devices need to have an additional bluetooth radio on top of the Thread IEEE 802.15.4 radio, just for the initial pairing, then it's never used again.
I'm disappointed that Ikea decided to switch their smart home gadgets from using Zigbee to Matter over Thread.
IIRC they're still offering Zigbee that you can switch to via a manual switching sequence, though some people say it still runs a little worse than the older Zigbee line(s).
As far as I know, pretty much all commercially available Thread SoCs also support Bluetooth LE. They both operate in the 2.4 GHz band, so supporting both protocols isn't that hard.
I thought so too until I found out Thread currently (AFAIK) doesn't support multicast requests while Zigbee does, which can be hugely important for smart lighting with 30+ bulbs. Until that gets added to the spec (and supported) you might be better off putting the lighting part of the setup on Zigbee.
> Group messages are also important as they allow simultaneous control of several Matter Nodes through Multicast. In order to route this traffic into the Thread network, both Matter and Thread implement the Unicast Prefix-based IPv6 Multicast Addressing Scheme defined by RFC 3306.
> Thread defines a series of features to support multicast across a heterogeneous network (Thread and Wi-Fi/Ethernet Network segments) for multicast addresses with scope larger than realm local.
I got the phrasing a bit wrong, but it's Matter over Thread that doesn't support multicast on an application level. It's part of the spec that just shipped (1.6.1) but no controllers actually support it right now. It's about to ship with HA:
Do you have any favorites for single smart plug or multi switch smart plug strip? I used to be a heavy tp link kasa user since they have smart plug strips that worked locally (polling I think and not push) but I am now doing my best to move everything to plugs that don't even need setup through an app, just home assistant. Wish I had thought about this more when I first started with home automation but I think the migration will be worth it.
Look at Zooz, they have a variety of plugs and multi-port strips with control of individual outlets (both at the outlet and via zwave). They have options rated for being outdoors (holiday lights, etc), heavy duty ones for large motors, in wall switches and relays that go in the electrical box if you want something more permanent or invisible, etc. Solid support in home assistant. Push, 100% local, no shenanigans, no apps, direct to home assistant even for the updates.
I also have some eve energy plugs that do matter/thread (also push/local). There are apps here but i generally haven't needed them once i got things set up in home assistant.
For single plugs, IKEA INSPELNING are really great if you can still get them (I stocked a bunch of them before they sold out here). Zigbee, affordable and very reliable. However, I had two units with loose solder balls in them, so I shake them a bit as a check before starting to use them:
(This production issue is a shame, great plug otherwise and they were something like 8 Euro at some point.)
GRILLPLATS is the successor. I have one, but haven’t tried it much yet because it is Matter over Thread and we are still mostly on Zigbee. They have a hidden Zigbee mode, but it doesn’t expose power use:
But if Matter over Thread fits into your system, it’s an affordable option.
For power strips, we are now using the Gen4 Shelly Power Strip. It supports pretty much anything: its own API over WiFi that is supported by HA, Shelly cloud, Matter over WiFi, Zigbee, and Bluetooth. It can even work as a WiFi range extender and as a Bluetooth proxy for HA. Initial setup is by connecting to it as a WiFi AP and the connection to the built-in web interface. There were some issues with Zigbee flooding, but I think that was improved with firmware updates.
I have a ton of Kasa. Local polling, but since it's more for on/off, I don't care about a 5s poll. I had to buy about ~30 light switches for my house, and using Kasa saved thousands over Lutron.
Smart Plugs, sensors, etc: Ikea is amazing. You need to buy a hub, but then accessories are 3x cheaper. Their plugs are $9 and have energy monitoring which is amazing. They have water leak sensors for $12. Incredible value.
Except VARMBLIXT. I can't connect it to HomeAssistant no matter how I much try. It sort of begins discovery, but then can't connect to it. Could be some weird networking issue, but all IKEA lamps connect without any issue.
> Their plugs are $9 and have energy monitoring which is amazing.
That's 2nd generation, first one was just a basic on/off smar plug. I still got these and so far I had no issues even after Dirigera hub got that firmware update which enabled Matter. Ikea 'gear' overall work much better than Meross one I had before.
Switchbot smart plugs around$5 in a four pack and you can flash them over the air to ESP home or tasmota so they run completely local if you want a Wi Fi option
Esp32c3
IKEA is a bit hit and miss. Their light bulbs are great for the price. Their power plugs also work pretty well, but at some point I started shaking them before use because two of them had pretty large solder balls that would have shorted things out:
Most of their sensors are meh. Their PIR sensors are known to be mediocre compared to the competition. Their air quality sensor is ok to get trends, but e.g. the CO2 measurements are quite a bit off compared to quality sensors and are way off if you live at a higher altitude (due to using a thermal conductivity sensor and not NDIR like better sensors).
Also, the IKEA smarthome app (used in conjunction with the Hub) uses Play Integrity and therefore blocks e.g. GrapheneOS. But you are better off with Home Assistant and their ZBT-2 antenna (can be flashed for Zigbee or Thread) anyway.
"Anything that can run Tasmota" for me. See here: https://templates.blakadder.com/plug.html Straightforward web interface, scriptable, native HA, supports every protocol under the sun. Works fine here.
Currently using a bunch of Nous A8T (EU plug) because they come with Tasmota ootb.
I've gone one level further and only use open source smart devices (excluding the big ticket items such as the robot vacuum or air purifier, where it's often impossible to find a 'Local' option or the difference in quality of its main purpose is just not negotiable...).
ESPHome or Tasmota work really well for all kinds of things.
The last one I replaced was an infrared blaster for controlling my AC. The previous one, even though it was in an IoT VLAN decided to spam the network with requests to its mothership (never succeeding, fortunately).
thanks, that's a useful filter. i started moving to home assistant last year but i still want honor the same rules (mostly, again they are guidelines to strive for!).
Not predicting to something controversial when doing predictive text isn't censorship. You can still type any of these things. When I swing my finger around wildly it's making a best guess; not jumping to a phrase that makes me look bad or get arrested (in some countries) isn't an awful design choice.
I'm a pretty quick and accurate swipe typer, and it's so frustrating to repeatedly get the wrong result and get NO signal that it's not my error, meaning I usually try one or two more times.
Picked at random: "right wing propaganda" isn't severe enough a phrase that I instinctively know I'm hitting a swear filter. After swiping the last word iOS goes back and changes the second word to "Wong". In order to understand what was happening and then fix it, I had to backspace (deleting the whole phrase), re-swipe both words, backspace, re-swipe both words again, double tap the incorrect word, backspace, type the word, tap to return to the end of the sentence, and hit space. All to fix one letter.
If this was really about protecting people from accidentally typing these that might land them in hot water, why wouldn't iOS at least offer the word it knew I wanted in the suggestion bar?
That is still predictive. Words can be fine in their own and then contentious when strung together. I agree this is not censorship and that if peoples phones were autocompleting to something horrible then people would be up in arms.
This is no different than what happens when you give Google a very NSFW search but it doesn't contain any keywords that turn on the NSFW results. And while it's easy enough to work around with search, translate is another matter. What is the translation of the dirty word for penis? Google knows it--it can translate it to English. But the two standard terms have other meanings and I have figured out a way around that.
The problem is not accuracy -- the words from the problem phrases are never suggested either. Worse, on the three-word phrases, iOS goes back and changes earlier words to avoid completing the phrase, and removes the original from the suggestions.
I tried typing "propaganda" ten times with 100% accuracy. When it's preceded with "[left] wing", the accuracy drops to 0%.
It's still predictive. It's using the context of surrounding words, and guessing the whole phrase.
Try swiping "grocery lost" and it will predict "grocery list", even you swiped over "o" and not "i". "put", "pit" and "pot" are all exactly the same swipe, and it always has to guess which you want.
It really is always guessing, and never completely sure. Every good consumer predictive system has some guardrails. Not guessing "free tibet" for someone in China could keep them out of serious trouble.
Could then tune the confidence thresholds instead of block list? Probably. Are they censoring you? No, just tap the letters.
if you backspace and try swiping again, you immediately get "grocery lost". with "left wing propaganda" you can backspace and try again 100 times, you will only get "left wring propaganda".
now ask yourself, since you insist it's probabilistic: what is statistically more probable, "left wing propaganda" or "left wring propaganda"?
wtf you're absolutely right. ffs. On gboard it can only change the last word and not previous accepted words for me and I get left wing programs every time.
So what does this mean? What means would they be using to avoid this prediction?
You're deft at using it well enough that their predictive engine can guess your intent. There's never been a swipe keyboard that isn't guessing. Your finger literally goes over many chars it doesn't use. The team building the predictive engine has to make calls balancing their accuracy vs failure modes.
Parent is kind of right too. This should be configurable or people will complain the other way. Even something simple as "I am swiping heck but it's typing hell".
Yea it's not the best option in all situations but fixing voice dictation glitches is way less tapping over the long haul; fingers no go numb using it as much as possible by saving the tap tap tap for absolutely necessary moments
Nothing is, or really can be, as precise as typing. Judging from the other sides of my Discord conversations, nobody really ever goes back to "fix voice dictation glitches".
Try getting things done on such a device in a military context. A thousand acronyms adopted by generals who dont know the internet from a hole in the ground. I saw a Gunnary Seargent scream when he saw his rank acronym (GySgt) get autocorrected on a form to gynocologist (i suspect he hit a suggested word popup). It is bad enough having to deal with the keyboard rejecting capital letters in the middle of a word.
True, it's a very user-oriented feature. That's why they let you disable it, and also let you edit the blacklist, so that you can prevent accidentally typing words that are offensive in your own cultural context.
... Actually I'm not sure if they let you do those.
TBF user choice and customizability aren’t very Apple things in any context.
My Apple Watch has a setting for how long the display stays on before sleep. It’s a dropdown with two choices: 15 seconds or 70 seconds. Would you prefer 30 seconds? Why not 60 seconds instead of 70 you say? Fuck you, you don’t know better than Apple designers. Take it or leave it, best we can do is 15 seconds.
Yes! I got annoyed that I had to manually close old finder/preview windows, so I OSSd an app that closes any stale window. There have been commercial apps in this space a decade, but none worked how I wanted (per window not per app, config each app separately). So I just made it with Claude using some spare tokens at the end of the week. You can customize anything to be the way you like it now.
This particular item never been a problem for me because I incessantly close everything after I am done with it but it is super cool that we can ask AI to dig into the weeds for us now and fix our annoyances -- or even search repos online where someone else had and fixed this problem
Sometimes I am shocked by how blind Spotlight is. I will have something in my applications folder with a straightforward name -- say, Steam.app -- and it will keep picking the shortcut generated by CrossOver for the Windows version of Steam, that is nested inside a bunch of subfolders. Wtf? I ended up renaming the CrossOver version to WinSteam so it stops prefix-matching.
I've been building a tuner that tunes the harness (prompts, skills, tools, etc) instead of the weights of models: https://github.com/Kiln-AI/harness-tuner -- Might try to publish some "untuned" vs "tuned" comparisons, the numbers are dramatic.
reply