yeah, basemaps are small enough to store and host cost effectively, but high res satellite imagery gets big and expensive fast as you go up in zoom level. I built my own satellite mosaic of North America for my app so I can offer users offline access to the tiles and had to stop at z17 because of projected storage requirements. high res global coverage can get you into the hundreds of TB range which is no longer in the "stick it on r2 and serve it with a worker" or "download and upload the entire dataset sequentially over your residential internet" territory
the fun/hard parts of the project weren't really scale dependent, although downloading source imagery and building the tile pyramids did take literally weeks to run. actually, the hardest part was getting demosaicing right, e.g. making the quilt seamless. filtering out cloudy and smokey/hazy tiles while picking alternatives from the same time of year so groundcover & crops are in a consistent growth stage so you don't end up with checkerboard patterns or cloud silhouettes randomly across your map. and there were other problems that came up more or less just because of how & when the satellite took the images that require exposure correction and more filtering
I also have a pelias index with 170M records for search and self-host valhalla for routing. the open source mapping ecosystem has really impressed me the more I learn with each project
I was playing with their elevation data last week. Maptiler provides elevation data as well but it gets expensive quickly with them. We managed to blow through our 50$ limit in two days with them just testing this out. So, we tried out Mapterhorn.
I really like their data set. For self hosting, vector map tiles are small enough that storing it in a bucket won't break the bank. I think you need somewhere around 100-150GB or so for protomaps. A few dollars per month goes a long way. However Mapterhorn's elevation data set is something like 11 TB or so. That's going to cost some money to store somewhere. Satellite imagery is similarly expensive to host. And you typically want both.
There is of course an asymmetric relation with what you store and what you serve. Ideally, you store data for the whole planet. But you probably are only going to serve a tiny portion of it. This favors hosted setups where you pay for what you use via a CDN and share the storage with others. A shared service also relieves you from the burden of having to keep that data up to date. The main issue with that is commercial map services tend to overcharge for more intensive usage to the point where self hosting actually becomes attractive despite this.
The main issue is that the number of requests really goes through the roof if you start getting tiles beyond certain zoom levels, which pushes you into the expensive price tiers. With most providers you pay per request and you can easily generate hundreds of thousands of requests just panning around at low altitude over a map, which means you download lots of small tiles. That's how we hit the Maptiler limits. They have high resolution elevation data and exploring the Alps is a lot of fun. We hit 800K requests per day without any real users using our system even. This was just us playing with the data. We switched to Mapterhorn and cap the zoom level at zl-12 for now. Since it's a free service I want to be a bit mindful of not hammering their service.
I'd be interested in reasonable commercial solutions that won't cost me 300+$ per month that would allow me to remove zoom level limitations. At those price levels, self hosting becomes a possibility of course. Which suggests to me that somebody should be able to undercut that price and still make a decent profit.
yeah, I heard about them for the first time a few weeks ago. they have a fantastic data set just judging with my bare eyes. I built my topo map from terrain tiles on aws + gebco et al for the bathymetry data which is just personal preference (I love me an old globe with the tactile mountains and seafloor)
oh sorry, no I didn't know about that. where are they posting about it? I don't see anything on their website or bluesky. there's an nlnet project page for it but it just links back to their site. I'd love to know what their goals are
what data sources are you using for the global satellite imagery? I’m building mosaics for my own app but don’t have the pockets to pay for licensed data right now. NAIP is fantastic for the US but I’d really like to get z16+ coverage for Europe, too
Claude code writes adds a `User-Agent` request header like `my-session-slug/1 ` when searching the web and requesting data. This violates the privacy of paying customers and has resulted in me getting dozens of scam and phishing emails per day from properties I have never personally visited. This policy is asinine: it does nothing but harm me, the customer, while providing immense value for scrapers and scammers.
The reason they do this is because otherwise they block the requests.
I'm not sure what the issue is with publishing your email address in principle, you are reaching out, you want to inquire about some business. If anything use a different address. But why would you asymmetrically expect businesses with websites to bend over to give you all the advantages, they need some assurances as well.
Anonimity usually works one way, one party can be anonymous, not both, if you come from consumer land, it's the companies exposing themselves and doing all the compliance, if you choose privacy and force the companies to expose themselves, you'll get your wish granted, but there will be a cost, it's unlikely you'll be in the more advantageous position.
As implemented, it strongly violates the principle of least surprise (assuming this is true).
A more sane implementation of this would involve a prompt the first time an agent attempts to connect to each of the sites that Anthropic thinks it necessary to send this email header to, asking if this is OK (and bailing on all requests to that domain if the prompt was rejected).
>” A more sane implementation of this would involve a prompt the first time an…”
but users of this type of software generally want an easy experience and don’t want to be bothered with details, only conveniently retrospectively users demand to be involved in the decisions.
it reminds me of a pattern I see in small business owner managers where they want to simultaneously be consulted about everything, but are never around the business. You can’t have it both ways.
I must defend Anthropic. It is normal to tell every website who (email) and why (session slug) you are looking at a page they publish. The visited website has to be reassured for all the compliance they have to do and how they expose themselves. If scammers use that against you, just change your account address on anthropic's product. You must assume everything you give them will be leaked. Probably should even use a separate payment method uniquely for that. The era when you could just read a web page is bygone. Website holsters can't keep keep giving website browsers all the advantageous positions without a cost. Good thing anthropic already made the correct decision here, once again, as always. Amen.
I think you're overestimating the design intent that goes into a product that's proudly dogfooding the future of vibe coding.
It's very possible that nobody decided this at all and that's it just something an agent threw in on its own because some Chain of Thought imagined it was a clever way to correlate queries back with user accounts or something.
just relating my personal/professional setup, I run Pocket ID for myself and made an agent account for claude. I have it set up with limited access policies to pre-prod Cloudflare resources and Tailscale servers for occasional tasks. I really like the control I have over access, grant it just when it’s needed and only exactly as much as needed. I wish SSO didn’t require biz/enterprise plans on so many services so I could use my OIDC in more ways
I got blocked by Microsoft for trying to advertise on DDG. I signed up for a new account (no prior M$ affiliation to start from), imported my Google Ads campaign, paid them a hundred bucks and was almost ready to spend. Then I uploaded my WA state business license and immediately got force logged out without any reason. An email came by to say I had been blocked. Support was utterly useless, there was no way to appeal that I could find, and I had to file a chargeback with my bank to get the $100 out. Don't waste your time & money, folks
the fun/hard parts of the project weren't really scale dependent, although downloading source imagery and building the tile pyramids did take literally weeks to run. actually, the hardest part was getting demosaicing right, e.g. making the quilt seamless. filtering out cloudy and smokey/hazy tiles while picking alternatives from the same time of year so groundcover & crops are in a consistent growth stage so you don't end up with checkerboard patterns or cloud silhouettes randomly across your map. and there were other problems that came up more or less just because of how & when the satellite took the images that require exposure correction and more filtering
I also have a pelias index with 170M records for search and self-host valhalla for routing. the open source mapping ecosystem has really impressed me the more I learn with each project
reply