What is the preferred way to download planet diff files?

I am keeping a copy of the planet file which I update every few weeks with the daily diff files. This used to work quite well until some time ago, when I guess some sort of rate limiting was implemented. Now it takes more time to download the diffs (without applying them) than it would take to download the whole planet dump again. In order to keep transfer sizes low for the servers, I still continue to apply daily diffs, but I wonder if this slow downloading behaviour is maybe a hint to operate in a different way?

I am seeing lots of timeouts, like this:

--2024-02-03 12:28:36--  (try: 3)  https://osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com/planet/replication/day/000/004/139.osc.gz
Connecting to osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com (osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com)|2a05:d050:40af:8651:34db:a9a6::|:443... failed: Operation timed out.
Connecting to osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com (osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com)|2a05:d050:404f:85c9:34db:ab6e::|:443... failed: Operation timed out.
Connecting to osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com (osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com)|2a05:d050:4020:441:34db:2fe5::|:443... failed: Operation timed out.
Connecting to osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com (osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com)|2a05:d050:4020:181:34db:2e24::|:443... failed: Operation timed out.
Connecting to osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com (osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com)|2a05:d050:4020:180:34db:2e20::|:443... failed: Operation timed out.
Connecting to osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com (osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com)|2a05:d050:404c:450:34db:8cb9::|:443... failed: Operation timed out.
Connecting to osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com (osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com)|2a05:d050:404f:8541:34db:ab7e::|:443... failed: Operation timed out.
Connecting to osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com (osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com)|2a05:d050:40af:84c9:34db:a926::|:443... failed: Operation timed out.
Connecting to osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com (osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com)|52.219.169.21|:443... failed: Operation timed out.
Connecting to osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com (osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com)|52.219.171.133|:443... failed: Operation timed out.
Connecting to osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com (osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com)|52.219.171.105|:443... failed: Operation timed out.
Connecting to osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com (osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com)|52.219.72.139|:443... failed: Operation timed out.
Connecting to osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com (osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com)|52.219.46.35|:443... failed: Operation timed out.
Connecting to osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com (osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com)|52.219.140.183|:443... failed: Operation timed out.
Connecting to osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com (osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com)|52.219.170.177|:443... failed: Operation timed out.
Connecting to osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com (osm-planet-eu-central-1.s3.dualstack.eu-central-1.amazonaws.com)|52.219.171.25|:443... failed: Operation timed out.
Retrying.

3 posts - 3 participants

Read full topic


Ce sujet de discussion accompagne la publication sur https://community.openstreetmap.org/t/what-is-the-preferred-way-to-download-planet-diff-files/108854