geolite 
making openstreetmap data usable by anyone: from .osm.pbf to the first query answer.
open source geocode
getting started
with cargo
$ cargo install geolite
$ geolite --version
$ geolite build brazil # needs +/- 45gb while building then +/- 3gb when finished
$ geolite http-server # open http://localhost:8080
with docker
# command line
$ docker run pjpimentel/geolite:latest --version
# pre-built countries geocodes
$ docker run -p 8080:8080 pjpimentel/geolite:prebuilt-brazil
# open http://localhost:8080
docs
docker images
base image
$ docker run pjpimentel/geolite:latest --version
$ docker run -t -v ./geolite-data:/.geolite pjpimentel/geolite:latest build brazil
pre-built images
size matters: once a country’s data grows too large for a docker image, its pre-built image stops being updated and may be dropped.
docker run -p 8080:8080 pjpimentel/geolite:prebuilt-brazil
docker run -p 8080:8080 pjpimentel/geolite:prebuilt-brazil-YYYYMMDDHHMM
# open http://localhost:8080
available country list
| # | country | docker cmd |
|---|---|---|
| 1 | brazil | docker run -p 8080:8080 pjpimentel/geolite:prebuilt-brazil |
| 2 | portugal | docker run -p 8080:8080 pjpimentel/geolite:prebuilt-portugal |
| 3 | argentina | docker run -p 8080:8080 pjpimentel/geolite:prebuilt-argentina |
| 4 | bolivia | docker run -p 8080:8080 pjpimentel/geolite:prebuilt-bolivia |
| 5 | chile | docker run -p 8080:8080 pjpimentel/geolite:prebuilt-chile |
| 6 | colombia | docker run -p 8080:8080 pjpimentel/geolite:prebuilt-colombia |
| 7 | ecuador | docker run -p 8080:8080 pjpimentel/geolite:prebuilt-ecuador |
| 8 | guyana | docker run -p 8080:8080 pjpimentel/geolite:prebuilt-guyana |
| 9 | paraguay | docker run -p 8080:8080 pjpimentel/geolite:prebuilt-paraguay |
| 10 | peru | docker run -p 8080:8080 pjpimentel/geolite:prebuilt-peru |
| 11 | suriname | docker run -p 8080:8080 pjpimentel/geolite:prebuilt-suriname |
| 12 | uruguay | docker run -p 8080:8080 pjpimentel/geolite:prebuilt-uruguay |
| 13 | venezuela | docker run -p 8080:8080 pjpimentel/geolite:prebuilt-venezuela |
| 14 | netherlands | docker run -p 8080:8080 pjpimentel/geolite:prebuilt-netherlands |
| 15 | switzerland | docker run -p 8080:8080 pjpimentel/geolite:prebuilt-switzerland |
cookbook
execute all necessary steps
$ geolite build brazil
# or with docker
$ docker run --rm -t -v ./geolite-data:/.geolite pjpimentel/geolite:latest build brazil
$ docker run --rm -p 8080:8080 -v ./geolite-data:/.geolite pjpimentel/geolite:latest http-server
extract each data from a pbf file
$ geolite osm-pbf-file download brazil
$ geolite exec extract-osm-pbf-blob-chunks brazil
$ geolite exec extract-osm-pbf-header brazil
$ geolite exec extract-osm-pbf-data brazil
$ geolite --preset brazil exec extract-osm-admin-levels
$ geolite --preset brazil exec extract-osm-house-numbers
index extracted data
$ geolite exec index-admin-levels-hierarchy
$ geolite --preset brazil exec index-addresses
$ geolite exec index-coordinates
optimize data
# deletes a country without children and a street without parents, then folds the ways of one
# street into one row: both read the hierarchy and clear the addresses and coordinates indexes,
# which have to be built again after them
$ geolite exec optimize-delete-isolated-admin-levels
$ geolite exec optimize-merge-admin-levels
$ geolite exec optimize-delete-intermediary-data
$ geolite exec optimize-sqlite-file
geolite exec runs one stage per call and refuses a stage whose requirements did not run, naming
the one to run first; geolite exec --help lists the stages in the order the pipeline runs.
geolite build and geolite merge run optimize-delete-isolated-admin-levels and
optimize-merge-admin-levels right after index-admin-levels-hierarchy, so the two other indexes
are built once, over the rows that stay. no index stage deletes or folds on its own.
query
$ geolite query "-23.970949,-46.318730"
$ geolite query "rua castro alves, embare, santos, sao paulo"
explore
$ geolite http-server
merge separately-built databases
# machine 1
$ geolite --sqlite-path sudeste.sqlite3 build sudeste
# machine 2
$ geolite --sqlite-path nordeste.sqlite3 build nordeste
# merge all builds into a new combined database (created fresh)
$ geolite --preset brazil merge brazil.sqlite3 sudeste.sqlite3 nordeste.sqlite3 ...
# or with docker
$ docker run --rm -t -v ./geolite-data:/.geolite pjpimentel/geolite:latest --sqlite-path /.geolite/sudeste.sqlite3 build sudeste
$ docker run --rm -t -v ./geolite-data:/.geolite pjpimentel/geolite:latest --sqlite-path /.geolite/nordeste.sqlite3 build nordeste
$ docker run --rm -t -v ./geolite-data:/.geolite pjpimentel/geolite:latest --preset brazil merge /.geolite/brazil.sqlite3 /.geolite/sudeste.sqlite3 /.geolite/nordeste.sqlite3
$ docker run --rm -p 8080:8080 -v ./geolite-data:/.geolite pjpimentel/geolite:latest --sqlite-path /.geolite/brazil.sqlite3 http-server
run the end-to-end suite
$ cargo test --profile e2e --test '*' -- --ignored --nocapture
question and answers
a) why geolite?
“geo” comes from “geocode” and “lite” comes from “sqlite”, no special meaning.
b) where does the data come from?
the data comes from any .osm.pbf file (openstreetmap’s protobuffer format). this project uses geofabrik as the initial source, which extracts its data from openstreetmap.
c) which license applies?
the source code is under agplv3 and its dependencies keep their own (all permissive: mit, apache-2.0, bsd, zlib, isc). databases built from openstreetmap data — including the ones shipped in the prebuilt-* images — are derivative databases under the odbl, filtered and restructured for geocoding: © openstreetmap contributors.
d) why build another geocoder?
two reasons, in this order: to practice rust on a problem large enough to be interesting, and because the alternatives i tried were a bit too complex for a quick run.