Waldo #

This is a low-resource forward/reverse geocoder that uses only WhosOnFirst administrative data. That means: no addresses, no postal codes, no points of interest. It only geocodes adminstrative boundaries, so you should think cities, regions, neighborhoods, countries and the like, but not addresses like 1600 Pennsylvania Avenue.

Why? If you look at the typical resource requirements for running a planet-scale geocoder like Nominatim or Pelias, it requires on the order of 64GB of RAM to hold the dataset and can have a complicated operational architecture with Elasticsearch in the mix. That's because it's dealing with a lot of disparate data sources including postal addresses from all over the U.S. and Europe. But what if you don't need all that?

I use this geocoder in my adventure motorcyle trip planning app: https://rtw.run. When planning a motorcycle trip, searching for a city like Washington DC, is sufficient for planning a trip. You don't need the precise location of the Army-Navy Country Club or the White House.

If you can get away with city or neighborhood-level precision, the benefit is you can run a planet-scale geocoder on less than 2GB of RAM with response times < 250ms.

Data sources #

Head over to https://geocode.earth/data to download a parquet-formatted extract of the WhosOnFirst administrative data for a country or the whole world.

Running as a service #

waldo --host 0.0.0.0 --port 8999 whosonfirst-data-admin-latest.parquet

Routes available:

GET /forward?q=<query>&lat=<lat>&lon=<lon>
GET /reverse?lat=<lat>&lon=<lon>
GET /health

The forward geocoding route allows you to set a bias point by passing lat and lon query parameters. This orders results by their distance from your bias point. To keep response times reasonable, these routes only return the top 10 results as you would typically want in an autocomplete component. The bias point helps find results that are more relevant to the current map bbox.

You can update the database on the fly by downloading a new parquet file, overwriting the old one, and then sending SIGHUP to the process.

Response type #

Responses for both forward and reverse geocoding have the same shape (in Typescript format):

{
  results: Array<{
    Gid: number; // int32 database id
    Name: string; // english
    Placetype: string; // see wof for types here: typically neighbourhood, locality, city, region, country
    Lon: number; // float64
    Lat: number; // float64
    Distance: number; // float64 distance in meters from bias point to centroid of the location (approximate)
    Bearing: number; // azimuth from bias point to location 000 = N, 090 = E, 180 = S, 270 = W (can be helpful if you want to draw an arrow in the direction of the point)
    Region: Array<string>; // traversing up from this location to its parents, usually ending at the continent (e.g., [Washington, District of Columbia, United States, North America])
    Country: string; // two-letter ISO code (e.g., US)
  }>;
  num_results: number; // max 10 for forward geocoding, always 1 for reverse geocoding
}

Finding points of interest #

Running lean requires using a database that only includes adminstrative boundaries. But users may also want to forward geocode points of specific places they want to go that won't be in the database, such as a restaurant, hotel, or venue. For that use case, the forward geocoder can also decode Google's open location codes, also known as plus codes.

Every location on Google Maps also includes a plus code that looks something like 35X7+FH Shimla, Himachal Pradesh, India. When one of these values is passed to the forward geocoding route or the Forward function, it will first forward geocode the reference location, here Shimla, HP, then use the associated plus code to find an exact location.

Usage as a library #

db, err := waldo.LoadParquetFile("world.parquet")
if err != nil {
  return err
}
results, err := waldo.Forward(db, "my query", [2]float64{lon, lat})
// do something with results

How it works #

A curated subset of the data in the parquet file is loaded to an in-memory DuckDB database and then queried with SQL. An index on a 3-letter prefix of the location name speeds up queries substantially. Reverse geocoding queries are slow, but the cost to use an RTree index in terms of memory consumption is very high.

Performance #

Benchmarking code lives in the bench directory. To run all of them:

go run bench/*.go

Or just forward benchmarks:

go run bench/run.go bench/forward.go

Typical forward geocoding performance on my ancient laptop (quad core i5, 2.6GHz) is:

 Benchmark            Min      Max       Median   Mean     p95
 Forward N=3          27.3 ms  230.8 ms  84.9 ms  89.8 ms  172.0 ms
 Forward N=4          27.8 ms  165.6 ms  65.2 ms  67.5 ms  127.4 ms
 Forward No Bias N=3  52.3 ms  163.4 ms  68.5 ms  72.8 ms  100.4 ms
 Forward No Bias N=4  54.3 ms  125.4 ms  74.8 ms  79.6 ms  115.7 ms

Steady state RAM usage is < 1GB, but can be higher when the dataset is initially loaded and transformed in memory (~1.2 - 1.5GB). If you use the server SIGHUP database reload while running, you 2X RAM usage while the new database is loaded in parallel with serving requests from the old one. More benchmarks and data is located in the bench/ directory.

License #

MIT