Setup GeoServer for a local GIS Application (like CesiumJS)
This is a summary of how to get GeoServer working nicely together on a local machine. I had a heck of a time doing this, going to a bunch of different groups, posts, YT videos, etc. So I am putting this all in one place for people to follow. And for me later down the line if I have to do this again! It should not be this hard to get this going so I hope this helps you. Here is my one caveat: I had to get this running in a ‘disconnected from the Internet’ kind of way. When we are done here you will have your own mapping server. And then you can jump to the other article I finally completed on Getting CesiumJS to work with a local GeoServer for GIS map data.
Introduction, a.k.a. WHY I had to do this
Recently I had to create a Geospatial Information System (GIS) for displaying a map, points on the map (tracks), trackpoints (movement of tracks), points of interest and drawings on the map. I had used CesiumJS in an earlier lifetime and really liked it so I chose to start with that. In addition, I needed to load GIS maps, images, streets, etc. locally where I may not have a broadband Internet connection. So I went back to GeoServer which I had talked about in my past lifetime but not used. This is to document what I did, how I did it, and hopefully help someone else from going to 17 articles and websites to find all the pieces you need to get started!
When we are done, you should have a working skeleton for running local maps. In another post, I will explain how I got CesiumJS to call this and dropped in a sliding menu to show/hide options, added event listeners for keyboard commands and line drawings, and other things to make your GIS-based application get up and running quickly!
For now, let us just get this piece started…
Install all the foundational pieces
First things first: you need to pull down PostgreSQL and make sure the PostGIS extensions are setup. On my Mac, I ran the command brew install postgis and it used Homebrew (which I love!) to connect and get every single thing you need. This takes a little bit to pull down all the pieces and dependencies. I have a script to run PostgreSQL when I want to. Or you can set it up as a service to run on boot. pg_ctl -D /usr/local/var/postgres start is the command I use and put into a .sh file. Make sure you chmod +x to allow it to run as a shell command. NOTE: If you have PostgreSQL you may have to uninstall an older one or just get the PostGIS extension working with your current one. As an example,rm -rf /usr/local/var/postgres && initdb /usr/local/var/postgres -E utf8 would be the “blow it all up nuclear” option on this. Do what you need to do base on your specific setup.
If you do not have a Mac, then you can Download PostgreSQL separately. And download the PostGIS extension as well. Follow the instructions for installation of both of these applications with defaults. PostgreSQL will run on port 5432 unless you specifically change it. (As an alternative, you can pull down the PostgreSQL and PostGIS setup inside a Docker image to run locally.)
Next get the latest GeoServer stable release for your operating system and unzip into the /usr/local/geoserver/ directory. You can cheat and run brew install geoserver however you then have to specify a data directory and such. I wanted all mine together so I downloaded and copied it to that folder. Make sure you setup permissions so you can add data to this later by owning the folder. Something like chown -R your-user-name:group geoserver to recursively setup ownership. In the /usr/local/geoserver/bin folder is a ./startup.sh file that runs the default setup. We will get to this later.
Another thing you need to have is bzip2 which is a zip/unzip tool for the OpenStreetMap data. If you do not have that you can use brew to install (or upgrade) it or go to https://www.sourceware.org/bzip2/ to download the proper version.
Finally you need to have the utility osm2pgsql to take OpenStreetMap (osm) files and put them into PostgreSQL. Again using Homebrew you can do a brew install osm2pgsql command to install osm2pgsql locally. Ok now that all that is on your box, let’s get to some fun stuff!
Download some Imagery and Databases for GIS information
First, get your world view. If you do not want to use the Bing Maps or other providers for a base layer map, you need a “world map” view to show continents and land masses/water correctly. I used the Natural Earth files at https://www.naturalearthdata.com/downloads/50m-raster-data/50m-natural-earth-1/ and they worked great. I am using the Shaded Relief and Water one. You need to download that zip file and then unzip the contents into /usr/local/geoserver/data_dir/data/ (or wherever you put your GeoServer installation). I actually made a “worldmap” directory to keep this data area from getting too busy with imagery files. You will point to this later in your GeoServer setup. The file NE1_50M_SR_W.tif is the file you need to put into a folder under the ./data/ directory mentioned. So mine ended up being /usr/local/geoserver/data_dir/data/worldmap/ and I put all the files from that zip in that directory. Now you have the whole world!
Second, pull down some more detailed data for when you zoom into a certain area on your map. I am based in the USA I am going to pull down the state of Florida GIS data. You can use the website at https://download.geofabrik.de/north-america.html to find and pull down the .osm.bz2 file for the proper data similar to OpenStreetMap. I downloaded the linked file and then used the bunzip2 florida-latest.osm.bz command to uncompress it. That gives us a raw file to load into our database with the osm2pgsql utility you downloaded earlier. Geofabrik has a boatload of map data up there and the process to load that data in GeoServer is the same as what we will go over below. If you do not want to use Florida then adjust accordingly for your particular mapping needs. Soon you need to load this into your PostgreSQL database. This is something that takes a while to run. Just be aware…
Make sure GeoServer works
Now go to your /usr/local/geoserver/bin/ directory and run ./startup.sh. You should see a lot of logging information load into your screen and eventually a “Server:main Started” message to show this is running on port 8080. Open a local web browser to http://localhost:8080/geoserver/. You can log in with the default admin user/pwd and then see a Welcome page. If you can do that, you are good to go! If not troubleshoot this and get GeoServer running locally. Make sure nothing else is running on port 8080 locally when you do this.
If GeoServer works as is, now we can load some data into it for use. That is the main point of this anyway!
Load Local GIS maps with PostgreSQL and PostGIS
Now you need to add the downloaded detailed map data from Florida (or whatever you downloaded) into PostgreSQL to use. To do this, connect into your PostgreSQL machine and create a database to load the Florida data. I am using psql to talk to PostgreSQL as I am a CLI person, so I ran psql postgres to connect into the database. To setup this database for GIS data you need to run the commands below to create the database, switch to it, then setup all the extensions:
- create database floridamaps;
- \c floridamaps;
- create extension postgis;
- create extension postgis_topology;
- create extension hstore;
Finally, we need to run the osm2pgsql command to load your Florida data into the new database like below. This takes all the OSM data and sets it up to insert into the proper tables in PostgreSQL with PostGIS setup. I ran the following command as I did not need to worry about logging. If you do, you will want to tweak the osm2pgsql command as it runs FOR A LONG TIME with logging. Here is what I ran: osm2pgsql -C 2048 -s -H localhost -P 5432 -d floridamaps ./florida-latest.osm -C 2048 --slim --number-processes 2 to load the .OSM file into the database floridamaps we just made using my login as the user login. I have no password on my local instance as I am just testing. Update this command as appropriate for your server and user/pwd. You also can add --unlogged to make it go a little faster if you wish.
Let that run for a bit (mine ran for 12 minutes on a 5 yr old MacBook) until it is done in the command window where you ran it. To make sure it works, connect your psql command to the database and run the command below in the new database. I have ~811k points, ~1.92M polygons, and ~1.34M lines in mine for now.
SELECT ‘point’ AS tbl, COUNT(*) AS cnt
FROM public.planet_osm_point
UNION
SELECT ‘line’, COUNT(*)
FROM public.planet_osm_line
UNION
SELECT ‘polygon’, COUNT(*)
FROM public.planet_osm_polygon;
Load extra GIS maps and data in GeoServer
So let’s see if we can load that imported Florida map data and Natural Earth layer…
In Geoserver, log in w/ the default credentials and click on the Workspaces link. Then click “Add new workspace” to add a new area to put all this data. I use workspaces to group a collection of files, databases, and layers logically so I can structure sets of data. As you use GeoServer you will come up with ways to keep your data structured and keep your sanity in all this! For this walkthrough, call the workspace “florida” so we can reference it later. Make the NamespaceURI “http://geoserver.org/florida” and if you wish, make it the default workspace as I did. Click Submit to save that.
Now click Stores just under Workspaces and add a new store. A store in GeoServer is a place that your GIS data is coming from. It can be things such as a file, a directory, or in our case for Florida a database. For this first part to get the world map, we are going to use that TIF file we downloaded from Natural Earth. Click “Add a new Store” on the Stores page and then choose the GeoTIFF option. Name it whatever you want (i.e. world-geotiff) and enter a description you prefer. For the URL, click the Browse button link and go to the data/worldmap/ directory we made. Find the NE1_50M_SR_W.tif file and click on it. Click the Save button to save this store. That was step 1! Thankfully once all this is done, it is done. But as I said at the outset, it is kind of a PITA.
Now you need to publish this so click the Publish link that shows up. Keep all the defaults and scroll down to the bottom of that page. Click the Save button and now you should see that as a Layer in the listing within GeoServer. Congratulations you just added the world to your GIS Server!! To check it out, click the Layer Preview link just above Workspaces in GeoServder. Then in the search field near the top right of the page type “florida” and press the Enter key to filter the layers. You should see your layer listed now. You can click the “OpenLayers” link to see something like the image just below. If all works, you can now serve this up as a layer in GeoServer for your GIS related applications.
To import, publish, and use the Florida map database we imported earlier you actually go back to Stores and choose PostGIS under adding a new store. Specify a name for this data store, the server and port (i.e. localhost and 5432) and for me I used my MacBook login name as the user. I have no password setup locally as I bring this up and down disconnected from the Internet. You would NOT do this for any stable development, test, or production server. Use whatever server, login, and password you require. And specify the floridamaps database we setup earlier. Click Save to test the connection and if all is well, it saves and again presents you with layers to publish as it did with the Natural Earth TIF file. You can click Publish on the first layer (I chose the roads) and update the name and title so it is not so generic. If you import a lot of areas or states or regions, you will want to come up with a good naming convention so you don’t drive yourself crazy! I put “FL_” at the beginning of mine to make sure I knew they all were for Florida.
Scroll down to the “Bounding Boxes” area and click the “Compute from data” link and “Computer from native bounds” link to give a bounding box around the area we are importing based on the actual data being used. This sets up the data so GeoServer knows its geolocation and only appears when the GIS is called with a bounding box that includes these features. (This will make sense later on when we demo the layers near the bottom, don’t worry!) Before you save click the Publishing tab on the top of the Edit Layer page and make sure the Default Style field in the middle of the page is set as a line. Click the Save button and BOOM! You just published the roads layer! Again, to view this you can click Layer Preview, enter “florida” in the search box and press enter. Click the “OpenLayers” next to the roads to see the roads appear in a new window.
That is pretty cool to see as you start to see this come to life. You can repeat these steps with other Layers (6 remaining for Florida) by clicking Layers in the left menu of GeoServer, and clicking the “Add a new layer” link. Choose the “florida:floridamaps” area (workspace:datastore) from the dropdown and pick another layer that is not yet published. Repeat the steps above until all 7 layers from the Florida data are imported. Once all are in you can actually create a Layer Group so you can call one group from GeoServer and all layers will show up correctly with a smaller call URL. The “Optional” area below talks on that as well with the GitHub link. For each layer you can go to Layer Preview when done, filter on “florida” and see the OpenLayers version of the file as you did before. The one below is the polygon layer from OSM.
FYI, if you click on the “Compute from data” when adding layers and it does not show valid information for the Geo referenced data (a 0 or a -1), I usually just cancel and move on to the next one. If someone knows of a better way please leave a comment below and I can update that.
View the GIS data by calling the Web Mapping Service (WMS)
Once all the data is in GeoServer, there is a way to view these layers by calling the same URL or API that GIS-enabled applications would call on GeoServer. You can go to the Demos link in GeoServer on the left menu near the bottom and then click “Demo Requests” to get a list of Requests you can choose. The URL is like below and you can see there are styles, format, and the layers listed as well as the height and width of the image returned. The “bbox=” area of the URL shows the bounding box (think of a “top left corner, bottom right corner” for the map) as well as the layers to use. The default uses the topp:states layer however you could enter one of the workspace:layers combinations from the Layers listing from the Florida data r a Layers group to get back your own data. You even could call the florida:world-geotiff layer we added for the Natural Earth picture for that bounding box area.
I would spend some time playing with and hacking against these URL calls like the map URL or the OpenLayers URL to see how the GeoServer works behind the scenes when your GIS-enabled application calls it. It helps to explain in your mind how GeoServer works from front to back.
So, that is it! You have a working local GeoServer to connect your GIS enabled applications with such as CesiumJS or even OpenLayers. Yes I know…it is a lot to setup. However, once you know the pieces you can repeat sections to add more or different data, add layers, and call the Web Mapping Server URL to see what it returns. See if you can add more layers, import them, reference them in GeoServer, and use the Demo URL area to bring up the area you wish to see.
Optional Step: Customize the look and feel of the OSM data
If you do not want the gray/black/blue lines on your map and want a softer look, I actually liked the way https://github.com/fegyi001/osmgwc ran through the look and feel of Google Maps using CSS to make GeoServer serve up roads, waterways, counties, etc. So I highly recommend going here to see that. I did not write this up on this blog, as this person has a great writeup on it in GitHub and it is fairly easy to follow. I did the layer groups and used the CSS add-in as described in their GH README. I recommend this as the end result looks fantastic IMO! Remember to download the CSS plugin for the correct version of GeoServer you are using. This setup has about 17 layers it creates versus the 7 that are made when you import the OSM data. It logically separates the data into things like roads, counties, country border, waterways, buildings, etc. and styles them accordingly.
The CSS used in the link from GitHub above makes the Open Street Map view much more familiar as most have used Google Maps one way or the other. See if you can try that yourself!
Reference Links
CesiumJS Documentation: http://cesiumjs.org/refdoc.html
Downloading OpenStreetMap data by country: https://download.geofabrik.de/
Natural Earth map download: https://www.naturalearthdata.com/downloads/50m-raster-data/
PostgreSQL Download: https://www.postgresql.org/download/
Homebrew for package management: https://brew.sh/
