Difference between revisions of "Installing OTP"

Latest revision as of 17:35, 11 June 2015

>> return to Cedeus IDE

Pre-Note: To install OTP on a VM, e.g. Ubuntu 14.04, follow the VM setup instructions from Setting up Elgg VM on CedeusGeoNode

currently:

• OTP is installed on gisciencegroup.ucalgary.ca
• not run anymore with the Winstone Server, but also with Tomcat (in usr/local/tomcat), since only either server can use port 8080 (... and Tomcat is used by GeoServer and GeoNetwork already)
• webapps loaded are (all use the same graph):
  • trip planner: http://gisciencegroup.ucalgary.ca:8080/opentripplanner-webapp/
  • analysis app: http://gisciencegroup.ucalgary.ca:8080/opentripplanner-analyst-client/
  • PYP walkshed test-app: http://gisciencegroup.ucalgary.ca:8080/pyp-walkshed-client/

An now to..

Install from Scratch

Official OTP Install Instructions [outdated due to new OTP structure]

The instructions below are old, until OTP 0.11. Now there is a new OTP that runs with grizzly server and does not need apache/tomcat. The install instruction for this one can be found on: http://docs.opentripplanner.org/en/latest/

• the initial Install of OTP with TriMet data is described here: http://github.com/openplans/OpenTripPlanner/wiki/TwoMinutes
• the installation with building a graph, etc, for TriMet (Portland) is described here. Including use of OpenStreetMap and Google Transit Feed (GTFS) data http://github.com/openplans/OpenTripPlanner/wiki/FiveMinutes

Create a running OTP

- follow the two guides to install OTP; i.e. the 2 minutes introduction and the 5 minutes dive-in

notes:

• I installed OTP, as recommended, under /otp on the GIScience Group server. This may require the change of permissions for the newly generate /otp folder (chmod 755). - Also creating the "otp" folder under / root may require "sudo"s
• start OTP with /otp/bin/start-server.sh from a console. To keep it running add the "&" sign, so you can close the console without the java (winstone) process being stopped.
• For some reasons I had some trouble with running first, i.e. I could see the OTP web interface, but the routing was not performed. This problem got solved, when I did build the graph file "Graph.obj" new with /otp/bin/build-graph.sh
• Building the graphs took quite some time and memory. For that reason I needed to do this on the GIScience group server (10 min for the TriMET/Portland graph) and not on my laptop (broke after 70+min).
• Note, instead of using winstone, one can also upload the war files to the Tomcat directly (this will inlcude the graph location configuration file). However, one needs most likely to adjust the file size limit for upload in Tomcat and also check for the memory settings - i.e. make sure the app gets loaded (which depends on graph size). (=> see here my note in the section Setting up osgeo vm )

Create OTP with Calgary Data

• try to build the graph with Trimet data.
• if this works:
  • download OpenStreetMap file for Alberta, alberta.osm.bz2, from http://download.geofabrik.de/osm/north-america/canada/ [^] - and unzip: the alberta.osm file will be about 1.2 GB from 77 MB (while the or-wa.osm file is just about 390 MB) => hence, do unzipping on the server (where also the graph should be build) as transferring such a big file later is ... difficult (Btw. city extracts are available here: http://mapzen.com/metro-extracts/)
  • download the Google/General Transit Feed for Calgary. This "zip" package of several files is Available from the cities online store http://cityonline.calgary.ca/Pages/Category.aspx?cat=CITYonlineDefault&category=PublicData&publicdata [^] or from http://www.gtfs-data-exchange.com/
  • In the current install I have placed the alberta.osm file under /otp/cache/osm and the gtfs zip file under /otp/cache/gtfs (calling it gtcgy.zip)
  • to build the graph for the Calgary data you need to modify the xml file "graph-builder.xml". This can be done on the server using the "nano" editor (do a copy of the xml file first). You need to edit the file in 3 spots:
    • (a) adapt the gtfs file name and location (note, I actually switched from a www path to a local file path)
    • (b) change the transit entity name (defaultAgencyId = CAL)
    • (c) adapt the osm file name
    • now start the building with /otp/bin/build-graph.sh => check the outputs of the graph building process as errors may occur. If no errors occur (i.e. the program actually terminates) then you are lucky, however, I had messages that transfers exists between (bus)stops that were not defined. To fix that see the next point "Solving GTFS problems" below. Note, on the GISgroup server the graph building for the Calgary data took for me about 25 mins.
  • if the graph is build you can start OTP again using /otp/bin/start-server.sh & ...with "&" for keeping the process running if you want so.

Solving GTFS problems

Problem: The GTFS data from the city are not formatted in a way expected by the graph builder and in an older GTFS specification. My building process broke down in particular as in the transfers.txt file the transfers were defined between stops that have not been defined in stops.txt. Hence, I could add either fake stops or remove all invalid transfers ... afterwards, I found about 1500 invalid transfers. So, I decided to remove the invalid transfers.

Note: There are more problems in the data, for instance a wrong format and high-speed travels. You can identify those problems using Google Feed-Validator from: http://code.google.com/p/googletransitdatafeed/wiki/FeedValidator and looking into the GTFS specifications and examples here: http://developers.google.com/transit/gtfs/ It actually seems like OTP was happy with the data after the older specification, while the July 2012 Feed Validator complained about the transfer types etc.

Solution: I removed all transfers that had undefined bus stop ids. For this I programmed and adapted two Plugins in/for OpenJUMP GIS (available in the PYP Development folders on the SVN). To obtain all valid transfers do proceed as follows:

• unzip the GTFS zip file and extract the transfers.txt file and the stops.txt file
• open the transfers.txt and stops.txt file in a handy TextEditor (e.g. Text Wrangler on Mac or Notepad++ for Win), so you know what they look like.
• Start OpenJUMPs PLUS edition, that contains a CSV file loading option. Load the file "stops.txt" using "File>Open..." with the CSV option choosing the "set option" variant. In the setting you should change the field separator, define the x and y coordinate source as columns 6 & 5 (note the inverse order!), and maybe use UTF-8 as character encoding type. (Can't remember if anything else to set, but checkout yourself.) After the file has been opened you should see all the bus stops as dots; though, in geographic coordinates.
• Next you need to change the data type for the ids-containing columns/attributes from "String" to "Integer". Do this at least for the field "stop_id". Make the layer "editable" and use Schema>View/Edit Schema from the layer context menu. Before hitting "Apply changes" do check the box for "force invalid conversions...". Maybe save this file as jml or shp file.
• In the Text editor modify the file "transfers.txt" as follows:
  • (a) change the header/column names toward the names of the specification (but keep the order) .. you may actually check how my file looked like at the end
  • (b) remove all >"< using find/replace
  • (c) change all >,< to >;<
• save this file; probably under a new name, e.g.: transfers_mod.txt
• Go to OpenJUMP and start the function "PYP>Open CSV Table Data..." (I assume that you installed the OpenJUMP PYP extension (jar file) that you can get from our SVN). Browse to the modified transfers_mod.txt file and load it.
• Now start the function "PYP>GTFS - Match Transfers with Stops"
• assign the layers and attributes as asked for, and hit "ok"
• you should receive two new layers, one called "Transfers Missing Stops" and another called "Transfers Exiting". Note, you will actually also see some "unreasonable" transfers, that are some kms long. You could sort those data out, but here I will not describe this.
• select the layer "Transfers Existing" and chose from the layer context menu "Save Dataset As...". Choose the "CSV" format, change the field separator to "," and tick the box for "Select Attribute". Hit "ok", and in the next dialog unselect the X and Y options, and click "ok" again.
• go to the just written "Transfers Existing.csv" file. Rename the file to "transfers.txt" and copy it into the gtfs folder where the other files are (i.e. overwrite the old one). Then zip the folder (check for the correct name and structure within the zip file) and place the zipped GTFS file on the server for building the graph. Finally, build the graph as described above.

Note, I have said earlier that the Feed Validator from July 2012 complains about the use of invalid transfer types (e.g.: 7) and times (e.g.: -1). This issue I did not fix in my install, because it seems like OTP still knows what to do (maybe they changed the specifications recently). I.e. there are no complains during graph building. However, this needs to be checked/asked for newer OTP versions.

BTW: there is also a graph visualization tool by OTP (/bin/graph-viz.sh). Meanwhile I have also written and OpenJUMP plugin that can display the graph.

Install OTP on a VM with existing OTP files

Below I outline the process to install OTP on a VM on CedeusDB. I had already a copy of a previous OTP folder (used for graph compilation) and a graph folder.

VM setup

If OTP should run in a VM, e.g. from my Ubuntu 14.04 standard VM:

• follow the VM setup instructions from Setting up Elgg VM on CedeusGeoNode
• Besides access via http and ssh, I also added a TCP rule to access Tomcat later on via port 30088
• then install Tomcat 7:
  • one can check if tomcat stuff is there for instance with sudo find / -name "tomcat7"
  • some install instructions can be found here: http://www.digitalocean.com/community/tutorials/how-to-install-apache-tomcat-7-on-ubuntu-14-04-via-apt-get
  • very basic: sudo apt-get install tomcat7 tomcat7-admin
  • allow access to manage servlets:
    • sudo nano /etc/tomcat7/tomcat-users.xml
    • add something like <user username="admin" password="password" roles="manager-gui,admin-gui"/> (used here c...p)
    • restart tomcat: sudo service tomcat7 restart
  • check if it works : http://your_ip_address:8080 (or in my case 146.155.17.18:30088)
  • manage the apps: http://your_ip_address:8080/manager/html:
• change the tomcat memory allocation:
  • create setenv.sh in /usr/share/tomcat7/bin
  • add:

#!/bin/sh 
JAVA_OPTS="-Xms256m -Xmx4g -XX:MaxPermSize=270m -server -Djava.awt.headless=true" 
export JAVA_OPTS

• restart tomcat

Setup OTP

• ftp the OTP files to home
• create two folders under root:
  • /otp
  • /otpg
• copy the otp graph files to /otpg
• copy the otp binary files for graph building to /otp
• copy the Tomcat webapps under /otp/webapps to /var/lib/tomcat7/webapps/
• go to the admin app and update the list
• open the tomcat logs: tail -f /var/lib/tomcat7/logs/catalina.out
• restart the "opentripplanner-api-webapp" and check the logs
• e.g. I had to change the Graph path from /home/sstein/otpg to /otpg in the OTP data sources config file /var/lib/tomcat7/webapps/opentripplanner-api-webapp/WEB-INF/classes/data-sources.xml

Creating a new actual graph for Santiago

• download latest OSM file for Santiago from Mapzen (https://mapzen.com/data/metro-extracts) and transfer it to /otp/cache/osm
• download latest GTFS file from http://datos.gov.cl and transfer ot to /otp/cache/gtfs
• adjust the source location settings in the file /otp/graph-builder.xml
• switch to /otp folder
• run sh lib/build-graph.sh
  • check the assigned memory for building the graph (seems like I needed 3GB for Santiago)
  • perhaps run as sudo, so the otp is able to write the graph to the /otp folder
• otp did compile the graph now (17:17 - 17:27 on cedeusDB)
• copy the Graph.obj to /otpg folder and reload the "opentripplanner-api-webapp" in the manager or, alternatively, restart tomcat7

Accounting for DEM

To account for elevation in the routing, the graph building process requires the provision of a DEM (of sufficient accuracy).

I did the following steps so far:

(i) create a DEM:

• I got an ArcInfo binary grid from Andrew for Calgary, called NTS mosaic, see svn://subversion.engg.ucalgary.ca/hunter/PYP%20Data/City%20DEM [^]
• i converted the ArcInfo grid to a GeoTiff with GDAL in WGS84, since all the other data, i.e. street data from OSM and GTFS locations are in WGS84 = EPSG:4326
  • According to Andrew the DEM should have been in CRS EPSG:3776 = Alberta 3TM, ref merid 114 W
• the following commands are useful for the conversion (see als the txt file in the SVN folder):
  • converting ArcInfo binary grid to GeoTIFF for OpenTripPlanner + check the source file

    gdalinfo /Volumes/WINDOOF/otpdev/dem/DMTI_utmz11_30m/nts_mosaic/ 

  • do the file format transform

    gdal_translate -of GTiff /Volumes/WINDOOF/otpdev/dem/DMTI_utmz11_30m/nts_mosaic/ /Volumes/WINDOOF/otpdev/dem/cgydem_grs80deg.tif

  • check the output

    gdalinfo /Volumes/WINDOOF/otpdev/dem/cgydem_grs80deg.tif 

  • transfrom to WGS84 in lat/lon - note, NED DEM data are actually in NAD83 = EPSG:4269, i.e. not in WGS84

gdalwarp -t_srs "EPSG:4326" -r bilinear /Volumes/WINDOOF/otpdev/dem/cgydem_grs80deg.tif /Volumes/WINDOOF/otpdev/dem/cgydem_wgs84deg.tif

• afterwards I checked if our other city data overlay with the DEM in QGIS.

Alternatively it is possible to use a previously build OTP graph from the OSM data, and overlay it with the created GeoTIFF => this is actually best, as it also enables to see if some graph edges will not receive elevations (i.e. to see if the DEM fully covers the graph)

(ii) build the new graph

• put all the data in the appropriate Cache sub folders, i.e. the GeoTIFF in the otp/cache/NED folder
• modify the graph-building script (I attach my version for the GIScience server)
• start the build process and watch the output logs (building took me about 10mins)

(iii) use the new graph and test:

• shut the opentripplanner-webapp-api process down (with the tomcat manager app),
• replace the old Graph.obj file (on the GISc server it is in folder /otpg) with the newly build graph file
• restart the OTP api process and check if the new graph is loaded

(btw. catalina.out logging can be directed to the console by using: "tail -f catalina.out" or for Tomcat 7: "tail -f /var/log/tomcat7/catalina.out")

• start the OTP webapp to do a walk or bike routing; e.g. along the hill on 19th NW from Kensington Road to 16th Avenue.
• the routing result should show the elevation profile along the path (below the map)

Notes

Adding a new graph

The GTFS data have an expiry date. Hence, after download the new actual GTFS data, the graph needs to be rebuild.

Rebuilding can be done on the server (note: my local build version did not work for some reason). However, then - it seems - the service needs to be stopped and re-started to make it using the new Graph.obj file. Because the graph is usually kept in-memory. In my case I had to stop apache tomcat and then to restart (in /usr/local/tomcat). Earlier I was already wondering why I am still getting no routes and the app informs me that there are no valid schedule data. After restarting tomcat the problem was solved.

Adjusting OTP memory

#!/bin/sh 
JAVA_OPTS="-Xms512m -Xmx4g -XX:MaxPermSize=270m -server -Djava.awt.headless=true -Djava.util.prefs.systemRoot=$CATALINA_HOME/content/thredds/javaUtilPrefs" 
export JAVA_OPTS

graphbuilder.xml for including a DEM into build process

?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xmlns:context="http://www.springframework.org/schema/context"
    xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-2.5.xsd
    http://www.springframework.org/schema/context http://www.springframework.org/schema/context/spring-context-2.5.xsd">

    <bean id="gtfsBuilder" class="org.opentripplanner.graph_builder.impl.GtfsGraphBuilderImpl">
        <property name="gtfsBundles">
            <bean id="gtfsBundles" class="org.opentripplanner.graph_builder.model.GtfsBundles">
                <property name="bundles">
                    <list>
                        <bean class="org.opentripplanner.graph_builder.model.GtfsBundle">
                            <!-- property name="url" value="http://gisciencegroup.ucalgary.ca/gtcgy.zip" /-->
                            <property name="path" value="/otp/cache/gtfs/gtfscgymodwinter2012.zip" />
                            <property name="defaultAgencyId" value="CAL" />

                            <!-- By default, bikes may only be taken along on transit trips if the GTFS data allows them to be.
                                 If the GTFS data doesn't contain appropriate data, but the actual transit trips do allows bikes
                                 to be taken along, a defaultBikesAllowed property may be specified to allow bikes.
                             -->
                            <property name="defaultBikesAllowed" value="true" />
                        </bean>
                    </list>
                </property>
            </bean>
        </property>

        <!-- GTFS-rt ALERTS -->
        <property name="gtfsGraphBuilders">
            <list>
                <bean class="org.opentripplanner.graph_builder.impl.transit_index.TransitIndexBuilder" />
            </list>
        </property>
    </bean>


    <bean id="osmBuilder" class="org.opentripplanner.graph_builder.impl.osm.OpenStreetMapGraphBuilderImpl">
        <!-- Use an OSM provider that reads a file -->
        <property name="provider">
            <bean class="org.opentripplanner.openstreetmap.impl.AnyFileBasedOpenStreetMapProviderImpl">
                <property name="path" value="/otp/cache/osm/calgary.osm" />
            </bean>
        </property>

        <!-- Street Traversal Permisssion and Biking weighting is configurable, but we'll use the defaults for OSM...
 
             Street Traversal Permisssion: what modes are allowed on certain road/surface types (e.g., no cars on bike lanes, etc...)

             Bike weighting: the planner is using a 'shortest path' algorithm to choose one route from another.  To prefer certain paths
                             for biking (e.g., bike lanes), we can make the theoretical lenght of a street type shorter or longer, thus 
                             more or less attractive to the routing engine.  Weights of > 1.0 make a road type shorter (more attactive)

             @see http://wiki.openstreetmap.org/wiki/OSM_tags_for_routing/Access-Restrictions 
             @see https://github.com/openplans/OpenTripPlanner/blob/master/opentripplanner-graph-builder/src/main/java/org/opentripplanner/graph_builder/impl/osm/DefaultWayPropertySetSource.java
        -->
        <property name="defaultWayPropertySetSource">
            <bean class="org.opentripplanner.graph_builder.impl.osm.DefaultWayPropertySetSource" />
        </property>

        <!-- custom unamed street namer -->
        <property name="customNamer">
            <bean class="org.opentripplanner.graph_builder.impl.osm.PortlandCustomNamer" />
        </property>
    </bean>


    <bean id="nedBuilder" class="org.opentripplanner.graph_builder.impl.ned.NEDGraphBuilderImpl">
        <property name="gridCoverageFactory">
                <bean class="org.opentripplanner.graph_builder.impl.ned.GeotiffGridCoverageFactoryImpl">
                    <property name="path" value="/otp/cache/ned/cgydem_wgs84deg.tif" />
                </bean>
        </property>
    </bean>

    <bean id="transitStreetLink" class="org.opentripplanner.graph_builder.impl.TransitToStreetNetworkGraphBuilderImpl" />
    <!--
    <bean id="optimizeTransit" class="org.opentripplanner.graph_builder.impl.OptimizeTransitGraphBuilderImpl" />
    -->
    <bean id="checkGeometry" class="org.opentripplanner.graph_builder.impl.CheckGeometryGraphBuilderImpl" />
    <bean id="mapBuilder" class="org.opentripplanner.graph_builder.impl.map.MapBuilder"/>

    <bean id="graphBuilderTask" class="org.opentripplanner.graph_builder.GraphBuilderTask">
        <!-- typical config -->
        <property name="path" value="/otp" />

        <property name="graphBuilders">
            <list>
                <ref bean="gtfsBuilder" />
                <ref bean="osmBuilder" />
<!-- -->
                <ref bean="mapBuilder" />
<!-- -->
                <ref bean="checkGeometry" />
                <ref bean="transitStreetLink" />

                <ref bean="nedBuilder" />