Difference between revisions of "Installing CartoDB for CEDEUS"

Revision as of 17:23, 27 October 2015

>> return to Cedeus IDE

Installing CartoDB

Setup of CartoDB VM

Setup of VM based on Ubuntu 1204 VM

1. copying tilestream.vdi file (we need Ubuntu 12.04!) and renaming to cartodb.vdi
2. creating the cartodb VM on CedeusDB (ip.18):
   • VBoxManage createvm --name cartodb --ostype Ubuntu_64 --register
   • VBoxManage modifyvm cartodb --memory 8096
   • VBoxManage modifyvm cartodb --cpus 6
   • VBoxManage modifyvm cartodb --nic1 nat
   • VBoxManage storagectl cartodb --name "SATA Controller" --add sata --controller IntelAhci
   • assign the (old) disk image a new uuid

     VBoxManage internalcommands sethduuid cartodb.vdi

   • attach the (old) disk image:

     VBoxManage storageattach "cartodb" --storagectl "SATA Controller" --port 0 --device 0 --type hdd --medium cartodb.vdi

   • VBoxManage storagectl cartodb --name "IDE Controller" --add ide --controller PIIX4
   • set the nat rules (ports):

     VBoxManage modifyvm cartodb --natpf1 "ssh,tcp,,19022,,22"

     VBoxManage modifyvm cartodb --natpf1 "apache,tcp,,19080,,80"

     VBoxManage modifyvm cartodb --natpf1 "pgsql,tcp,,19432,,5432"

3. optional - set VRDE port:
   • VBoxManage modifyvm cartodb --vrdeport 7763 (Note: 3389 is the default port anyway)
4. check all settings with VBoxManage showvminfo cartodb
5. start the VM
   • VBoxHeadless -s cartodb --vrde on &

   the VM will listen on port 7763 - connect with GTKvncViewer to check

   • open /etc/udev/rules.d/70-persistent-net.rules and comment out existing network adapters
   • reboot
   • check if ssh connection works
6. change the computers name in /etc/hosts and /etc/hostname
7. restarting Apache gives the error message: "apache2: Could not reliably determine the server's fully qualified domain name, using 127.0.1.1 ..."

   => solve this by adding ServerName cartodb as the last line in /etc/apache2/apache2.conf file

Resize the CartoDB VM

SINCE CartoDB seems to require Ubuntu 12.04 I did NOT do the following with the tilestream.vdi based disk:

• shut the VM down to resize the HDD if necessary
• check the HDD size: VBoxManage showhdinfo cartodb.vdi returns:

UUID:           e148d758-00ac-4b60-a067-d5c97316069d
Parent UUID:    base</br>
State:          created
Type:           normal (base)
Location:       /home/ssteinig/cartodb.vdi
Storage format: VDI
Format variant: dynamic default
Capacity:       100000 MBytes
Size on disk:   6224 MBytes
In use by VMs:  nominatim (UUID: be795392-3786-4f3b-9714-a9445f91855d)

• so I resized the disk to 200GB as, according to Nominatim infos the disk size for a full planet is 700GB

  VBoxManage modifyhd cartodb.vdi --resize 200000

• to make this resizing effective I actually have to change the partitions:
  • create a zip-copy of the vdi file,
  • download & start the VM with Ubuntu Desktop(!) 14.04 iso

  VBoxManage storageattach cartodb --storagectl "IDE Controller" --port 1 --device 0 --type dvddrive --medium ubuntu-14.04.2-desktop-amd64.iso

  • choose the "Try Ubuntu" mode
  • use the program "gparted" to adjust/extend the partition's size
  • to be able to enlarge "extended" partition, on needs to click on the key symbol and chose "deactivate", as described for instance here: http://www.howtogeek.com/114503/how-to-resize-your-ubuntu-partitions/
  • on 12.04 when extending GeoNode1204 VM: one may also need to extend /dev/sda2 (cyan color) first before extending /dev/sda5 (brownish color)
  • remove the "iso" and restart the Vm; check size if the disks is really 200GB now

  VBoxManage storageattach cartodb --storagectl "IDE Controller" --port 1 --device 0 --type dvddrive --medium none

  • => Well it turned out that I used LVM (Logical Volume Manager) so the new space is not taken into account. Some more stuff needs to be adjusted!
    • after a bit of reading here: http://www.rootusers.com/how-to-increase-the-size-of-a-linux-lvm-by-expanding-the-virtual-machine-disk/ and here: http://www.thegeekstuff.com/2010/08/how-to-create-lvm/ and checking the status with commands such as pvscan, pvdisplay and lvdisplay; I figured that I simply need to extend the size of the logical volume (see also http://serverfault.com/questions/501895/used-vgextend-lvextend-to-add-addtional-8gb-space-but-it-is-not-reflected-in-df ) This is done with lvextend - so I used:
    • sudo lvextend -l +100%FREE /dev/basicubuntu1404-vg/root
    • sudo resize2fs /dev/mapper/basicubuntu1404--vg-root

    => this extended the disk to 187GB (see with sudo lvdisplay). The second command is needed to have this really available (see with df -h or sudo fdisk -l)

Setup of CartoDB itself

• General install instructions for 12.04 are here: https://github.com/CartoDB/cartodb
• launchpad (precise): http://ppa.launchpad.net/cartodb/base/ubuntu/dists/
• Google email Group: https://groups.google.com/forum/#!forum/cartodb

Had the following errors when executing

• sudo PGUSER=postgres make installcheck

The following tests are failing: test_cdb_querytables_returns_schema_and_table_name

• on pip install --no-use-wheel -r python_requirements.txt

/usr/local/lib/python2.7/dist-packages/pip-7.1.0-py2.7.egg/pip/_vendor/requests/packages/urllib3/util/ssl_.py:90: 
InsecurePlatformWarning: A true  SSLContext object is not available.
This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. For more information, see  
https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.
 InsecurePlatformWarning
Could not open requirements file: [Errno 2] No such file or directory: 'python_requirements.txt'

• on installing Windshaft and doing npm install

No package 'pangocairo' found
gyp: Call to 'pkg-config pangocairo --libs' returned exit status 1. while trying to load binding.gyp
gyp ERR! configure error 
gyp ERR! stack Error: `gyp` failed with exit code: 1
gyp ERR! stack     at ChildProcess.onCpExit (/usr/lib/node_modules/npm/node_modules/node-gyp/lib/configure.js:343:16)

....and later:

Success: "/usr/include/CartoDB-SQL-API/Windshaft-cartodb/node_modules/windshaft/node_modules/grainstore/node_modules/millstone/node_modules/srs/lib /binding/node-v11-linux-x64/srs.node" is installed via remote
npm ERR! canvas@1.2.1 install: `node-gyp rebuild`
npm ERR! Exit status 1
npm ERR! 
npm ERR! Failed at the canvas@1.2.1 install script.
npm ERR! This is most likely a problem with the canvas package,
npm ERR! not with npm itself.

Trabajo Cristian

Trabajo basado en el tutorial de arjendk en https://gist.github.com/arjendk/6080388 Fue posible ejecutar una máquina virtual con Ubuntu 14.04.3x64 con las configuraciones básicas de cartoDB siguiendo los siguientes pasos de instalación:

• Copia de archivos de cartoDB

 sudo apt-get update
 sudo apt-get install git
 cd ~
 git clone --recursive https://github.com/CartoDB/cartodb20.git
 cd cartodb20/
 git checkout master

• Tomar café...

• Agregar sources para repositorios

 sudo apt-get install -y software-properties-common python-software-properties
 sudo add-apt-repository -y ppa:cartodb/gis
 sudo add-apt-repository -y ppa:cartodb/nodejs
 sudo add-apt-repository -y ppa:cartodb/redis
 sudo add-apt-repository -y ppa:cartodb/postgresql
 sudo add-apt-repository -y ppa:mapnik/v2.2.0

• En /etc/apt/sources.list.d/ cambiar sources a "precise" con gis, nodejs y postgresql, y en "lucid" para redis.

 sudo apt-get update

• Instalación de PostgreSQL, PostGIS y sus dependencias

 sudo apt-get install -y unp vim zip libgeos-c1 libgeos-dev gdal-bin libgdal1-dev libjson0 python-simplejson libjson0-dev proj-bin proj-data libproj-dev
 sudo apt-get install -y postgresql-9.1 postgresql-client-9.1 postgresql-contrib-9.1 postgresql-server-dev-9.1
 sudo apt-get install -y postgresql-plpython-9.1
 sudo apt-get install -y make

 cd /usr/local/src
 sudo wget http://download.osgeo.org/postgis/source/postgis-2.0.2.tar.gz
 sudo tar xzf postgis-2.0.2.tar.gz
 cd postgis-2.0.2
 ./configure --with-raster --with-topology
 make
 sudo make install

• Crear archivo para la configuración de template para postGIS, crear archivo en directorio /var/lib/postgresql/template_postgis.sh con la siguiente información:

 #!/usr/bin/env bash
 POSTGIS_SQL_PATH=`pg_config --sharedir`/contrib/postgis-2.0
 createdb -E UTF8 template_postgis
 createlang -d template_postgis plpgsql
 psql -d postgres -c "UPDATE pg_database SET datistemplate='true' WHERE datname='template_postgis'"
 psql -d template_postgis -f $POSTGIS_SQL_PATH/postgis.sql
 psql -d template_postgis -f $POSTGIS_SQL_PATH/spatial_ref_sys.sql
 psql -d template_postgis -f $POSTGIS_SQL_PATH/legacy.sql
 psql -d template_postgis -f $POSTGIS_SQL_PATH/rtpostgis.sql
 psql -d template_postgis -f $POSTGIS_SQL_PATH/topology.sql
 psql -d template_postgis -c "GRANT ALL ON geometry_columns TO PUBLIC;"
 psql -d template_postgis -c "GRANT ALL ON spatial_ref_sys TO PUBLIC;"

• Dar permisos de ejecución al archivo y luego ejecutar como usuario postgres

 sudo chmod +x /var/lib/postgresql/template_postgis.sh
 sudo -u postgres -i
 ./template_postgis.sh
 exit
 cd ~

• Instalar Ruby mediante rvm

 sudo apt-get install -y curl
 gpg --keyserver hkp://keys.gnupg.net --recv-keys 409B6B1796C275462A1703113804BB82D39DC0E3
 curl -L https://get.rvm.io | bash -s stable
 source /home/<user>/.rvm/scripts/rvm
 rvm requirements
 rvm install 1.9.3
 rvm use ruby-1.9.3-p551@cartodb --create

• Instalar Redis y Varnish y sus dependencias

 sudo apt-get install -y nodejs npm redis-server
 sudo apt-get install -y python-setuptools
 sudo apt-get install -y python-dev
 sudo apt-get install -y python-gdal
 sudo easy_install pip

 cd ~/cartodb20/

• Editar el archivo python_requirements.txt y borrar gdal=1.10.0 (ya se instaló en pasos anteriores)

 sudo pip install -r python_requirements.txt
 sudo pip install -e git+https://github.com/RealGeeks/python-varnish.git@0971d6024fbb2614350853a5e0f8736ba3fb1f0d#egg=python-varnish