default install docs and script to poetry based installs

This commit is contained in:
Blake Harnden 2020-07-14 10:42:24 -07:00
parent 6b5aaa6b19
commit dfb3e0c424
4 changed files with 175 additions and 637 deletions

View file

@ -5,102 +5,123 @@
## Overview ## Overview
This section will describe how to install CORE from source or from a pre-built package. CORE provides a script to help automate installing all required software
CORE has been vetted on Ubuntu 18 and CentOS 7.6. Other versions and distributions to build and run, including a python virtual environment to run it all in.
can work, assuming you can get the required packages and versions similar to those
noted below for the tested distributions.
> **NOTE:** iproute2 4.5+ is a requirement for bridge related commands The following tools will be leveraged during installation:
|Tool|Description|
|---|---|
|pip|used to install pipx|
|pipx|used to install standalone python tools (invoke, poetry)|
|invoke|used to run provided tasks (install, daemon, gui, tests, etc)|
|poetry|used to install the managed python virtual environment for running CORE|
## Required Hardware ## Required Hardware
Any computer capable of running Linux should be able to run CORE. Since the physical machine will be hosting numerous Any computer capable of running Linux should be able to run CORE. Since the physical machine will be hosting numerous
containers, as a general rule you should select a machine having as much RAM and CPU resources as possible. containers, as a general rule you should select a machine having as much RAM and CPU resources as possible.
## Operating System ## Supported Linux Distributions
CORE requires a Linux operating system because it uses namespacing provided by the kernel. It does not run on Plan is to support recent Ubuntu and CentOS LTS releases.
Windows or Mac OS X operating systems (unless it is running within a virtual machine guest.) The
technology that CORE currently uses is Linux network namespaces.
Ubuntu and CentOS Linux are the recommended distributions for running CORE. However, these distributions are Verified:
not strictly required. CORE will likely work on other flavors of Linux as well, assuming dependencies are met. * Ubuntu - 18.04, 20.04
* CentOS - 7.8, 8.0*
> **NOTE:** CORE Services determine what run on each node. You may require other software packages depending on the > **NOTE:** Ubuntu 20.04 requires installing legacy ebtables for WLAN
services you wish to use. For example, the HTTP service will require the apache2 package. > functionality
## Installed Files > **NOTE:** CentOS 8 does not provide legacy ebtables support, WLAN will not
> function properly
CORE files are installed to the following directories by default, when the installation prefix is **/usr**. ## Utility Requirements
Install Path | Description * iproute2 4.5+ is a requirement for bridge related commands
-------------|------------ * ebtables not backed by nftables
/usr/bin/core-gui|GUI startup command
/usr/bin/coretk-gui|BETA Python GUI
/usr/bin/core-daemon|Daemon startup command
/usr/bin/{core-cleanup, coresendmsg, core-manage}|Misc. helper commands/scripts
/usr/lib/core|GUI files
/usr/lib/python{3.6+}/dist-packages/core|Python modules for daemon/scripts
/etc/core/|Daemon and log configuration files
~/.core/|User-specific GUI preferences and scenario files
/usr/share/core/|Example scripts and scenarios
/usr/share/man/man1/|Command man pages
/etc/init.d/core-daemon|SysV startup script for daemon
/usr/lib/systemd/system/core-daemon.service|Systemd startup script for daemon
## Automated Install ## Automated Installation
There is a helper script in the root of the repository that can help automate > **NOTE:** installs OSPF MDR
the CORE installation. Some steps require commands be ran as sudo and you
will be prompted for a password. This should work on Ubuntu/CentOS and will > **NOTE:** sets up script files using the prefix provided
install system dependencies, python dependencies, and CORE. This will target
system installations of python 3.6. > **NOTE:** install a systemd service file to /lib/systemd/system/core-daemon.service
```shell ```shell
# clone CORE repo
git clone https://github.com/coreemu/core.git git clone https://github.com/coreemu/core.git
cd core cd core
# run install script
# script usage: install.sh [-d] [-v]
#
# -v enable verbose install
# -d enable developer install
# -p install prefix, defaults to /usr/local
./install.sh ./install.sh
``` ```
You can target newer system python versions using the **-v** flag. Assuming ## Manual Installation
these versions are actually available on your system.
> **NOTE:** install OSPF MDR by manual instructions below
```shell ```shell
# ubuntu 3.7 # clone CORE repo
./install.sh -v 3.7 git clone https://github.com/coreemu/core.git
# centos 3.7 cd core
./install.sh -v 37
# install python3 and venv support
# ubuntu
sudo apt install -y python3-pip python3-venv
# centos
sudo yum install -y python3-pip
# install system dependencies
# ubuntu
sudo apt install -y automake pkg-config gcc libev-dev ebtables iproute2 \
ethtool tk python3-tk
# centos
sudo yum install -y automake pkgconf-pkg-config gcc gcc-c++ libev-devel \
iptables-ebtables iproute python3-devel python3-tkinter tk ethtool \
make kernel-modules-extra
# install grpcio-tools
python3 -m pip install --user grpcio==1.27.2 grpcio-tools==1.27.2
# build core
./bootstrap.sh
# centos requires --prefix=/usr
./configure
make
sudo make install
# install pipx, may need to restart terminal after ensurepath
python3 -m pip install --user pipx
python3 -m pipx ensurepath
# install poetry
pipx install poetry
# install poetry virtual environment
cd daemon
poetry install --no-dev
cd ..
# install invoke to run helper tasks
pipx install invoke
# install core scripts leveraging poetry virtual environment
# centos requires --prefix=/usr
inv install-scripts
# optionally install systemd service file
# centos requires --prefix=/usr
inv install-service
``` ```
## Pre-Req Installing Python ## Manually Install OSPF MDR (Routing Support)
Python 3.6 is the minimum required python version. Newer versions can be used if available.
These steps are needed, since the system packages can not provide all the
dependencies needed by CORE.
### Ubuntu
```shell
sudo apt install python3.6
sudo apt install python3-pip
```
### CentOS
```shell
sudo yum install python36
sudo yum install python3-pip
```
### Dependencies
Install the current python dependencies.
```shell
sudo python3 -m pip install -r requirements.txt
```
## Pre-Req Installing OSPF MDR
Virtual networks generally require some form of routing in order to work (e.g. to automatically populate routing Virtual networks generally require some form of routing in order to work (e.g. to automatically populate routing
tables for routing packets from one subnet to another.) CORE builds OSPF routing protocol configurations by tables for routing packets from one subnet to another.) CORE builds OSPF routing protocol configurations by
@ -110,21 +131,14 @@ default when the blue router node type is used.
suite with a modified version of OSPFv3, optimized for use with mobile wireless networks. The **mdr** node type suite with a modified version of OSPFv3, optimized for use with mobile wireless networks. The **mdr** node type
(and the MDR service) requires this variant of Quagga. (and the MDR service) requires this variant of Quagga.
### Ubuntu
```shell ```shell
sudo apt install libtool gawk libreadline-dev # system dependencies
``` # ubuntu
sudo apt install -y libtool gawk libreadline-dev
# centos
sudo yum install -y libtool gawk readline-devel
### CentOS # build and install
```shell
sudo yum install libtool gawk readline-devel
```
### Build and Install
```shell
git clone https://github.com/USNavalResearchLaboratory/ospf-mdr git clone https://github.com/USNavalResearchLaboratory/ospf-mdr
cd ospf-mdr cd ospf-mdr
./bootstrap.sh ./bootstrap.sh
@ -135,167 +149,64 @@ make
sudo make install sudo make install
``` ```
Note that the configuration directory */usr/local/etc/quagga* shown for Quagga above could be */etc/quagga*, ## Manually Install EMANE
if you create a symbolic link from */etc/quagga/Quagga.conf -> /usr/local/etc/quagga/Quagga.conf* on the host.
The *quaggaboot.sh* script in a Linux network namespace will try and do this for you if needed.
If you try to run quagga after installing from source and get an error such as: EMANE can be installed from deb or RPM packages or from source. See the
[EMANE GitHub](https://github.com/adjacentlink/emane) for full details.
Here are quick instructions for installing all EMANE packages for Ubuntu 18.04:
```shell ```shell
error while loading shared libraries libzebra.so.0 # install dependencies
# ubuntu
sudo apt-get install libssl-dev libxml-libxml-perl libxml-simple-perl
wget https://adjacentlink.com/downloads/emane/emane-1.2.5-release-1.ubuntu-18_04.amd64.tar.gz
tar xzf emane-1.2.5-release-1.ubuntu-18_04.amd64.tar.gz
# install base emane packages
sudo dpkg -i emane-1.2.5-release-1/deb/ubuntu-18_04/amd64/emane*.deb
# install python3 bindings
sudo dpkg -i emane-1.2.5-release-1/deb/ubuntu-18_04/amd64/python3*.deb
``` ```
this is usually a sign that you have to run ```sudo ldconfig```` to refresh the cache file. ## Using Invoke Tasks
## Installing from Packages The invoke tool installed by way of pipx provides conveniences for running
CORE tasks to help ensure usage of the create python virtual environment.
The easiest way to install CORE is using the pre-built packages. The package managers on Ubuntu or CentOS
will help in automatically installing most dependencies, except for the python ones described previously.
You can obtain the CORE packages from [CORE Releases](https://github.com/coreemu/core/releases).
### Ubuntu
Ubuntu package defaults to using systemd for running as a service.
```shell ```shell
sudo apt install ./core_$VERSION_amd64.deb Available tasks:
cleanup run core-cleanup removing leftover core nodes, bridges, directories
cli run core-cli used to query and modify a running session
daemon start core-daemon
gui start core-pygui
install install core, poetry, scripts, service, and ospf mdr
install-scripts install core script files, modified to leverage virtual environment
install-service install systemd core service
test run core tests
test-emane run core emane tests
test-mock run core tests using mock to avoid running as sudo
uninstall uninstall core
``` ```
### CentOS Example running the core-daemon task from the root of the repo:
```shell
inv daemon
```
**NOTE: tkimg is not required for the core-gui, but if you get an error message about it you can install the package Some tasks are wrappers around command line tools and requires running
on CentOS <= 6, or build from source otherwise** them with a slight variation for compatibility. You can enter the
poetry shell to run the script natively.
```shell ```shell
yum install ./core_$VERSION_x86_64.rpm # running core-cli as a task requires all options to be provided
``` # within a string
inv cli "query session -i 1"
Disabling SELINUX:
# entering the poetry shell to use core-cli natively
```shell cd $REPO/daemon
# change the following in /etc/sysconfig/selinux poetry shell
SELINUX=disabled core-cli query session -i 1
# add the following to the kernel line in /etc/grub.conf # exit the shell
selinux=0 exit
```
Turn off firewalls:
```shell
systemctl disable firewalld
systemctl disable iptables.service
systemctl disable ip6tables.service
chkconfig iptables off
chkconfig ip6tables off
```
You need to reboot after making these changes, or flush the firewall using
```shell
iptables -F
ip6tables -F
```
## Installing from Source
Steps for building from cloned source code. Python 3.6 is the minimum required version
a newer version can be used below if available.
### Distro Requirements
System packages required to build from source.
#### Ubuntu
```shell
sudo apt install git automake pkg-config gcc libev-dev ebtables iproute2 \
python3.6 python3.6-dev python3-pip python3-tk tk libtk-img ethtool autoconf
```
#### CentOS
```shell
sudo yum install git automake pkgconf-pkg-config gcc gcc-c++ libev-devel iptables-ebtables iproute \
python36 python36-devel python3-pip python3-tkinter tk ethtool autoconf
```
### Clone Repository
Clone the CORE repository for building from source.
```shell
git clone https://github.com/coreemu/core.git
```
### Install grpcio-tools
Python module grpcio-tools is currently needed to generate gRPC protobuf code.
Specifically leveraging 1.27.2 as that is what will be used during runtime.
```shell
python3 -m pip install --user grpcio==1.27.2 grpcio-tools==1.27.2
```
### Build and Install
```shell
./bootstrap.sh
./configure
make
sudo make install
```
## Building Documentation
Building documentation requires python-sphinx not noted above.
```shell
sudo apt install python3-sphinx
sudo yum install python3-sphinx
./bootstrap.sh
./configure
make doc
```
## Building Packages
Build package commands, DESTDIR is used to make install into and then for packaging by fpm.
**NOTE: clean the DESTDIR if re-using the same directory**
* Install [fpm](http://fpm.readthedocs.io/en/latest/installing.html)
```shell
./bootstrap.sh
./configure
make
mkdir /tmp/core-build
make fpm DESTDIR=/tmp/core-build
```
This will produce and RPM and Deb package for the currently configured python version.
## Running CORE
Start the CORE daemon.
```shell
# systemd
sudo systemctl daemon-reload
sudo systemctl start core-daemon
# sysv
sudo service core-daemon start
```
Run the GUI
```shell
# default gui
core-gui
# new beta gui
coretk-gui
``` ```

View file

@ -1,212 +0,0 @@
# CORE Installation
* Table of Contents
{:toc}
## Overview
CORE provides a script to help automate installing all required software
to build and run, including a python virtual environment to run it all in.
The following tools will be leveraged during installation:
|Tool|Description|
|---|---|
|pip|used to install pipx|
|pipx|used to install standalone python tools (invoke, poetry)|
|invoke|used to run provided tasks (install, daemon, gui, tests, etc)|
|poetry|used to install the managed python virtual environment for running CORE|
## Required Hardware
Any computer capable of running Linux should be able to run CORE. Since the physical machine will be hosting numerous
containers, as a general rule you should select a machine having as much RAM and CPU resources as possible.
## Supported Linux Distributions
Plan is to support recent Ubuntu and CentOS LTS releases.
Verified:
* Ubuntu - 18.04, 20.04
* CentOS - 7.8, 8.0*
> **NOTE:** Ubuntu 20.04 requires installing legacy ebtables for WLAN
> functionality
> **NOTE:** CentOS 8 does not provide legacy ebtables support, WLAN will not
> function properly
## Utility Requirements
* iproute2 4.5+ is a requirement for bridge related commands
* ebtables not backed by nftables
## Automated Installation
> **NOTE:** installs OSPF MDR
> **NOTE:** sets up script files using the prefix provided
> **NOTE:** install a systemd service file to /lib/systemd/system/core-daemon.service
```shell
# clone CORE repo
git clone https://github.com/coreemu/core.git
cd core
# run install script
# script usage: install.sh [-d] [-v]
#
# -v enable verbose install
# -d enable developer install
# -p install prefix, defaults to /usr/local
./install.sh
```
## Manual Installation
> **NOTE:** install OSPF MDR by manual instructions below
```shell
# clone CORE repo
git clone https://github.com/coreemu/core.git
cd core
# install python3 and venv support
# ubuntu
sudo apt install -y python3-pip python3-venv
# centos
sudo yum install -y python3-pip
# install system dependencies
# ubuntu
sudo apt install -y automake pkg-config gcc libev-dev ebtables iproute2 \
ethtool tk python3-tk
# centos
sudo yum install -y automake pkgconf-pkg-config gcc gcc-c++ libev-devel \
iptables-ebtables iproute python3-devel python3-tkinter tk ethtool \
make kernel-modules-extra
# install grpcio-tools
python3 -m pip install --user grpcio==1.27.2 grpcio-tools==1.27.2
# build core
./bootstrap.sh
# centos requires --prefix=/usr
./configure
make
sudo make install
# install pipx, may need to restart terminal after ensurepath
python3 -m pip install --user pipx
python3 -m pipx ensurepath
# install poetry
pipx install poetry
# install poetry virtual environment
cd daemon
poetry install --no-dev
cd ..
# install invoke to run helper tasks
pipx install invoke
# install core scripts leveraging poetry virtual environment
# centos requires --prefix=/usr
inv install-scripts
# optionally install systemd service file
# centos requires --prefix=/usr
inv install-service
```
## Manually Install OSPF MDR (Routing Support)
Virtual networks generally require some form of routing in order to work (e.g. to automatically populate routing
tables for routing packets from one subnet to another.) CORE builds OSPF routing protocol configurations by
default when the blue router node type is used.
* [OSPF MANET Designated Routers](https://github.com/USNavalResearchLaboratory/ospf-mdr) (MDR) - the Quagga routing
suite with a modified version of OSPFv3, optimized for use with mobile wireless networks. The **mdr** node type
(and the MDR service) requires this variant of Quagga.
```shell
# system dependencies
# ubuntu
sudo apt install -y libtool gawk libreadline-dev
# centos
sudo yum install -y libtool gawk readline-devel
# build and install
git clone https://github.com/USNavalResearchLaboratory/ospf-mdr
cd ospf-mdr
./bootstrap.sh
./configure --disable-doc --enable-user=root --enable-group=root --with-cflags=-ggdb \
--sysconfdir=/usr/local/etc/quagga --enable-vtysh \
--localstatedir=/var/run/quagga
make
sudo make install
```
## Manually Install EMANE
EMANE can be installed from deb or RPM packages or from source. See the
[EMANE GitHub](https://github.com/adjacentlink/emane) for full details.
Here are quick instructions for installing all EMANE packages for Ubuntu 18.04:
```shell
# install dependencies
# ubuntu
sudo apt-get install libssl-dev libxml-libxml-perl libxml-simple-perl
wget https://adjacentlink.com/downloads/emane/emane-1.2.5-release-1.ubuntu-18_04.amd64.tar.gz
tar xzf emane-1.2.5-release-1.ubuntu-18_04.amd64.tar.gz
# install base emane packages
sudo dpkg -i emane-1.2.5-release-1/deb/ubuntu-18_04/amd64/emane*.deb
# install python3 bindings
sudo dpkg -i emane-1.2.5-release-1/deb/ubuntu-18_04/amd64/python3*.deb
```
## Using Invoke Tasks
The invoke tool installed by way of pipx provides conveniences for running
CORE tasks to help ensure usage of the create python virtual environment.
```shell
Available tasks:
cleanup run core-cleanup removing leftover core nodes, bridges, directories
cli run core-cli used to query and modify a running session
daemon start core-daemon
gui start core-pygui
install install core, poetry, scripts, service, and ospf mdr
install-scripts install core script files, modified to leverage virtual environment
install-service install systemd core service
test run core tests
test-emane run core emane tests
test-mock run core tests using mock to avoid running as sudo
uninstall uninstall core
```
Example running the core-daemon task from the root of the repo:
```shell
inv daemon
```
Some tasks are wrappers around command line tools and requires running
them with a slight variation for compatibility. You can enter the
poetry shell to run the script natively.
```shell
# running core-cli as a task requires all options to be provided
# within a string
inv cli "query session -i 1"
# entering the poetry shell to use core-cli natively
cd $REPO/daemon
poetry shell
core-cli query session -i 1
# exit the shell
exit
```

View file

@ -3,57 +3,6 @@
# exit on error # exit on error
set -e set -e
ubuntu_py=3.6
centos_py=36
reinstall=
function install_python_depencencies() {
sudo python3 -m pip install -r daemon/requirements.txt
}
function install_grpcio_tools() {
python3 -m pip install --only-binary ":all:" --user grpcio-tools
}
function install_ospf_mdr() {
rm -rf /tmp/ospf-mdr
git clone https://github.com/USNavalResearchLaboratory/ospf-mdr /tmp/ospf-mdr
cd /tmp/ospf-mdr
./bootstrap.sh
./configure --disable-doc --enable-user=root --enable-group=root --with-cflags=-ggdb \
--sysconfdir=/usr/local/etc/quagga --enable-vtysh \
--localstatedir=/var/run/quagga
make -j8
sudo make install
cd -
}
function build_core() {
./bootstrap.sh
./configure $1
make -j8
}
function install_core() {
sudo make install
}
function uninstall_core() {
sudo make uninstall
make clean
./bootstrap.sh clean
}
function install_dev_core() {
cd gui
sudo make install
cd -
cd netns
sudo make install
cd -
cd daemon
}
# detect os/ver for install type # detect os/ver for install type
os="" os=""
if [[ -f /etc/os-release ]]; then if [[ -f /etc/os-release ]]; then
@ -62,100 +11,47 @@ if [[ -f /etc/os-release ]]; then
fi fi
# parse arguments # parse arguments
while getopts "drv:" opt; do dev=""
verbose=""
prefix=""
while getopts "dvp:" opt; do
case ${opt} in case ${opt} in
d) d)
dev=1 dev="-d"
;; ;;
v) v)
ubuntu_py=${OPTARG} verbose="-v"
centos_py=${OPTARG}
;; ;;
r) p)
reinstall=1 prefix="-p ${OPTARG}"
;; ;;
\?) \?)
echo "script usage: $(basename $0) [-d] [-r] [-v python version]" >&2 echo "script usage: $(basename $0) [-d] [-v]" >&2
echo "" >&2
echo "-v enable verbose install" >&2
echo "-d enable developer install" >&2
echo "-p install prefix, defaults to /usr/local" >&2
exit 1 exit 1
;; ;;
esac esac
done done
shift $((OPTIND - 1)) shift $((OPTIND - 1))
# check if we are reinstalling or installing echo "installing CORE for ${os}"
if [ -z "${reinstall}" ]; then case ${os} in
echo "installing CORE for ${os}" "ubuntu")
case ${os} in sudo apt install -y python3-pip python3-venv
"ubuntu") ;;
echo "installing core system dependencies" "centos")
sudo apt install -y automake pkg-config gcc libev-dev ebtables iproute2 \ sudo yum install -y python3-pip
python${ubuntu_py} python${ubuntu_py}-dev python3-pip python3-tk tk libtk-img ethtool autoconf ;;
install_grpcio_tools *)
echo "installing ospf-mdr system dependencies" echo "unknown OS ID ${os} cannot install"
sudo apt install -y libtool gawk libreadline-dev ;;
install_ospf_mdr esac
if [[ -z ${dev} ]]; then
echo "normal install" python3 -m pip install --user pipx
install_python_depencencies python3 -m pipx ensurepath
build_core export PATH=$PATH:~/.local/bin
install_core pipx install invoke
else inv install ${dev} ${verbose} ${prefix}
echo "dev install"
python3 -m pip install pipenv
build_core
install_dev_core
python3 -m pipenv sync --dev
python3 -m pipenv run pre-commit install
fi
;;
"centos")
echo "installing core system dependencies"
sudo yum install -y automake pkgconf-pkg-config gcc gcc-c++ libev-devel iptables-ebtables iproute \
python${centos_py} python${centos_py}-devel python3-pip python3-tkinter tk ethtool autoconf
install_grpcio_tools
echo "installing ospf-mdr system dependencies"
sudo yum install -y libtool gawk readline-devel
install_ospf_mdr
if [[ -z ${dev} ]]; then
echo "normal install"
install_python_depencencies
build_core --prefix=/usr
install_core
else
echo "dev install"
sudo python3 -m pip install pipenv
build_core --prefix=/usr
install_dev_core
sudo python3 -m pipenv sync --dev
python3 -m pipenv sync --dev
python3 -m pipenv run pre-commit install
fi
;;
*)
echo "unknown OS ID ${os} cannot install"
;;
esac
else
branch=$(git symbolic-ref --short HEAD)
echo "reinstalling CORE on ${os} with latest ${branch}"
echo "uninstalling CORE"
uninstall_core
echo "pulling latest code"
git pull
echo "installing python dependencies"
install_python_depencencies
echo "building CORE"
case ${os} in
"ubuntu")
build_core
;;
"centos")
build_core --prefix=/usr
;;
*)
echo "unknown OS ID ${os} cannot reinstall"
;;
esac
echo "installing CORE"
install_core
fi

View file

@ -1,57 +0,0 @@
#!/bin/bash
# exit on error
set -e
# detect os/ver for install type
os=""
if [[ -f /etc/os-release ]]; then
. /etc/os-release
os=${ID}
fi
# parse arguments
dev=""
verbose=""
prefix=""
while getopts "dvp:" opt; do
case ${opt} in
d)
dev="-d"
;;
v)
verbose="-v"
;;
p)
prefix="-p ${OPTARG}"
;;
\?)
echo "script usage: $(basename $0) [-d] [-v]" >&2
echo "" >&2
echo "-v enable verbose install" >&2
echo "-d enable developer install" >&2
echo "-p install prefix, defaults to /usr/local" >&2
exit 1
;;
esac
done
shift $((OPTIND - 1))
echo "installing CORE for ${os}"
case ${os} in
"ubuntu")
sudo apt install -y python3-pip python3-venv
;;
"centos")
sudo yum install -y python3-pip
;;
*)
echo "unknown OS ID ${os} cannot install"
;;
esac
python3 -m pip install --user pipx
python3 -m pipx ensurepath
export PATH=$PATH:~/.local/bin
pipx install invoke
inv install ${dev} ${verbose} ${prefix}