afonso/core-extra
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity

updated install docs to try and be more explicit, updated install tasks to allow forcing an install type override for unsupported OSs

Browse source
This commit is contained in:
Blake Harnden 2020-09-26 09:42:15 -07:00
parent 0306f77147
commit a3d2e6dfe3
2 changed files with 187 additions and 90 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

159
docs/install.md
View file

@ -1,29 +1,70 @@
# Installation
* Table of Contents
{:toc}
## Overview
CORE provides a script to help automate the installation of dependencies,
build and install, and either generate a CORE specific python virtual environment
or build and install a python wheel.
CORE provides a script to help automate installing all required software
to build and run, including a python virtual environment to run it all in.
### Tools Used
The following tools will be leveraged during installation:
|Tool|Description|
|---|---|
|[pip](https://pip.pypa.io/en/stable/)|used to install pipx|
|[pipx](https://pipxproject.github.io/pipx/)|used to install standalone python tools (invoke, poetry)|
|[invoke](http://www.pyinvoke.org/)|used to run provided tasks (install, daemon, gui, tests, etc)|
|[poetry](https://python-poetry.org/)|used to install the managed python virtual environment for running CORE|
|[invoke](http://www.pyinvoke.org/)|used to run provided tasks (install, uninstall, reinstall, etc)|
|[poetry](https://python-poetry.org/)|used to install python virtual environment or building a python wheel|
## Required Hardware
### Files
The following is a list of files that would be installed after running the automated installation.
> **NOTE:** the default install prefix is /usr/local, but can be changed as noted below
* executable files
* <prefix>/bin/{core-daemon, core-gui, vcmd, vnoded, etc}
* tcl/tk gui files
* <prefix>/lib/core
* <prefix>/share/core/icons
* example imn files
* <prefix>/share/core/examples
* python files
* poetry virtual env
* `cd <repo>/daemon && poetry env info`
* ~/.cache/pypoetry/virtualenvs/
* local python install
* default install path for python3 installation of a wheel
* `python3 -c "import core; print(core.__file__)"`
* configuration files
* /etc/core/{core.conf, logging.conf}
* ospf mdr repository files
* <repo>/../ospf-mdr
* emane repository files
* <repo>/../emane
### Installed Executables
After the installation complete it will have installed the following scripts.
| Name | Description |
|---|---|
| core-cleanup | tool to help removed lingering core created containers, bridges, directories |
| core-cli | tool to query, open xml files, and send commands using gRPC |
| core-daemon | runs the backed core server providing TLV and gRPC APIs |
| core-gui | runs the legacy tcl/tk based GUI |
| core-imn-to-xml | tool to help automate converting a .imn file to .xml format |
| core-manage | tool to add, remove, or check for services, models, and node types |
| core-pygui | runs the new python/tk based GUI |
| core-python | provides a convenience for running the core python virtual environment |
| core-route-monitor | tool to help monitor traffic across nodes and feed that to SDT |
| core-service-update | tool to update automate modifying a legacy service to match current naming |
| coresendmsg | tool to send TLV API commands from command line |
### Required Hardware
Any computer capable of running Linux should be able to run CORE. Since the physical machine will be hosting numerous
containers, as a general rule you should select a machine having as much RAM and CPU resources as possible.
## Supported Linux Distributions
### Supported Linux Distributions
Plan is to support recent Ubuntu and CentOS LTS releases.
Verified:
@ -46,14 +87,14 @@ sudo yum install -y kernel-modules-extra
sudo modprobe sch_netem
```
## Utility Requirements
### Utility Requirements
The following are known dependencies that will result in errors when not met.
* iproute2 4.5+ is a requirement for bridge related commands
* ebtables not backed by nftables
## Upgrading
Please make sure to uninstall the previous installation of CORE cleanly
## Upgrading from Older Release
Please make sure to uninstall any previous installations of CORE cleanly
before proceeding to install.
Previous install was built from source:
@ -72,92 +113,97 @@ sudo yum remove core
sudo apt remove core
```
## Automated Installation
> **NOTE:** installing globally can have issues with dependency conflicts etc
The automated install will install do the following:
## Automated Install
The automated install will do the following:
* install base tools needed for installation
* python3, pip, pipx, invoke, poetry
* installs system dependencies for building core
* installs latest version of [OPSF MDR](https://github.com/USNavalResearchLaboratory/ospf-mdr)
* clone/build/install working version of [OPSF MDR](https://github.com/USNavalResearchLaboratory/ospf-mdr)
* installs core into poetry managed virtual environment or locally, if flag is passed
* installs scripts pointing to python interpreter being used
* installs systemd service, disabled by default
* installs scripts pointing pointing to appropriate python location based on install type
* installs systemd service pointing to appropriate python location based on install type
After installation has completed you should be able to run the various
CORE scripts for running core.
After installation has completed you should be able to run `core-daemon` and `core-gui`.
> **NOTE:** provide a prefix that will be found on path when running as sudo
> if the default prefix is not valid
> **NOTE:** installing locally comes with its own risks, it can result it potential
> dependency conflicts with system package manager installed python dependencies
> **NOTE:** provide a prefix that will be found on path when running as sudo,
> if the default prefix /usr/local will not be valid
`install.sh` will attempt to determine your OS by way of `/etc/os-release`, currently it supports
attempts to install OSs that are debian/redhat like (yum/apt).
```shell
# clone CORE repo
git clone https://github.com/coreemu/core.git
cd core
# run install script
# script usage: install.sh [-v] [-d] [-l] [-p <prefix>]
#
# -v enable verbose install
# -d enable developer install
# -l enable local install, not compatible with developer install
# -p install prefix, defaults to /usr/local
./install.sh
# install core to virtual environment
./install.sh -p <prefix>
# install core locally
./install.sh -p <prefix> -l
```
### Unsupported Linux Distribution
For unsupported OSs you could attempt to do the following to translate
an installation to your use case.
If you are on an unsupported distribution, you can look into the
[install.sh](https://github.com/coreemu/core/blob/master/install.sh)
and
[tasks.py](https://github.com/coreemu/core/blob/master/tasks.py)
files to see the various commands ran to install CORE and translate them to
your use case, assuming it is possible.
* make sure you have python3.6+ with venv support
* make sure you have python3 invoke available to leverage `<repo>/tasks.py`
If you get install down entirely, feel free to contribute and help others.
```shell
cd <repo>
## Installed Scripts
# Usage: inv[oke] [--core-opts] install [--options] [other tasks here ...]
#
# Docstring:
# install core, poetry, scripts, service, and ospf mdr
#
# Options:
# -d, --dev install development mode
# -i STRING, --install-type=STRING
# -l, --local determines if core will install to local system, default is False
# -p STRING, --prefix=STRING prefix where scripts are installed, default is /usr/local
# -v, --verbose enable verbose
After the installation complete it will have installed the following scripts.
# install virtual environment
inv install -p <prefix>
| Name | Description |
|---|---|
| core-cleanup | tool to help removed lingering core created containers, bridges, directories |
| core-cli | tool to query, open xml files, and send commands using gRPC |
| core-daemon | runs the backed core server providing TLV and gRPC APIs |
| core-gui | runs the legacy tcl/tk based GUI |
| core-imn-to-xml | tool to help automate converting a .imn file to .xml format |
| core-manage | tool to add, remove, or check for services, models, and node types |
| core-pygui | runs the new python/tk based GUI |
| core-python | provides a convenience for running the core python virtual environment |
| core-route-monitor | tool to help monitor traffic across nodes and feed that to SDT |
| core-service-update | tool to update automate modifying a legacy service to match current naming |
| coresendmsg | tool to send TLV API commands from command line |
# indstall locally
inv install -p <prefix> -l
# this will print the commands that would be ran for a given installation
# type without actually running them, they may help in being used as
# the basis for translating to your OS
inv install --dry -v -p <prefix> -i <install type>
```
## Running User Scripts
If you create your own python scripts to run CORE directly or using the gRPC/TLV
APIs you will need to make sure you are running them within context of the
installed virtual environment. To help support this CORE provides the `core-python`
executable. This executable will allow you to enter CORE's python virtual
environment interpreter or to run a script within it.
> **NOTE:** the following assumes CORE has been installed successfully
For installations installed to a virtual environment:
```shell
core-python <script>
```
If CORE was installed locally, then you can run scripts using the default python3
interpreter.
For local installations:
```shell
python3 <script>
```
## Installing EMANE
> **NOTE:** installng emane for the virtual environment is known to work for 1.21+
> **NOTE:** automated install currently targets 1.25
@ -205,7 +251,6 @@ poetry run pip install <EMANE_REPO>/src/python
```
## Using Invoke Tasks
The invoke tool installed by way of pipx provides conveniences for running
CORE tasks to help ensure usage of the create python virtual environment.

118
tasks.py
View file

@ -8,7 +8,7 @@ from contextlib import contextmanager
from enum import Enum
from pathlib import Path
from tempfile import NamedTemporaryFile
from typing import Optional
from typing import Optional, List
from invoke import task, Context
@ -16,6 +16,14 @@ DAEMON_DIR: str = "daemon"
DEFAULT_PREFIX: str = "/usr/local"
EMANE_CHECKOUT: str = "v1.2.5"
OSPFMDR_CHECKOUT: str = "26fe5a4401a26760c553fcadfde5311199e89450"
REDHAT_LIKE = {
"redhat",
"fedora",
}
DEBIAN_LIKE = {
"ubuntu",
"debian",
}
class Progress:
@ -55,11 +63,28 @@ class Progress:
class OsName(Enum):
UBUNTU = "ubuntu"
CENTOS = "centos"
UNKNOWN = "unknown"
@classmethod
def get(cls, name: str) -> "OsName":
try:
return OsName(name)
except ValueError:
return OsName.UNKNOWN
class OsLike(Enum):
DEBIAN = "debian"
REDHAT = "rhel fedora"
REDHAT = "rhel"
@classmethod
def get(cls, values: List[str]) -> Optional["OsLike"]:
for value in values:
if value in DEBIAN_LIKE:
return OsLike.DEBIAN
elif value in REDHAT_LIKE:
return OsLike.REDHAT
return None
class OsInfo:
@ -68,6 +93,22 @@ class OsInfo:
self.like: OsLike = like
self.version: float = version
@classmethod
def get(cls, name: str, like: List[str], version: Optional[str]) -> "OsInfo":
os_name = OsName.get(name)
os_like = OsLike.get(like)
if not os_like:
like = " ".join(like)
print(f"unsupported os install type({like})")
sys.exit(1)
if version:
try:
version = float(version)
except ValueError:
print(f"os version is not a float: {version}")
sys.exit(1)
return OsInfo(os_name, os_like, version)
def get_python(c: Context, warn: bool = False) -> str:
with c.cd(DAEMON_DIR):
@ -85,28 +126,24 @@ def get_pytest(c: Context) -> str:
return os.path.join(venv, "bin", "pytest")
def get_os() -> OsInfo:
d = {}
with open("/etc/os-release", "r") as f:
for line in f.readlines():
line = line.strip()
if not line:
continue
key, value = line.split("=")
d[key] = value.strip("\"")
name_value = d["ID"]
like_value = d["ID_LIKE"]
version_value = d["VERSION_ID"]
try:
name = OsName(name_value)
like = OsLike(like_value)
version = float(version_value)
except ValueError:
print(
f"unsupported os({name_value}) like({like_value}) version({version_value}"
)
sys.exit(1)
return OsInfo(name, like, version)
def get_os(install_type: Optional[str]) -> OsInfo:
if install_type:
name_value = OsName.UNKNOWN.value
like_value = install_type
version_value = None
else:
d = {}
with open("/etc/os-release", "r") as f:
for line in f.readlines():
line = line.strip()
if not line:
continue
key, value = line.split("=")
d[key] = value.strip("\"")
name_value = d["ID"]
like_value = d["ID_LIKE"]
version_value = d["VERSION_ID"]
return OsInfo.get(name_value, like_value.split(), version_value)
def check_existing_core(c: Context, hide: bool) -> None:
@ -304,9 +341,13 @@ def install_scripts(c, local=False, verbose=False, prefix=DEFAULT_PREFIX):
"verbose": "enable verbose",
"local": "determines if core will install to local system, default is False",
"prefix": f"prefix where scripts are installed, default is {DEFAULT_PREFIX}",
"install-type": "used to force an install type, "
"can be one of the following (redhat, debian)",
},
)
def install(c, dev=False, verbose=False, local=False, prefix=DEFAULT_PREFIX):
def install(
c, dev=False, verbose=False, local=False, prefix=DEFAULT_PREFIX, install_type=None
):
"""
install core, poetry, scripts, service, and ospf mdr
"""
@ -315,9 +356,10 @@ def install(c, dev=False, verbose=False, local=False, prefix=DEFAULT_PREFIX):
c.run("sudo -v", hide=True)
p = Progress(verbose)
hide = not verbose
os_info = get_os()
with p.start("checking for old installations"):
check_existing_core(c, hide)
os_info = get_os(install_type)
if not c["run"]["dry"]:
with p.start("checking for old installations"):
check_existing_core(c, hide)
with p.start("installing system dependencies"):
install_system(c, os_info, hide)
with p.start("installing system grpcio-tools"):
@ -342,16 +384,18 @@ def install(c, dev=False, verbose=False, local=False, prefix=DEFAULT_PREFIX):
help={
"verbose": "enable verbose",
"local": "used determine if core is installed locally, default is False",
"install-type": "used to force an install type, "
"can be one of the following (redhat, debian)",
},
)
def install_emane(c, verbose=False, local=False):
def install_emane(c, verbose=False, local=False, install_type=None):
"""
install emane and the python bindings
"""
c.run("sudo -v", hide=True)
p = Progress(verbose)
hide = not verbose
os_info = get_os()
os_info = get_os(install_type)
with p.start("installing system dependencies"):
if os_info.like == OsLike.DEBIAN:
c.run(
@ -458,11 +502,19 @@ def uninstall(c, dev=False, verbose=False, local=False, prefix=DEFAULT_PREFIX):
"verbose": "enable verbose",
"local": "determines if core will install to local system, default is False",
"prefix": f"prefix where scripts are installed, default is {DEFAULT_PREFIX}",
"branch": "branch to install latest code from, default is current branch"
"branch": "branch to install latest code from, default is current branch",
"install-type": "used to force an install type, "
"can be one of the following (redhat, debian)",
},
)
def reinstall(
c, dev=False, verbose=False, local=False, prefix=DEFAULT_PREFIX, branch=None
c,
dev=False,
verbose=False,
local=False,
prefix=DEFAULT_PREFIX,
branch=None,
install_type=None
):
"""
run the uninstall task, get latest from specified branch, and run install task
@ -479,7 +531,7 @@ def reinstall(
c.run("git pull", hide=hide)
if not Path("tasks.py").exists():
raise FileNotFoundError(f"missing tasks.py on branch: {branch}")
install(c, dev, verbose, local, prefix)
install(c, dev, verbose, local, prefix, install_type)
@task