Funtoo Telemetry Phase4 Activity

   Activity Info

Parent Project	Funtoo:Telemetry
Start Date	17 March 2022
End Date	None
Activity Kind	Development
Status	Not yet started
Previous	Phase3

Overview

Deploy any new Prometheus related infrastructure including Prometheus instances and Prometheus exporters to measure different metrics all throughout and across the Funtoo internal infrastructure.

Objectives

Here are the general objectives of Phase 4 of the Telemetry Project:

• Test and leverage all autogen work done in Phase 1-3 on Funtoo Infrastructure
• Deploy an internal Funtoo Prometheus instance
• Deploy an internal Funtoo Prometheus instance
• Deploy an internal Funtoo Grafana instance
• Deploy at minimum Prometheus node_exporter and the funtoo_exporter from Phase 3 for each Funtoo Infrastructure system
• Deploy Blackbox Exporter for up/down probing of the Funtoo Infrastructure
• Leverage Mottainai CI/CD service to help automate deployments of all new Telemetry software
• Any Telemetry software deployed as part of this Phase should be as minimal as possible and require almost no reconfiguration of any pre-existing Funtoo Infrastructure
• Ensure there this no impact on any Funtoo production services and systems
• Once Prometheus, exporters, and Grafana are running, and metrics are scraping into the Prometheus TSDB, create some useful high level Grafana dashboards to visualize all relevant metrics

Research and Development

To accomplish the objectives this Project the some research detailed below will be required

Infrastructure Inventory

One of the first major tasks of this Phase is a discovery and documentation task. To better understand what type of metrics we can and should collect, we need to inventory all Funtoo Infrastructure that could leverage Prometheus metrics like physical systems, virtual machines, and containers. The inventory of the Funtoo Infrastructure should be stored on code.funtoo.org somewhere, fully codified and serialized into some common data format like JSON, YAML, TOML, etc.