Funtoo Telemetry Project

Latest Status

Long needed update to the Project Telemetry. There has been lots of activity in Phase 4 of this project recently which included deploying Phase 1 components onto internal Funtoo Infrastructure. This has progressed significantly and we now have Prometheus/Alertmanager/Grafana/Blackbox Exporter all deployed for internal Funtoo containers and infrastructure, running in and LXD container itself. The Funtoo Prometheus Exporter planning is ongoing, active development has not yet started. Phase 2 is not as active but will be moving along so we can start compiling Prometheus components for the arm64 architecture, specifically raspi4.

21 June 2022

Telemetry is a new project to upgrade all the core Prometheus and Grafana related packages in Funtoo so that the latest real time metrics collection and analysis tools are available to both desktop and server users.

What are the Benefits?

Modern time series and real time Telemetry and Observibility tools like Prometheus and Grafana are great to precisely monitor and alert on various computer application based infrastructure that run at production web scale web. They are also highly useful in the hobbyist realm instrumenting and measuring all types of different technologies such as smart home devices, energy production systems like solar arrays, any remote devices that can be scraped for useful data, and just abut anything that can run the Linux operating system.

Having the most recent up to date versions of Prometheus tools allows Funtoo users to leverage them for whatever monitoring and alerting project and task they desire. In addition to ensuring these packages have the latest security patches, having them updated a maintained ndregularly is very useful for folks running Funtoo Servers and would like to leverage them to monitor those systems.

Lastly, having these packages available and automatically generating in the Funtoo Portage tree will allow them to the be leveraged for any internal Funtoo Infrastructure.

What are we Building?

We are going to be building out complete support for the latest and greatest Prometheus and Grafana software in Funtoo Linux in addition to engineering new telemetry software for Funtoo Linux.

This project is being approached in multiple phases to better track overall progress and features. Some Phases may be ongoing in parallel. Below are the current Phases in sequential order:

Phase 1 (Completed)

Parent Issue: FL-9365 Refactor all Prometheus and Grafana related CatPkgs to kit-fixups autogen ebuild

Details: Convert all Prometheus and Grafana related Gentoo staging packages to funtoo-metatools autogenerated binary packages, that dynamically pull the latest release from the associated package's GitHub Releases. Also, during this Phase, the initial foundation of autogenerated templates will be worked through so if can be built upon in Phase 2.

Scope: Prometheus and Grafana related packages that are pulled from Gentoo Staging. Only Official core or High Quality community Prometheus and Grafana software will be targeted

Phase 2

Parent Issue: FL-9365 Refactor all Prometheus and Grafana related CatPkgs to kit-fixups autogen ebuild

Details: Create new funtoo-metaltools powered compile based autogenerated packages that build from the upstream Prometheus related package source code.

References to Gentoo staging packages will be removed during this phases to avoid any package name collisions.

Phase 3

Parent Issue: core-server-kit app-metrics/funtoo_exporter: Engineer new Funtoo Prometheus Exporter]

Details: Engineer a custom Golang Funtoo Prometheus exporter on code.funtoo.org that can expose all types of fun and useful Prometheus metrics about a Funtoo Linux system. Create associated Grafana dashboard JSON template that Funtoo Usesrs can use with this exporter in their local Grafana instances.

Phase 4

Parent Issue: Not created yet

Details: Deploy any new or needed Prometheus related infrastructure including Prometheus instances and Prometheus exporters to measure different metrics all throughout and across the Funtoo internal infrastructure. This phase will leverage all of the autogenerated and up-to-date packages created in Phase 1 through 3.

Phase 5

Parent Issue: Not created yet

Details: Create and launch a publicly viewable Grafana Site running atop Funtoo Infrastructure to visualize all Funtoo Infrastructure related Prometheus data. Make it available through a domain like observe.funtoo.org and integrate Grafana's authentication and authorization directly with auth.funtoo.org (possibly protected by VPN if public DNS and IP space is not available) . Instrument a new Discord channel called #observe-feed or #alerts-feedthat streams Alertmanager Alerts directly to Funtoo Discord Server and to the community regarding infrastructure health just like the #wiki-feed and #bugs-feed

How Can I Get Involved?

Join us on Funtoo Discord and also see below for status updates and instructions on how to join our collaborative effort.

Status Updates

2022-06-21: Long needed update to the Project Telemetry. There has been lots of activity in Phase 4 of this project recently which included deploying Phase 1 components onto internal Funtoo Infrastructure. This has progressed significantly and we now have Prometheus/Alertmanager/Grafana/Blackbox Exporter all deployed for internal Funtoo containers and infrastructure, running in and LXD container itself. The Funtoo Prometheus Exporter planning is ongoing, active development has not yet started. Phase 2 is not as active but will be moving along so we can start compiling Prometheus components for the arm64 architecture, specifically raspi4.

2022-03-16: Phase 2 is currently ongoing but partially blocked. Phase 3, building a Funtoo Linux Prometheus Exporter, has officially started its R&D phase. New Project Telemetry Activities have been published.

2022-03-10: Started mapping Phase 2 of the Funtoo Project Telemetry. Promu is required before starting and will be the first Golang source based autogen of Phase 2.

2022-03-08: Refactored of all of the autogens created during the Project Telemetry's Phase 1 to group autogens in preparation for starting Phase 2

2022-02-22: Phase 1 completed on the Funtoo Discord Live Stream. The final *-bin Prometheus exporter (mysqld_exporter-bin-0.13.0) autogen is now completed.

2022-02-18: * FL-9455 app-metrics/blackbox_exporter_bin autogen failing was mitigated during a Funtoo Discord Community Live Stream

Consul_exporter-bin 0.8.0 auto-generated package created during a Funtoo Discord Community Live Stream and are now available in Funtoo 1.4 and next
There is only one more official autogen to create for Phase 1 of the Telemetry Project: mysqld_exporter-bin

2022-02-15: Memcached_exporter-bin 0.9.0 and redis_exporter-bin v1.35.1 auto-generated packages created during a Funtoo Discord Community Live Stream and are now available in Funtoo 1.4 and next

2022-02-14: Alertmanager-bin v0.23.0 and blackbox_exporter-bin v0.19.0 auto-generated packages are now available in Funtoo 1.4 and next

2022-02-12: * Prometheus-bin autogen created and merged, making it available in Funtoo 1.4 and next (autogened to v2.33.1, v2.33.2, and most recently v2.33.3)

Alertmanager-bin autogen created and is in code review.
The latest Prometheus, Alertmanager, node_exporter and Grafana bin CatPkgs all have been tested locally via their OpenRC init scripts and are working on Funtoo 1.4
Here is a screenshot of the beginnings of a Funtoo node_exporter Grafana dashboard dashboard, all powered by the above software on a local Funtoo 1.4 system:

2022-02-09: * Grafana-bin v8.3.4 autogen has been created via Funtoo Discord Live Stream hacking event and merged, making it available in Funtoo 1.4 (autogened to v8.3.5 and v8.3.6 days after)

Prometheus node_exporter v1.3.1 autogen has been created, merged, and is available in Funtoo 1.4
Partial autogen.yml foundation for single binary Prometheus exporter GitHub released tarballs has been created. Subsequent autogens of Official Prometheus upstream and community exporters can leverage reuse this work directly.