Recently I had to configure some raspberry pies to gather information on the connectivity to external services from within different locations. I needed ssh access to the machines and also they had to be reachable for Prometheus to scrape. Once I configured the machines I had to ship them onsite. Someone at the location would then plug the pies to the network and the data should start flowing into Prometheus. Then we I had Grafana dashboards to visualize the data.
On the networking side, the challenge was to make sure the machines were reachable from the outside. Also Prometheus had to be able to access the exporters running on the pies. Those machines will be behind a firewall. No public IPs. No port forwarding setup for them.
Thankfully we have Tailscale. I configured the machines to run tailscaled at boot up. I run the daemon passing a preconfigured key, that way authentication was not necessary. I also provided a tag to be able to manage the access of all the machines via a acls collectively. With this in place, I know the pies will connect to my tailnet and I will have access to them after each reboot. Also, I instructed tailscaled to run its ssh server so I can disable all the ports in the physical interface. Fantastic.
A stack of raspberry pies.
The machines had to run a couple of processes. Both of them prometheus exporters. The first one exposes metrics on connections to different web servers on the Internet as well as icmp times. The other one is a custom exporter that exposes metrics that capture the running time of a list of SQL queries. I had to make sure those processes run constantly, even after reboots. The traditional approach here is to use systemd. Systemd is a bit tedious and, what happens if you move to another non-Unix platform to run this (granted, that is very unlikely). Is there any other way?
In this post I read an interesting way to use tmux to provide "most" of the functionality that you have with systemd. Basically you map the processes to tmux sessions. Here is the script I use to configure the pies:
#!/bin/bash
#
set -e
if [ ".$1" == "." ];then
echo "need pi dental number (01 to 08)"
exit
fi
h="pidental-$1"
if [ ! -f ./ts-key ];then
echo "./ts-key not found"
exit
fi
TS_KEY=$(cat ./ts-key)
echo "copying public key ..."
ssh-copy-id drio@${h}.local
echo "updating software ..."
ssh drio@${h}.local "sudo apt-get update -y && sudo apt-get upgrade -y && sudo apt-get install vim tmux git -y"
echo "installing TS ..."
ssh drio@${h}.local "curl -fsSL https://tailscale.com/install.sh | sh && \
sudo tailscale up --advertise-tags=tag:dental-pi \
--auth-key=$TS_KEY \
--accept-routes --accept-dns --ssh"
echo "rsyncing software ..."
HOST=${h} make rsync
echo "starting services ..."
ssh drio@${h} "./services/start"
echo "installing cronjob ..."
ssh drio@${h} "echo '@reboot cd ~/services/ && ./start' | crontab"
echo "disabling ssh ..."
ssh drio@${h} "sudo systemctl disable ssh && sudo systemctl stop ssh"
echo "sleeping for 2secs ..."
sleep 2
echo "query endpoint to get version"
curl http://${h}:9200/version
The relevant bits are the "starting services" and "installing cronjob" sections. Each "service" I want to run has its own start/stop script. The start script in one of those looks like this:
#!/bin/bash
while :; do
./pidental.linux.arm64 | tee pidental.log
sleep 10
done
Nothing crazy. Just starting a binary and using tee to split the stdout of the process to a file and to the stdin of the bash running in that session.
Starting a service in the context of tmux means (for the service queries):
s=queries
tmux attach-session -t $s || tmux new-session -s $s -d
tmux send-keys -t $s 'cd; cd services/queries; ./start' C-m
Tell tmux to create a new session for the service (if it doesn't exist) and then run the necessary commands on it to start the service.
Finally, I use a functionality in crontab that I didn't know. By running:
"echo '@reboot cd ~/services/ && ./start' | crontab"
I am telling crontab to run that command every time the system reboots. Fantastic.
I believe using tmux in this way is a totally valid way to provide systemd like functionality in a different, arguably more friendly way. Would you agree?