On Tue, 28 Jul 2015 10:07:02 +0300
Ido Perlmuter <ido_at_ido50.net> wrote:
> Hello there.
>
> A few months ago I've transitioned some of my production servers from
> supervisord to perp. In order to make the transition smoother for
> other system admins on my servers I've decided to create a shell
> resembling supervisorctl, where you can check the status of services,
> stop/start/restart them, send them signals, etc. with simple commands
> (such as "restart nginx").
>
> I've also decided to write the shell such that it would support
> several supervision suites, starting with perp and s6. Of course, the
> shell merely calls the relevant utilities of the suite to actually do
> its work. For example, echoing "restart nginx" on the shell prompt
> simply calls "perpctl q nginx" internally.
>
> Anyway, while I've mostly implemented everything I need thus far,
> there is one point that is causing me grief. supervisorctl allowed me
> to view the stdout/stderr of running processes at any given time. It
> provided two ways to do this, one with the "fg" command (e.g. "fg
> nginx"), which would bring the process to the foreground (probably
> "virtually") and start displaying its stdout/stderr; the other with
> the "tail" command (e.g. "tail nginx stderr"), which allows viewing
> the latest messages in a specific channel (but not both).
>
> The thing is that supervisord basically enforces its own logging
> mechanism. It stores stdout and stderr to seperate files, and it can
> read them whenever it wants. With perp/s6/etc, however, the user is
> free to decide what logging mechanism to use.
>
> Currently, I've implemented the "fg" command in a pretty dumb way: I
> read the service's rc.log file for perp or log/run file for s6, and
> take the last argument to tinylog or s6-log to figure out where log
> files are stored. Of course, this is a very bad way to do this, since
> I'm assuming the services are using tinylog/s6-log, and I'm using a
> regular expression that could easily miss.
>
> I'm looking for some advice how reading the service's stdout/stderr
> streams could be done in a more fool proof, general way. The only way
> I know to tap into a process' output streams is via strace, but that
> means the user will have to install it, and run it as root, so that's
> not good.
>
> Any suggestions are welcome. If you're curious, the shell's
> repository is on github <https://github.com/ido50/Svsh>, it's
> currently written in Perl (I wanted get it up and running as fast as
> possible), I may reimplement it later with another language.
>
> Thanks,
> Ido Perlmuter.
Hi Ido,
That is a very cool idea to develop a shell-like interface!
As for logging display, you have gotten some good suggestions from
Laurent and Colin. With regards to perp, just a few more comments that
may be helpful (or not!).
perp actually ships with two stdin loggers:
* sissylog(8), for logging to a traditional syslog daemon
* tinylog(8), for logging to a set of rotating log files
There is nothing proprietary about logging with perp. Sites may use
either of the above, both, or none. Sites may prefer multilog (logger
shipping with daemontools), runitlog (logger shipping with runit), or
something else. For example, you could also use socat for, say, sending
the logging stream across the network to a dedicated logging server.
In any case, a site will -- or should -- have a standardized logging
policy. That is, all services will generally be setup to log the same
way, to the same base destination, under the same privileges.
To this end, on my own sites, all services are setup with an absolutely
identical rc.log as follows:
#!/bin/sh
# rc.log
# generic perpetrate logging runscript
# using tinylog_run wrapper configured by /etc/tinylog.conf
# wcm, 2009.12.28 - 2009.12.28
# ===
if test ${1} = 'start' ; then
exec tinylog_run ${2}
fi
exit 0
### EOF
All this does is to exec tinylog_run, which is a front-end for
tinylog(8), called with a single argument, which is the name of the
service.
Note: the tinylog_run wrapper script is provided in the perp source
distribution. It does the following:
* reads in sitewide configuration from /etc/tinylog.conf
* initializes a tinylog user for privilege drop
* builds a path to the service logging directory
* creates, chowns and chmods that directory as necessary
* drops privilege and execs tinylog
The point of all this is that it maintains a sane and consistent
logging policy for all services, very easily. Even if you don't care
for the particular details of this implementation, something similar is
highly recommended.
Of course, this type of system may cause problems for your shell
project if it tries to parse the logging directory directly out of the
rc.log script: you will notice it's not there.
But it will also make your shell easier to implement if you design it
to be used in conjunction with such a system. That way your shell can
also use /etc/tinylog.conf (or whatever) to determine directly that all
tinylog logs will be posted to, say, /var/log/<servicename>.
Couple of other comments:
* The fact that perpd will generally use fd4 within the procfs on linux
is not surprising. But it is not guaranteed, either. That is, nothing
within perpd explicitly requires fd4, it just normally works out that
way.
* The tail(1) command is not as portable as one may like. In
particular, OpenBSD uses the "-f" option (and no "-F" option) to get
what is provided by the "-F" option on gnu/linux and Free/NetBSD. Sigh.
All the best,
Wayne
Received on Tue Jul 28 2015 - 23:11:07 UTC