Wednesday, November 26, 2014

Converting to EL7: Solving The "Your Favorite Service Isn't Systemd-Enabled" Problem

Having finally gotten off my butt to knock out getting my RHCE (for EL6) before the 2014-12-19 drop-dead date, I'm finally ready to start focusing on migrating my personal systems to EL7-based distros.

My personal VPS is currently running CentOS 6.6. I use my VPS to host a couple of personal websites and email for family and a few friends. Yes, I realize that it would probably be easier to offload all of this to providers like Google. However, while Google is very good at SPAM-stomping, and provides me a very generous amount of space for archiving emails, one area that they do lack for is email aliases: whenever I have to register to a new web-site, I use a custom email address to do so. At my last pruning, I still had 300+, per-site, aliases. So, for me, number of available aliases ("unlimited" is best) and ease of creating them trumps all other considerations.

Since I don't have Google handling my mail for me, I have to run my own A/V and anti-spam engines. Being a good Internet Citizen, I also like to make use of Sender Policy Framework (via OpenSPF) and DomainKeys (currently via DKIMproxy).

I'm only just into the process of sorting out what I need to do to make the transition as quick and as painless (more for my family and friends than me) a process as possible. I hate outages. And, with a week off for the Thanskgiving holidays, I've got time to do things in a fairly orderly fashion.

At any rate, one of the things I discovered is that my current DomainKeys solution hasn't been updated to "just work" within the systemd framework used within EL7. This isn't terribly surprising, as it appears that the DKIMproxy SourceForge project may have gone dormant, in 2013 (so, I'll have to see if there's alternatives that have the appearance of still being a going concern - in the mean time...) Fortunately, the DKIMproxy source code does come with a `chkconfig` compatible SysV-init script. Even more fortunately, converting from SysV-init to a systemd-compatible service control is a bit more straight forward than when I was dealing with moving from Solaris 9's legacy init to Solaris 10's SMF.

If you've already got a `chkconfig` style init script, moving to systemd-managed is fairly trivial. Your `chkconfig` script can be copied, pretty much "as is" into "/usr/lib/systemd". My (current) preference is to create a "scripts" subdirectory and put it in there. Haven't read deeply enough into systemd to see if this is the "Best Practices" method, however. Also, where I work has no established conventions ...because they only started migrating to EL6 in fall of 2013 - so, I can't exactly crib anything EL7-related from how we do it at work.

Once you have your SysV-init style script placed where it's going to live (e.g., "/usr/lib/systemd/scripts"), you need to create associated service definition files. In my particular case, I had to create two as the DKIMproxy software actually has an inbound and an outbound funtion. Launched from normal SysV-init, it all gets handled as one piece. However, one of the nice things about systemd is it's not only a launcher framework, it's a service monitor framework, as well. To take full advantage, I wanted one monitor for the inbound service and one for the outbound service. The legacy init script that DKIMproxy ships with makes this easy enough as, in addition to the normal "[start|stop|restart|status]" arguments, it had per-direction subcommand (e.g., "start-in" and "stop-out"). The service-definition for my "dkim-in.service" looks like:
[Unit]
     Description=Manage the inbound DKIM service
     After=postfix.service


     [Service]
     Type=forking
     PIDFile=/usr/local/dkimproxy/var/run/dkimproxy_in.pid
     ExecStart=/usr/lib/systemd/scripts/dkim start-in
     ExecStop=/usr/lib/systemd/scripts/dkim stop-in


     [Install]
     WantedBy=multi-user.target

To break down the above:

  • The "Unit" stanza tells systemd a bit about your new service:
    • The "Description" line is just ASCII text that allows you to provide a short, meaningful of what the service does. You can see your service's description field by typing `systemctl -p Description show <SERVICENAME>`
    • The "After" parameter is a space-separated list of other services that you want to have successfully started before systemd attempts to start your new service. In my case, since DKIMproxy is an extension to Postfix, it doesn't make sense to try to have DKIMproxy running until/unless Postfix is running.
  • The "Service" stanza is where you really define how your service should be managed. This is where you tell systemd how to start, stop, or reload your service and what PID it should look for so it knows that the service is still notionally running. The following parameters are the minimum ones you'd need to get your service working. Other parameters are available to provide additional functionality:
    • The "Type" parameter tells systemd what type of service it's managing. Valid types are: simpleforking, oneshot, dbus, notify or idle. The systemd.service man page more-fully defines what each option is best used for. However, for a traditional daemonized service, you're most likely to want "forking".
    • The "PIDFile" parameter tells systemd where to find a file containing the parent PID for your service. It will then use this to do a basic check to monitor whether your service is still running (note that this only checks for presence, not actual functionality).
    • The "ExecStart" parameter tells systemd how to start your service. In the case of a SysV-init script, you point it to the fully-qualified path you installed your script to and then any arguments necessary to make that script act as a service-starter. If you don't have a single, chkconfig-style script that handles both stop and start functions, you'd simply give the path to whatever starts your service. Notice that there are no quotations surrounding the parameter's value-section. If you put quotes - in the mistaken belief that the starter-command and it's argument need to be grouped, you'll get a path error when you go to start your service the first time.
    • The "ExecStop" parameter tells systemd how to stop your service. As with the "ExecStart" parameter, if you're leveraging a fully-featured SysV-init script, you point it to the fully-qualified path you installed your script to and then any arguments necessary to make that script act as a service-stopper. Also, the same rules about white-space and quotation-marks apply to the "ExecStop" parameter as do the "ExecStart" parameter.
  • The "Install" stanza is where you tell systemd the main part of the service dependency-tree to put your service. You have two main dependency-specifiers to choose: "WantedBy" and "RequiredBy". The former is a soft-dependency while the latter is a hard-dependency. If you use the "RequiredBy" parameter, then the service unit-group (e.g., "mult-user.target") enumerated with the "RequiredBy" parameter will only be considered to have successfully onlined if the defined service has successfully launched and stayed running.  If you use the "WantedBy" parameter, then the service unit-group (e.g., "mult-user.target") enumerated with the "WantedBy" parameter will still be considered to have successfully onlined whether the defined service has successfully launched or stayed running. It's most likely you'll want to use "WantedBy" rather than "RequiredBy" as you typically won't want systemd to back off the entire unit-group just because your service failed to start or stay running (e.g., you don't want to stop all of the multi-user mode related processes just because one network service has failed.)

1 comment:

  1. Tom's post is spot-on. If you drop your traditional init script somewhere into /usr/lib/systemd/, create a (service).service that essentially passes the start|stop|restart options to it, and symlink it to an associated .service file in /etc/systemd/system/(runlevel).target.wants, you can easily get up and running w/out having to re-write your startup routine in true systemd vernacular. This approach works particularly well if you have an older style init script with numerous preparatory routines. One thing to note: you probably don't want to leave a copy of your newly-converted init script in /etc/rc.d/init.d. When I initially ran 'systemctl enable (service).service' without an explicit path to my new .service file, it appeared to find the old init script and load it as would a 'chkconfig --add ' routine. Subsequently, running 'systemctl show --all (service).service' reveals the SourcePath variable and shows that the script path is actually pointing back to /etc/rc.d/init.d rather than the .service file you created somewhere in the /usr/lib/systemd/ directory. I'm guessing that 'systemctl enable (service).service' defaults to /etc/rc.d/init.d as the default path unless overridden with an explicit path.

    I also found the RHEL Summit slide deck (http://rhsummit.files.wordpress.com/2014/04/summit_demystifying_systemd1.pdf) and tried to decompose a nagios init script for the fun of it. I got stumped here and there with letting systemd manage the service.pid file while defining the User and Group variables, due to file ownership issues. So I left those out. My first working systemd script looked something like this.

    [Unit]
    Description=Nagios server
    DefaultDependencies=no
    # My nagios instance has a dependency on syslog through setting the use_syslog variable in the nagios.cfg file, so
    # I wanted to define this suggested startup order
    After=syslog.target

    [Service]
    Type=forking
    PIDFile=/var/run/nagios.pid
    ExecStartPre=-/bin/touch /var/run/nagios.pid
    ExecStartPre=/bin/chown nagios:nagios /var/run/nagios.pid
    ExecStart=/usr/sbin/nagios -d /etc/nagios/nagios.cfg
    ExecStop=/bin/kill ${MAINPID}

    [Install]
    WantedBy=multi-user.target

    Later, I found this blog (http://blog.hqcodeshop.fi/archives/93-Handling-varrun-with-systemd.html) that offered some additional guidance. Using the PermissionsStartOnly allowed me to explicitly define the User and Group variables (normally, this shouldn't be necessary if your config file defines who owns the daemon, so I'm doing this merely to emulate what would happen if that weren't the case) while creating the .pid file. If I didn't define it this way, attempting to start the service would end with a permissions error due to nagios not being able to write to the /var/run directory. Here's what a working version ended up looking like:

    [Unit]
    Description=Nagios server
    DefaultDependencies=no
    After=syslog.target

    [Service]
    Type=forking
    User=nagios
    Group=nagios
    PIDFile=/var/run/nagios.pid
    PermissionsStartOnly=true
    ExecStartPre=-/bin/touch /var/run/nagios.pid
    ExecStartPre=/bin/chown nagios:nagios /var/run/nagios.pid
    ExecStart=/usr/sbin/nagios -d /etc/nagios/nagios.cfg
    ExecStop=/bin/kill ${MAINPID}

    [Install]
    WantedBy=multi-user.target

    So as you can see, systemd is pretty flexible and can presumably accommodate older style init scripts that carry out pre-and-post routines.

    ReplyDelete