phoenix-framework ubuntu upstart reliable service

How to reliably start a service with UpStart

Zohaib Rauf
Zohaib Rauf
3 min read

I recently had to start a service in Ubuntu server but I wanted it to be reliable. By reliable I mean it should be restarted in case of any failure and on machine restart as well. I stumbled upon UpStart which comes by default with Ubuntu and does exaclty what I wanted. In this post we are going to see how to setup the UpStart configuration and we will be starting the Phoenix server.

Creating the configuration file

The configuration for your service should exist in /etc/init/, this is how ubuntu knows which services are there. The extension of the configuration file is .conf. We will create a phoenix.conf and symlink it to /etc/init by sudo ln -s /path/to/phoenix.conf /etc/init, this allows us to edit the configuration without sudo.

The service can be started on various events

  1. start on startup: Run the service on machine startup, this happens even before the filesystem and network is completely ready.
  2. start on runlevel [Levels]: Start the service on various linux run levels.
  3. start on stopped Service_Name: Run the serivce when the mentioned service has stopped.
  4. start on started Service_Name: Run the service when the mentioned service has started.

Using the start on stopped and start on started you can chain various services together to create a flow of services so that they start one after the other e.g. setup database service maybe started before your web server starts etc.

Starting your service

The configuration file should contain the following block to start your service.

script
  #Commands to execute your service
end script

# If its just one command then you can do the following instead
exec #command

States

You can write commands in the configuration file to execute at various stages of a service. Following is the state transition. For complete diagram and details see here.

The states are self explanatory but I got confused between few.
Spawned vs Post-Started vs Running: The spawned is the state before your script is executed to start the service, post-started is the state before the event signalling that the job has started is emitted (due to some initialization phase) and running is when your job is considered to be started.

To do something at any of these states add the following

# Add the state name first e.g. for pre-started
pre-started script
	# Commands to execute
end script

# If its just one command then you can add the following instead
pre-started exec # Command

Respawning

Now unto the part which makes your service restart again. It's the respawn, adding this to your configuration file will respawn your service if it stops hence making it much more reliable. The other variation is respawn limit COUNT INTERVAL so if you don't want to respawn your service if it restarted by COUNT in INTERVAL seconds then you should use this. E.g. respawn limit 10 5 means that if your service restarted 10 times in 5 seconds then stop respawning.

Managing your service

Following are some of the commands to manage your service.

  • init-checkconf /path/to/your.conf to check if your configuration is valid or not.
  • initctl start <service> to start the service
  • initctl stop <service> to stop the service
  • initctl restart <service> to restart the service
  • initctl status <service> to see the status your service, whether its stopped, running etc.
  • initctl reload-configuration is used after you created a new configuration to reload the configurations
  • initctl list to see the list of all registered services
  • initctl list | grep <service> to see if your service is registered or not

You can find more commands here.

Logs

All your service logs are stored at /var/log/upstart/service_name.log. If you see any issues in service start etc. then just go over the logs to figure out what could be the issue. This has the stdout from your service.

Example

I'm going to create a phoneix based sample web service and build its release and then run that as a service.

First I create the sample phoneix application and create and elixir release.
I create the .conf configuration and make a link sudo ln -s /path/to/phoenix.conf /etc/init/.

The configuration for it looks like following.

start on runlevel [2345]

script
  # Change the working directory to where the realase is
  chdir /path/to/release
  
  # Start the phoenix server
  exec ./hello_phoenix foreground
end script

# Respawn and stop respawning if it got respawned 10 time in 10 seconds
respawn limit 10 10

After that I run init-checkconf /etc/init/phoenix.conf to see if the configruation is a valid configuration, syntactically. Now we need to reload the configuraiton sudo initctl reload-configuration.
Let's check if the service got registered by running sudo initctl list | grep phoenix, if it shows the services then it is registered.

Lastly let's start the service sudo initctl start phoenix.

That's all what is needed to reliably start anything. You can try restarting your server and your service will run once the machine boots up.

You might also like