Linux: Using flock to ensure only one instance of the script is running

It's very easy to create race conditions by accidentally running multiple instances of the script at the same time.

You can use this method to prevent that from happening simply by using the flock command (part of the standard util-linux package).

tl;dr

Example script:

#!/bin/bash

exec 7>/tmp/mylock
flock 7 || exit 1

echo "[$$] Starting..."
sleep 15
echo "[$$] Done..."

Example output (with three instances of the script running at the same time):

[1179045] Starting...
[1179045] Done...
[1179046] Starting...
[1179046] Done...
[1179047] Starting...
[1179047] Done...

How does it work?

First, let's create a simple script that will simulate some processing:

#/bin/bash

echo "[$$] Starting..."
sleep 15
echo "[$$] Done..."

Save that into a file named test and make it executable:

chmod +x test

Now, let's try to run multiple instances of the script:

./test & ./test &

(we're using ampersand at the end of each command to put it into the background)

[1183086] Starting...
[1183087] Starting...
[1183086] Done...
[1183087] Done...

As we can see from the output, there is an overlap between the processing phase of those two scripts.

So, what can we do to prevent this from happening?

Using flock outside of the script

The first method is to just use flock as an outside wrapper for our script:

flock /tmp/test_lock -c ./test &

We're using the flock command to obtain the exclusive lock on the /tmp/test_lock file (we could specify a file, directory, or a descriptor - more on that later), and we're specifying the command to run once the lock has been acquired with -c.

If we try to run multiple instances of this whole command:

flock /tmp/test_lock -c ./test & flock /tmp/test_lock -c ./test &

The result now should be as expected:

[1183024] Starting...
[1183024] Done...
[1183040] Starting...
[1183040] Done...

But if you try to run the test script in another terminal at the same, there'll be an overlap again.

This can easily happen in practice (for example, the command gets manually executed in the shell, while another instance is running from the crontab).

Using flock inside the script

To prevent that from happening, we need to put flock inside our script (and somehow tie it to the duration of our script).

A simple way to that is by using file descriptors:

exec 7>/tmp/mylock

We're using exec to assign the file descriptor (for reading that file) to our process. You can think of a file descriptor as a handle to access that file. 7 is an arbitrary number I decided to use here, but we could use any non-negative number (as long as it's not in use - standard file descriptors are 0 - STDIN, 1 - STDOUT, and 2 - STDERR).

Unless we explicitly close it, this descriptor will typically stay attached to our process until the process exits.

Now we just need to tell flock to use that file descriptor:

flock 7 || exit 1

Without any additional options, flock will typically block until the lock is acquired (exit is just a safeguard here if something goes wrong).

Full script:

#!/bin/bash

exec 7>/tmp/mylock
flock 7 || exit 1

echo "[$$] Starting..."
sleep 15
echo "[$$] Done..."

We can specify the timeout with -w option:

flock -w 500 7 || exit 1

This tells flock to wait 500 seconds before giving up.

If you want to exit immediately instead of waiting for the lock, you can use -n option.

flock -n 7 || exit 1

Alternatives

Alternatives include:

1. lockfile (provided by procmail)
2. manually creating and removing a pid file (which usually involves a race condition in the if statement)
3. directory variant of the previous alternative (which is using atomic mkdir operation)

I'll update this page with more examples.

Ask me anything / Suggestions