How to implement simple job queue on Linux/Unix

Dear lazyweb:

I need a simple job queue. I would like to have a system _similar_ to the ‘at’ command, with the difference that it does not execute the job at a specific time, but queues jobs into a queue, and executes one after the other. Even though ‘man at’ says it uses queues, they are not behaving like queues. It should be almost a nobrainer to implement, but I feel there should be a Unix command to do just that. Any ideas?

Advertisements

10 Responses to How to implement simple job queue on Linux/Unix

  1. The ‘at’ page lists both its alternative interface ‘batch’ and another tool ‘nqs’.

    But in any case queueing systems have been a staple of High Performance Computing since the dawn of … well … whatever, so maybe you’ll be able to Google something that way?

    Rgds

    Damon

  2. roman says:

    @Damon: Yeah, nqs does not seem to be free software, and batch basically the same as at (i.e. not useful). Google did not return anything immediately useful (commercial, obsolete, etc, stuff).

  3. mike says:

    How about “batch now”? It queues up jobs just fine. If you don’t like the batch queue’s behavior (only runs when load is low), you could do “at -q c now.”

  4. Maybe you could knock one up with a named pipe?

    Have something listening on the pipe taking one line at a time and executing it, and to queue just stuff the item in the other end?

    The thinkg reading off the named pipe will have to read ahead I suspect else the named pipe may block unless it is empty.

    Could probably do it with a /bin/sh script at a push.

    Rgds

    Damon

  5. roman says:

    mike, this seems to do the job perfectly, thanks. Must have been blind…

  6. alk says:

    batch and at both can run jobs in parallel.

    When I had similar requirement (re-deploying web app after each commit) I had to use filesystem-based locks.

    But right now named pipe approach looks much better to me.

    #mkfifo namedq

    # (while true; do sh &1 | mail -s ‘job output’ -E root; done) &

    # echo ‘echo “test” | wall’ >namedq

  7. Ken Bloom says:

    Would the “batch” command (part of at) do the trick?

    batch executes commands when system load levels permit; in other
    words, when the load average drops below 1.5, or the value
    specified in the invocation of atd.

  8. Bastiaan Bergman says:

    My linux distro doesn’t seme to have “at” or “batch”, is it part of a package?
    Thanks!

    • Roman Kennke says:

      Bastiaan: I guess that really depends on your distro. On Fedora, Debian and Ubuntu, it is in the ‘at’ package.

      • Bastiaan Bergman says:

        Yes, I have Angstrom. Any smart idea for how to find the package? opkg list | grep “at” is giving way to many lines,.. Can I install it from source if not available? Where can I find the source? Google “at” isn’t working very well either 😉

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: