Paralleljobs takes a job specification from a database, executes the
corresponding command, collects and stores information from the job’s output,
and repeats until all jobs were processed. Multiple instances may be run in
parallel against the same database.
You may obtain a copy of the git
Say you have a fancy new method to test whether a given integer is a prime
number. You casted your algorithm into an implementation that behaves like
(The idea is that isprime (or a wrapper script) produces information that will
be collected by paralleljobs.)
We would like to launch thousands of such invokations in parallel on our 12
core machine. Some invokations may take very long, others may be quite fast.
And we would like to collect data from our invokations, not only the exit code,
but also runtime, memory used, and application specific data. Paralleljobs
allows us to do this in a simple way. We first create a file with a list of
We do not know whether our code may use a huge amount of memory or may not
terminate at all. Hence, we limit resources with ulimit. We then tell
paralleljobs to create a sqlite database with a jobs table and we let it import
jobs from ‘jobs.sh’. (Instead of importing the jobs from a file, we could also
fill the ‘jobs’ table in the database by other means.) Furthermore, we would
like paralleljobs to collect data (key-value pairs) from the jobs’ output and
we specify in SQL which keys it must anticipate.
And now we can launch, say, 12 parallel processes that do all the work.
After some time you may wonder how much jobs were already processed.
If you have for each job an estimate for the workload you can adjust the
per-job estimates in the database (see table ‘jobs’) to get a better estimate
of the workload finished.
When all jobs were processed you can extract data from the properties table and
produce a plot in gnuplot, or whatever.