jobman package

Submodules

jobman.cli module

class jobman.cli.Cli[source]

Bases: object

exception UnrecognizedCommandError(msg=None, command=None, **kwargs)[source]

Bases: Exception

run(args=None)[source]
jobman.cli.main(args=None)[source]

jobman.constants module

class jobman.constants.JOB_STATUSES[source]

Bases: object

CLAIMED = 'CLAIMED'
COMPLETED = 'COMPLETED'
EXECUTED = 'EXECUTED'
FAILED = 'FAILED'
PENDING = 'PENDING'
RUNNING = 'RUNNING'
UNKNOWN = 'UNKNOWN'

jobman.jobman module

class jobman.jobman.CfgParams[source]

Bases: object

JobMan Config Parameters

auto_initialize = True

If True then create jobman db and initialize sources/engines.

dao = Ellipsis

An instance of a JobManDAO. Usually only used for testing.

db_uri = Ellipsis

A database uri.

Currently can only use Sqlite uris. For example, ‘sqlite://’ for an in-memory db, or ‘sqlite:////absolute/path/to/db.sqlite’ for a file-based db.

debug = Ellipsis

set to True to operate in debug mode. Logs more verbose statements.

classmethod get_cfg_infos()[source]
label = Ellipsis

A label to identify this jobman instance. Useful for logging.

logging_cfg = Ellipsis

logging config, per jobman.utils.logging_utils.generate_logger()

source_specs = Ellipsis

A dict of keyed source specs.

A source spec is a dict with this shape:

{
    # a dot_spec path to a source class
    'source_class': 'jobman.sources.dir_source:DirSource'
    # a dict of params to pass as kwargs when instantiating the
    # source.
    'source_params': {
        'root_path': self.root_path
    }
}
worker_specs = Ellipsis

A dict of keyed worker specs.

A worker spec is a dict with this shape:

{
    # a dot_spec path to a worker class
    'worker_class': 'jobman.workers.base_worker:BaseWorker'
    # a dict of params to pass as kwargs when instantiating the
    # worker.
    'worker_params': {
         # a spec for the worker's engine.
         'engine_spec': {
             'engine_class': (
                 'jobman.engines.local_engine:LocalEngine'
             ),
             'engine_params': {
                 'scratch_dir': self.scratch_dir,
                 # This is usually where engine configs go.
                 'cfg': {
                     'MY_EXE': '/my/cluster/path/to/my_exe',
                     # other cluster-specific params...
                 }
             }
         },
         # other worker params...
    }
}
class jobman.jobman.JobMan(label=None, cfg=None, source_specs=None, job_spec_defaults=None, worker_specs=None, dao=None, db_uri=None, auto_initialize=True, logging_cfg=None, debug=None)[source]

Bases: object

Central JobMan object.

Parameters:
class JOB_STATUSES

Bases: object

CLAIMED = 'CLAIMED'
COMPLETED = 'COMPLETED'
EXECUTED = 'EXECUTED'
FAILED = 'FAILED'
PENDING = 'PENDING'
RUNNING = 'RUNNING'
UNKNOWN = 'UNKNOWN'
exception SubmissionError[source]

Bases: Exception

classmethod from_cfg(cfg=None, overrides=None)[source]

Create a JobMan instance from a cfg.

Parameters:
  • cfg – can be an object or dict. Keys or attrs per CfgParams.
  • overrides – dict of cfg overrides
Returns:

JobMan instance
initialize()[source]

Initialize dao, sources, and workers.

job_spec_defaults = {'entrypoint': './entrypoint.sh'}

Default job_spec values.

submit_job_dir(job_dir=None, source_key=None, source_meta=None, job_spec_defaults=None, job_spec_overrides=None)[source]

Submit a job dir.

Creates a job_spec for dir. Attempts to read job_spec file from dir if provided.

Parameters:
Returns:

return per submit_job_spec().
submit_job_spec(job_spec=None, source_key=None, source_meta=None)[source]

Submit a job spec.

Parameters:
Returns:

return job record per

jobman.dao.jobman_sqlite_dao.JobmanSqliteDAO.JobSchema
tick()[source]

Tick JobMan components.