.. github display GitHub is NOT the preferred viewer for this file. Please visit https://flux-framework.rtfd.io/projects/flux-rfc/en/latest/spec_14.html 14/Canonical Job Specification ############################## A domain specific language based on YAML is defined to express the resource requirements and other attributes of one or more programs submitted to a Flux instance for execution. This RFC describes the canonical jobspec form, which represents a request to run exactly one program. .. list-table:: :widths: 25 75 * - **Name** - github.com/flux-framework/rfc/spec_14.rst * - **Editor** - Tom Scogland * - **State** - raw Language ******** .. include:: common/language.rst Related Standards ***************** - :doc:`spec_4` - :doc:`spec_20` - :doc:`spec_26` - :doc:`spec_31` Goals ***** - Express the resource requirements of a program to the scheduler. - Allow graph-oriented resource requirements to be expressed. - Express program attributes such as arguments, run time, and task layout, to be considered by the program execution service (RFC 12) - Express dependencies relative to other programs executing within the same Flux instance. - Emphasize expressivity over simplicity, as this canonical form may be generated from other user-friendly forms or interfaces. - Facilitate reproducible runs. - Promote sharing and reuse of jobspec. Overview ******** This RFC describes the canonical form of "jobspec", a domain specific language based on YAML [#f1]_. The canonical jobspec SHALL consist of a single YAML document representing a reusable request to run exactly one program. Hereafter, "jobspec" refers to the canonical form, and "non-canonical jobspec" refers to the non-canonical form. Non-canonical jobspec SHALL be decomposed into jobspec before it is enqueued for the scheduler and program execution service. User facing tools MAY generate jobspec from non-canonical jobspec, or other sources. Such tools MAY: - generate a batch of dependent jobspecs representing a scientific workflow - generate a stream of jobspecs representing a steered parameter study - convert simulation parameters into jobspec containing computed resource requirements, etc. - convert command line arguments to jobspec, e.g. "flux mpirun" Jobspec and Program Life Cycle ============================== The jobspec SHALL be submitted to a job submission service. Malformed jobspec SHALL be immediately rejected by the job submission service. A stack of plugins SHALL test jobspec against site or user defined criteria, and on failure, MAY reject the jobspec, or MAY warn the user and continue on. The job submission service SHALL enqueue the jobspec for consideration by the scheduler. The scheduler SHALL consider each enqueued jobspec in the context of its dependencies and the pool of available resources. When the scheduler chooses to execute a job, it allocates resources, associates them with the jobspec, and notifies the program execution service to start the program(s). The program execution service, described in RFC 12, launches the program(s). Task slots, containment, and task layout SHALL be created within the allocated resources as described by the jobspec, or if that is not possible, the job SHALL enter a failed state and resources SHALL be returned to the scheduler. Once a job is retired, the jobspec SHALL be retained as part of its provenance record. Resource Matching ================= Resources are represented as hierarchies or graphs, as described in RFC 4. FIXME: describe how Flux hierarchical resource representation affects jobspec design. Terminology =========== FIXME: Fill in Jobspec Language Definition *************************** A canonical jobspec YAML document SHALL consist of a dictionary defining the resources, tasks and other attributes of a single program. The dictionary MUST contain the keys ``resources``, ``tasks``, ``attributes``, and ``version``. Each of the listed jobspec keys SHALL meet the form and requirements listed in detail in the sections below. For reference, a ruleset for compliant canonical jobspec is provided in the **Schema** section below. Resources ========= The value of the ``resources`` key SHALL be a strict list which MUST define at least one resource. Each list element SHALL represent a **resource vertex** or resource descriptor object as a dictionary (described below). The list of resources defined under the ``resources`` key SHALL represent a composite resource request for the program defined in the jobspec. A resource vertex SHALL contain the following keys: **type** The ``type`` key SHALL be a string indicating the type of resource to be matched. Some type names MAY be reserved for use in the jobspec language itself. The currently reserved type is ``slot``, used to define **task slots**. Reserved types are described in the **Reserved Resource Types** section below. **count** The ``count`` key SHALL indicate the desired number or range of resources matching the current vertex and SHALL have one of three possible values: 1. A single positive integer representing a fixed count. 2. A string containing either an :doc:`RFC 22 ` idset representing all acceptable counts, or an :doc:`RFC 45 ` range as a compact alternative to the dictionary form given next in option 3. 3. A dictionary which SHALL contain the following key: **min** The ``min`` key SHALL be a positive integer indicating the minimum required count or amount of this resource. Additionally, it MAY contain the following keys: **max** The ``max`` key SHALL be an integer greater than or equal to ``min`` indicating the maximum required count or amount of this resource. If unset, the default ``max`` SHALL be *infinite*, therefore a ``count`` which specifies only the ``min`` key SHALL be considered a request for *at least* that number of a resource, and the scheduler SHALL generate the *R* that contains the maximum number of the resource that is available and subject to the ``operator`` and ``operand``. By contrast, if a fixed count is given to the ``count`` key, the scheduler SHALL match any resource that contains *at least* ``count`` of the resource, but its *R* SHALL contain exactly ``count`` of the resource (potentially leaving excess resources unutilized). **operator** The ``operator`` key SHALL be a single character string representing an operator applied between ``min`` and ``max`` which returns the next acceptable value. Currently defined operators are: addition ``+``, multiplication ``*``, or exponentiation ``^``. If ``^`` is specified, then ``min`` MUST be greater than or equal to two. If unset, the default ``operator`` SHALL be ``+``. **operand** The ``operand`` key SHALL be a positive integer used in conjunction with the ``operator``. If the ``operator`` is either ``*`` or ``^``, then the ``operand`` MUST be greater than or equal to two. If unset, the default ``operand`` SHALL be 1. The ``operator`` and ``operand`` keys MUST both be present if either is specified. A resource vertex MAY additionally contain one or more of the following keys: **unit** The ``unit`` key, if supplied, SHALL have a string value indicating the chosen units applied to the ``count`` value or values. **exclusive** The ``exclusive`` key SHALL be a boolean indicating, when true, that the current resource is requested to be allocated exclusively to the current program. If unset, the default value for ``exclusive`` SHALL be ``false`` for vertices that are not within a task slot. The default value for ``exclusive`` SHALL be ``true`` for task slots (``type: slot``) and their associated resources. **with** The ``with`` key SHALL indicate one or more edges of type ``out`` from this resource vertex to other resources. Therefore, the value of the ``with`` key SHALL be a strict list of dictionaries conforming to the resource vertex specification and MUST contain at least one such vertex. **label** The ``label`` key SHALL be a string that may be used to reference this resource vertex from other locations within the same jobspec. ``label`` SHALL be local to the namespace of the current jobspec, and each ``label`` in the current jobspec must be unique. ``label`` SHALL be mandatory in resource vertices of type ``slot``. **id** The value of the ``id`` key SHALL be a string indicating a set of matching resource identifiers. Reserved Resource Types ----------------------- **slot** A resource type of ``type: slot`` SHALL indicate a grouping of resources into a named **task slot**. A ``slot`` SHALL be a valid resource spec including a ``label`` key, the value of which may be used to reference the named task slot during tasks definition. The ``label`` provided SHALL be local to the namespace of the current jobspec. A task slot SHALL have at least one edge specified using ``with:``, and the resources associated with a slot SHALL be exclusively allocated to the program described in the jobspec, unless otherwise specified in the ``exclusive`` field of the associated resource. Tasks ===== The ``tasks`` key SHALL be a strict list which MUST define at least one task. Each list element SHALL be a dictionary representing a task or tasks to run as part of the program. A task descriptor SHALL contain the following keys: **command** The ``command`` key SHALL be a list representing an executable and its arguments. **slot** The ``slot`` key SHALL be a string matching the ``label`` of a resource vertex of type ``slot``. It is used to indicate the **task slot** on which this task or tasks shall be contained and executed. The number of tasks executed per task slot SHALL be a function of the number of resource slots and total number of tasks requested to execute. **count** The ``count`` key SHALL be a dictionary supporting at least the keys ``per_slot``, ``per_resource``, and ``total``, with other keys reserved for future or site-specific extensions, and SHALL contain exactly one supported key. **per_slot** The ``per_slot`` key SHALL be a positive integer indicating the number of tasks to execute per task slot allocated to the program. **per_resource** The ``per_resource`` SHALL be a dictionary which SHALL contain the following keys: - **type** The ``type`` key SHALL be a string matching a resource type explicitly declared in the associated task’s slot. - **count** The ``count`` key SHALL be a positive integer indicating the number of tasks to execute per resource of type ``type`` occurring in the task’s slot. **total** The ``total`` key SHALL be a positive integer indicating the total number of tasks to be run across all task slots, possibly oversubscribed. **attributes** The ``attributes`` key SHALL be a free-form dictionary of keys which may be used for platform independent or optional extensions. **distribution** The value of the ``distribution`` key SHALL be a string, which MAY be used as input to the launcher’s algorithm for task placement and layout among task slots. Attributes ========== The value of the ``attributes`` key SHALL be a dictionary of dictionaries. The ``attributes`` dictionary MAY contain one or both of the following keys which, if present, must have values. Values MAY have any valid YAML type. **user** Attributes in the ``user`` dictionary are unrestricted, and may be used as the application demands. Flux may provide additional tools that can identify jobs based on ``user`` attributes. **system** Attributes in the ``system`` dictionary are additional parameters to a Flux instance that affect program execution, scheduling, etc. All attributes in ``system`` are reserved words, however unrecognized words SHALL trigger no more than a warning. This permits jobspec reuse between multiple flux instances which may be configured differently and recognize different sets of attributes. Most system attributes are optional. Flux modules SHALL provide reasonable defaults for any system attributes that they recognize when at all possible. Some common system attributes are: **duration** The value of the ``duration`` attribute is a floating-point number greater than or equal to zero representing time span in seconds. If ``duration`` is greater than zero, then the scheduler SHALL allocate the requested resources for the number of seconds specified in ``duration``. A ``duration`` value of ``0.`` SHALL be interpreted as unlimited or unset, and the scheduler SHALL use a ``duration`` that is the minimum of any configured ``duration`` default and the time span to the currently defined instance expiration. If no default ``duration`` is configured, and the current instance has no expiration, then resources shall be allocated without expiration. **preemptible-after** A floating point number greater than or equal to zero representing job run time after which the job MAY be canceled in order to make way for other jobs. A value of zero indicates that the job is immediately preemptible. The attribute SHALL NOT be set if the job is never preemptible. **environment** The value of the ``environment`` attribute is a dictionary containing the names and values of environment variables that should be set (or unset) when spawning tasks. For each entry in the ``environment`` dictionary, the ``key`` is a string representing the environment variable name and the ``value`` is a string representing the environment variable value to set. A ``null`` ``value`` represents unsetting the environment variable given by ``key``. The values provided here can be overridden per-rank by providing the ``attributes.environment`` dictionary under the target task. **cwd** The value of the ``cwd`` attribute is a string containing the absolute path to the current working directory to use when spawning the task. **queue** The value of the ``queue`` attribute is a string containing the name of the job queue this job should be submitted to. **bank** The value of the ``bank`` attribute is a string containing the name of an accounting bank requested for this job. Support for the ``bank`` attribute MAY depend on installation and configuration of optional framework components, such as an accounting or other plugin. **project** The value of the ``project`` attribute is a string containing the name of an accounting project requested for this job. Support for the ``project`` attribute MAY depend on installation and configuration of optional framework components, such as an accounting or other plugin. **dependencies** The value of the ``dependencies`` attribute SHALL be a list of dictionaries following the format specified in RFC 26. **constraints** The value of the ``constraints`` attribute SHALL be a dictionary expressing job constraints following the specification in RFC 31. **job** The ``job`` attribute is an optional dictionary containing job metadata. This metadata may be used for searching and filtering of jobs. Every ``value`` in the dictionary must be a string. The application is free to create keys of any name, however the following are reserved for special use: **name** The ``name`` key contains the name of the job. The default name of a job is the first argument of the command run by the user, or it can be set by the user to an arbitrary value. **shell** The ``shell`` attribute is an optional dictionary containing job shell metadata, such as configuration options. The application is free to create keys of any name, however the following are reserved for special use: **options** The ``options`` key is a dictionary containing configuration options for the job shell. A job shell and its plugins are free to define what keys and values should go into ``options``. **files** The ``files`` key SHALL consist of a dictionary in RFC 37 File Archive Format containing files that SHALL be made available to the job. The ``files`` key is intended to allow batch scripts, configuration files, and user defined input files to be embedded in jobspec. The job shell SHALL unarchive each file encoded in the archive into a temporary directory for use by the job. The following top-level files in the input archive are reserved: **script** The ``script`` key SHALL be reserved for a batch job script. **conf.json** The ``conf.json`` key SHALL be reserved for use as a subinstance configuration file. Example Jobspec *************** Under the description above, the following is an example of a fully compliant canonical jobspec. The example below declares a request for 4 "nodes" each of which with 1 task slot consisting of 2 cores each, for a total of 4 task slots. A single copy of the command ``app`` will be run on each task slot for a total of 4 tasks. .. literalinclude:: data/spec_14/example1.yaml :language: yaml Another example, running one task on each of four nodes. .. literalinclude:: data/spec_14/example2.yaml :language: yaml Schema ****** A jobspec conforming to the canonical language definition SHALL adhere to the following ruleset, described using JSON Schema [#f2]_. .. literalinclude:: data/spec_14/schema.json :language: json Basic Use Cases *************** To implement basic resource manager functionality, the following use cases SHALL be supported by the jobspec: Section 1: Node-level Requests ============================== The following "node-level" requests are all requests to start an instance, i.e. run a single copy of ``flux start`` per allocated node. Many of these requests are similar to existing resource manager batch job submission or allocation requests, i.e. equivalent to ``oarsub``, ``qsub``, and ``salloc``. Use Case 1.1 Request Single Resource with Count Specific Example Request 4 nodes Existing Equivalents +-----------------------------------+-----------------------------------+ | Slurm | ``salloc -N4`` | +-----------------------------------+-----------------------------------+ | PBS | ``qsub -l nodes=4`` | +-----------------------------------+-----------------------------------+ Jobspec YAML .. literalinclude:: data/spec_14/use_case_1.1.yaml :language: yaml Use Case 1.2 Request a range of a type of resource Specific Example Request between 3 and 30 nodes Existing Equivalents +-----------------------------------+-----------------------------------+ | Slurm | ``salloc -N3-30`` | +-----------------------------------+-----------------------------------+ Jobspec YAML .. literalinclude:: data/spec_14/use_case_1.2.yaml :language: yaml Use Case 1.3 Request M nodes with a minimum number of sockets per node and cores per socket Specific Example Request 4 nodes with at least 2 sockets each, and 4 cores per socket Existing Equivalents +-----------------------------------+--------------------------------------------------------+ | Slurm (a) | ``srun -N4 --sockets-per-node=2 --cores-per-socket=4`` | +-----------------------------------+--------------------------------------------------------+ | Slurm (b) | ``srun -N4 -B '2:4:*'`` | +-----------------------------------+--------------------------------------------------------+ | OAR | ``oarsub -l nodes=4/sockets=2/cores=4`` | +-----------------------------------+--------------------------------------------------------+ Jobspec YAML .. literalinclude:: data/spec_14/use_case_1.3.yaml :language: yaml Use Case 1.4 Exclusively allocate nodes, while constraining cores and sockets. Specific Example Request an **exclusive** allocation of 4 nodes that have at least two sockets and 4 cores per socket: Jobspec YAML .. literalinclude:: data/spec_14/use_case_1.4.yaml :language: yaml Use Case 1.5 Complex example from OAR Specific Example ask for 1 core on 2 nodes on the same cluster with 4096 GB of memory and Infiniband 10G + 1 cpu on 2 nodes on the same switch with bicore processors for a walltime of 4 hours — https://oar.imag.fr/docs/2.5/user/usecases.html#mixing-every-together Existing Equivalents +-----------------------------------+-----------------------------------------------------------------------------------------------------------------------------+ | OAR | ``oarsub -I -l "{memnode=4096 and ib10g='YES'}/cluster=1/nodes=2/core=1+{nbcore=2}/switch=1/nodes=2/cpu=1,walltime=4:0:0"`` | +-----------------------------------+-----------------------------------------------------------------------------------------------------------------------------+ Jobspec YAML .. literalinclude:: data/spec_14/use_case_1.5.yaml :language: yaml Use Case 1.6 Request resources across multiple clusters Specific Example Ask for 30 cores on 2 clusters (total = 60 cores), with 1 flux broker launched per node Jobspec YAML .. literalinclude:: data/spec_14/use_case_1.6.yaml :language: yaml Use Case 1.7 Request N cores across M switches Specific Example Request 3 cores across 3 switches, with 1 flux broker launched per node Existing Equivalents +-----------------------------------+-----------------------------------+ | OAR | ``oarsub -I -l /switch=3/core=1`` | +-----------------------------------+-----------------------------------+ Jobspec YAML .. literalinclude:: data/spec_14/use_case_1.7.yaml :language: yaml Use Case 1.8 Request an irregular range of a type of resource Specific Example Similar to 1.2, request between 3 and 30 nodes, but must be a perfect square Existing Equivalents +-----------------------------------+-----------------------------------+ | Slurm | ``salloc -N4,9,16,25`` | +-----------------------------------+-----------------------------------+ Jobspec YAML .. literalinclude:: data/spec_14/use_case_1.8.yaml :language: yaml Section 2: General Requests =========================== The following use cases are more general and include more complex slot placement and task counts. Use Case 2.1 Run N tasks across M nodes Specific Example Run ``hostname`` 20 times on 4 nodes, 5 per node Existing Equivalents +-----------------------------------+-------------------------------------------+ | Slurm | ``srun -N4 -n20 hostname`` or | | | ``srun -N4 --ntasks-per-node=5 hostname`` | +-----------------------------------+-------------------------------------------+ | PBS | ``qsub -l nodes=4,mppnppn=5`` | +-----------------------------------+-------------------------------------------+ Jobspec YAML .. literalinclude:: data/spec_14/use_case_2.1.yaml :language: yaml Use Case 2.2 Run N tasks across M nodes, unequal distribution Specific Example Run 5 copies of ``hostname`` across 4 nodes, default distribution Existing Equivalents +-----------------------------------+-----------------------------------+ | Slurm | ``srun -n5 -N4 hostname`` | +-----------------------------------+-----------------------------------+ Jobspec YAML .. literalinclude:: data/spec_14/use_case_2.2.yaml :language: yaml Use Case 2.3 Run N tasks, Require M cores per task Specific Example Run 10 copies of ``myapp``, require 2 cores per copy, for a total of 20 cores Existing Equivalents +-----------------------------------+-----------------------------------+ | Slurm | ``srun -n10 -c 2 myapp`` | +-----------------------------------+-----------------------------------+ Jobspec YAML .. literalinclude:: data/spec_14/use_case_2.3.yaml :language: yaml Use Case 2.4 Run different binaries with differing resource requirements as single program Specific Example 11 tasks, one node, the first 10 tasks each using one core and 4G of RAM for ``read-db``, the last task using 6 cores and 24G of RAM for ``db`` Existing Equivalents None Known Jobspec YAML .. literalinclude:: data/spec_14/use_case_2.4.yaml :language: yaml Use Case 2.5 Run command requesting minimum amount of RAM per core Specific Example Run 10 copies of ``app`` across 10 cores with at least 2GB per core Existing Equivalents +-----------------------------------+---------------------------------------+ | Slurm | ``srun -n 10 --mem-per-cpu=2048 app`` | +-----------------------------------+---------------------------------------+ Jobspec YAML .. literalinclude:: data/spec_14/use_case_2.5.yaml :language: yaml Use Case 2.6 Run N copies of a command with minimum amount of RAM per node Specific Example Run 10 copies of ``app`` across 2 nodes with at least 4GB per node Existing Equivalents +-----------------------------------+-----------------------------------------------------------------------------------------------------+ | Slurm | ``srun -n10 -N2 --mem=4096 app`` | +-----------------------------------+-----------------------------------------------------------------------------------------------------+ | OAR | ``oarsub -p memnode=4096 -l nodes=2 "taktuk -c oarsh -f $OAR_FILE_NODES broadcast exec [app]"`` | +-----------------------------------+-----------------------------------------------------------------------------------------------------+ Jobspec YAML .. literalinclude:: data/spec_14/use_case_2.6.yaml :language: yaml Use Case 2.7 Override the global environment Specific Example Run two different tasks, one with the global environment and one with an overridden environment (i.e., unset FOO and set BAR=2). Jobspec YAML .. literalinclude:: data/spec_14/use_case_2.7.yaml :language: yaml Use Case 2.8 Specify dependencies Specific Example Depend on two previously submitted jobs. The first job’s Flux ID (fluid) is known (``hungry-hippo-white-elephant``). The second job’s fluid is not known but its ``out`` dependency (``foo``) is known. Also provide an ``out`` dependency (``bar``) that other jobs can depend on. Jobspec YAML .. literalinclude:: data/spec_14/use_case_2.8.yaml :language: yaml Use Case 2.9 Specify constraints based on properties Specific Example Require that allocated resources have the ``amd-mi50`` property Jobspec YAML .. literalinclude:: data/spec_14/use_case_2.9.yaml :language: yaml References ********** .. [#f1] `YAML Ain’t Markup Language (YAML) Version 1.1 `__, O. Ben-Kiki, C. Evans, B. Ingerson, 2004. .. [#f2] `JSON Schema: A Media Type for Describing JSON Documents `__; H. Andrews; 2022