Specifying options in the jobdef

Many of the sections in the jobdef accept key/value pairs, where the key is a keyword (i.e. starts with ":"). When fire! is executed near the bottom of your jobdef, the options from these sources are merged:

• base - from one or more files identified in (use-base ...)
• cluster - from your defcluster
• step - from the current defstep
• profiles - See PROFILES below
• command-line - options you specify on the lemur command line

The sources at the bottom take precedence.

SYNTAX

If you don't know Clojure, a few general rules:

• Maps are wrapped in {}
• Keys begin with colon
• Strings are wrapped in double quotes. These are java strings, so you can include special characters using Java syntax. E.g. in "hello\n\tworld" n is a newline, \t is a tab.
• A list is wrapped in []. Commas between list items are optional and are treated like white-space. E.g. ["foo" "bar", "baz"] is a list of 3 strings. You can also use spaces, tabs and new-lines between list items.

The Values

The value for each key is evaluated at runtime according to its type.

• String - "foo"

   Can contain variables which will be substituted at runtime. The variables
   are are enclosed in ${}.  For example
     "${app}-${runtime}"
   which would evaluate to something like "sample-2012-01-27-1200"

• Collection (or map) - ["foo" "bar"] or { :foo "1" }

   Each value in the collection is evaluated (recursively) using these same rules.

• [Advanced option] A clojure function -

   If you need some more complex logic, you can write a clojure function. If it is
   a 1-arg function, it will be called with the options map (eopts).  A 0-arg function,
   will simply be called. For example, you could define a function like:

     (fn [eopts] (str (:app eopts) "-" (:runtime eopts)))

   which would evaluate to the same value as the string example above. Note that
   functions should be idempotent, and may be executed many times.

   A helper, lfn, is available. lfn does two things. It creates a memoized function
   (i.e. it only executes once for a given set of arguments-- after that the return
   value is cached). And, second, lfn extracts the keys listed in its first
   argument from eopts.  Example:

      (defcluster my-cluster
        :foo 100
        :bar "a"
        :baz (lfn [foo bar] (str foo "-" bar)))

   would assign a value of "100-a" to :baz

• Anything else - 1 or :foo

   Everything else is just its literal value

The variables available for string interpolation (or available in the eopts map in the case of a function value), are all of the keys available in the base, cluster, step and command-line options. I.e. all the key values mentioned at the start of this section. This includes some key/values that are defined automatically:

:env - The environment (as per env.properties) :runtime - The current time as "yyyy-mm-dd-hhmm" for UTC :remaining - A collection of String arguments, which are left over after the command-spec is used to parse the command line. :cluster-name - The symbol at the start of your defcluster block :step-name - The symbol at the start of your defstep block :jobdef-file - The basename of the jobdef file (minus the "-jobdef.clj" suffix) :run-id - Used in the base path in S3, among other things. Has a value of /-<8-char-uuid>, where is the value of :jobdef-file by default :run-path - Equal to :run-id, except for 'lemur local', in which case it is equal to :app

In addition, System Environment Variables are available by prefixing the name with 'ENV.'; for example "my home dir is ${ENV.HOME}".

You might see an option represented in three different ways. The following all refer to the same value:

 :foo          - When used as a key in the jobdef
 --foo         - When used on the command line
 "${foo}"      - When used inside a string