submit - Submit KPs to Service Stations

Name

submit - tool to submit KPs to service stations

Synopsis

submit [options] name [file[,file,...]] [-- [argument] ...]

The submit utility is a command line tool to launch Knowbot Programs. By default, the standard I/O streams of the launched KPs and their clones are redirected to the I/O streams of the submit program, and submit waits until all launched KPs, including any clones that they may have spawned, have terminated. Options can modify this behavior.

Options generally have a long form (two dashes followed by a word) and a short form (a single dash followed by a letter).

Source file arguments are generally KP source files. By default, submit launches one KP for every comma-delimited list of source files, where a single filename is a list with one-element. Using the -1 argument, submit will launch a single KP consisting of all the files listed on the command line.

Any arguments appearing after a -- marker are passed on to the launched KP(s) as options and/or arguments.

Options

-k name
--kernel name: Kernel name (default: your hostname, with the domain name stripped off).
-m name
--monitor name: External reporting station name; a name looked up in the /tools context. Use this if you want the I/O to be handled by another process, such as the reporting station provided by the rs utility. (default: internal reporting station) (For experts only.)
-n
--nogo: Do everything except submit the KP(s). (A debugging feature.)
-i
--noioserver: Inhibit redirection of the KP's I/O (use kernel's console). (For experts only.)
-f dir
--freeze-dry dir: Store MIME representations of completed KPs (including clones) in the directory dir. As each KP terminates, the MIME representation of its state is written out in a file named reporting-id.mime in this directory. (A debugging feature.)
-x
--showmime: Just before submission, write the submitted KP as MIME to stdout. (A debugging feature.)
-1
--single: Only submit a single KP; arguments are modules.
-s suitcase-spec
--suitcase suitcase-spec: Suitcase specification (see below).
-r suitcase-spec
--read-suitcase suitcase-spec: Copy a file from the suitcase to the local filesystem when the KP finishes (see below).
-v
--verbose: Verbose mode -- be chatty about KP status changes etc. May increase verbosity if repeated. (A debugging feature.)

Details

Sources

Each non-option argument is either a MIME-encoded KP ("-" or a file ending in ".mime") or a comma-separated list of Python modules. If omitted or when an argument is "-", a single MIME-encoded KP is read from standard input.
If --single is given, a single KP is submitted and each argument specifies a module that becomes part of a list; thus, "submit --single a b c" is equivalent to "submit a,b,c". This may be handy if you rely on filename completion in your shell. In this case, '-' loses its special meaning and no MIME files can be submitted.
KP arguments

In order to pass command line options or arguments to the launched KP(s), specify them after a -- marker, after all the sources. These arguments are passed to each of the launched KPs. The KP can access them through inspection of sys.argv (sys.argv[0] is the KP name; the arguments are placed in sys.argv[1], sys.argv[2] and so on).
Suitcase specifications

One or more --suitcase options control the contents of the suitcase sent along with the KP. The suitcase is only added to the first KP sent. The suitcase-spec values can have the following forms, where path is the pathname in the suitcase where the data is stored:

path:file
Insert a file in the suitcase.

path:directory
Insert an entire directory tree in the suitcase.

path:=data
Insert literal data in the suitcase.

Suitcase reading specifications

One or more --read-suitcase options direct the transfer of suitcase contents back to the filesystem upon completion of the first KP launched. The specification is similar to that for the --suitcase option but only the form path:file is supported. If path is found in the KP's suitcase, it is copied to file. Suitcase files not listed in --read-suitcase options, or suitcase files in other KPs than the first KP launched by this submit instance, or in clones of the first, are ignored.
Examples

Assume we have two test KP source files, test1.py and test2.py; test1.py can be run stand-alone; test2.py requires the modules mod1.py and mod2.py. We also have two data files test1.dat and test2.dat. Here are some example runs:

submit test1.py
Run the test1.py KP.

submit test1
Run the test1.py KP. (The .py extension can be omitted.)

submit test1 -- -x
Run the test1.py KP, passing it the option "-x".

submit --suitcase input:test1.dat test1 -- input
Run the test1.py KP, placing the contents of the file test1.dat in its suitcase under the name input, and passing it the argument "input".

submit --suitcase a:test1.dat --suitcase b:test2.dat test1 -- a b
Run the test1.py KP, passing it the data files in its suitcase under the names "a" and "b", and passing it the arguments "a" and "b".

submit --suitcase snow:="Melted snow" test1 -- snow
Run the test1.py KP, passing it a suitcase file named "snow" containing the line "Melted snow" (terminated by a newline character), and passing it the argument "snow".

submit test1 test1
Run two simultaneous copies of test1.py.

submit test2,mod1,mod2
Run the test2.py KP.

submit --single test2 mod1 mod2
Run the test2.py KP.

submit test1 test2,mod1,mod2
Run the test1.py and test2.py KPs simultaneously.

submit test1 test2 mod1 mod2
Probably a mistake. This would launch four KPs: test1, test2, mod1, and mod2. The latter three would most likely all fail.

Interaction

Normally, anything written by the KP(s) to sys.stdout, e.g., output of the print statement, or to sys.stderr will go to the standard output or error stream of submit. Likewise, if a KP reads from sys.stdin or uses the input() or raw_input() functions, a line will be read from submit's standard input.
Errors and warnings from submit and messages from its verbose mode (--verbose) will also go to its standard error stream. All these messages are preceded by the string "submit:" for easy filtering.
Interrupt Handling

When the user hits the interrupt key (usually Control-C), submit takes the following actions.

It sets a flag that disables cloning. Whenever a KP launched by submit needs to clone, it requests permission from submit; once the no-clone flag is set these requests are denied.
It sends KillKP() requests for each of the KPs it knows about to the kernels where it believes they are running. This will normally terminate those KPs.
It continues to act as an I/O server for the launched KPs and waits until they have all been terminated.

Advanced Usage

The --verbose option prints various and sundry messages. It can be used to monitor a KP's migration and cloning patterns. It will also report the time elapsed for the submission and the execution. When repeated, it prints even more information. (Currently, this prints "Watchdog..." once a second to indicate that the internal "watchdog timer" is operational.)
The --monitor name option can be used to direct a KP's reporting events to another process. This feature has not yet been used in the base KOE distribution. It could be used to direct monitoring (but not I/O) of a KP to a running rs instance.
The --noioserver option can be used to avoid setting up an I/O server in the submit process. The KP's I/O will be directed to the console of the kernel where it is running, and will most likely be lost.
The --nogo option is useful to debug submit itself; it will go through all the motions of launching a KP except the actual launch. Best combined with --verbose.
The --showmime option is best combined with the --nogo option. This can be used to produce a MIME representation of the KP to be launched. The MIME representation is useful for inspection or for later launching from a filename ending in ".mime".
Bugs and Caveats

KP arguments containing spaces and quotes will not always be transferred to the KP correctly.

When multiple KPs are writing to sys.stdout or sys.stderr simultaneously, their output may be interleaved randomly. (The text written by a single call to sys.stdout.write() will always be kept together.)

When multiple KPs are reading from sys.stdin simultaneously, bad things may happen. Bad things may even happen when one KP is reading from sys.stdin and another KP is writing to sys.stdout or sys.stderr. The problem is that while submit is blocked reading from its standard input stream, it can honor no other I/O requests.

The interrupt handling is not perfect. There's a race condition under which submit does not know the correct host where a KP is executing. It is also possible that submit never learns about a terminated KP, and thus hangs forever. In this case, the best thing is to suspend the submit process (e.g., using Control-Z) and to kill it manually from the UNIX shell.

There is no way for a KP to catch termination caused by a user interrupt or the killkp utility.

See Also

There is a GUI tool with roughly the same functionality, rs. See also killkp, a utility to kill runaway KPs.