Introduction to Knowbot Programming

[Contents] [Prev] [Next]
Table of Contents

Introduction

A Knowbot Program written in Python consists of a main module and any number of additional modules. When the KP is executed, either on its initial submission or after a clone() or migrate() operation, the main module is loaded and executed with the module name (the variable __main__) set to '__main__'. Additional modules must be imported by the main module (or by some other module that is imported, directly or indirectly) in order to be available. All this is just like it is for normal Python programs.

The Main Module and the KP Class

The main module should define a class called "KP" which will be instantiated and invoked automatically by the KP supervisor after the main module has been loaded. (There's a way to choose a different name for this class but the KP launching tools don't support it yet.) The KP class should define a __main__() method taking one argument. If it has a constructor (__init__() method), it should take no arguments.

The KP class is instantiated differently depending on whether this is the initial submission or the result of a clone() or migrate() operation:

On initial submission:
A fresh instance of the KP class is created, by calling the KP class without arguments.

On clone() or migrate():
A pickled copy of the KP instance that was present in the KP at the time of the clone() or migrate() operation is unpickled. See the Python documentation for the pickle module for the restrictions that apply to instance variables so that they can be pickled, and how to customize the pickling and unpickling process.

The KOS Bastion

Once the KP instance has been created, its __main__() method is invoked with a single argument, the ``KOS bastion''. The KOS bastion defines a number of methods that enable communication with the KP supervisor and other KPs, including the clone() and migrate() operations. Its interface is documented in the KOS Bastion Methods section.

When the __main__() method returns, the KP terminates and its submittor (strictly speaking, the ``Reporting Station'' responsible for it) is notified of its termination. Destructors for objects that are still alive at this point may or may not be invoked (so don't count on them either way).

It is permitted to store the KOS bastion as an instance variable; after migration (or cloning) this instance variable will automatically refer to the KOS bastion for the current KP instance's supervisor.

Unhandled Exceptions

Python KPs can catch exceptions using the try-except statement. Unhandled exceptions are caught by the KP supervisor and cause termination of the KP. The SystemExit exception is handled specially in that is it handled like normal termination. This is analogous to its treatment in ordinary (non-KP) Python programs. All other unhandled exceptions cause the termination to be treated as "abnormal". For abnormal termination, a stack trace of the program is normally printed by the submittor; for normal termination, nothing is printed.

Restricted Execution Mode

A KP is executed in Python's restricted execution mode. This mode restricts the damage that a KP can to to its host environment. For this purposes, restrictions on which built-in (or dynamically loaded) modules and functions can be used exist, and some language features are disabled. Two examples of the latter: the function object attribute func_globals and the class and instance object attribute __dict__ are unavailable. The following sections discuss the restrictions on built-in functions and modules.

For full documentation on the functions and modules discussed here, see the Python Library Reference.

Restrictions on Built-in Functions

The following built-in functions are unavailable or restricted:

open()
This function is currently not available. In the future, it will be available, but restrictions will apply to which files are accessible in addition to the standard filesystem permissions.

Restrictions on Standard Library Modules

The following built-in modules are available without restrictions:

The following built-in modules are available with some restrictions:

No other built-in modules are available.

Note that even though there are no explicit restrictions on modules implemented in Python, the absence of built-in modules like socket means that any standard library modules that use it are also effectively unusable.

Support Modules

In addition to a selection of standard modules, some modules specific for KP programming are available. These are documented in more detail in their own manual pages; we give a summary here.

Modules Commonly Used by KPs

Other Modules Relevant to KPs

Other Typical Operations

Without turning into a tutorial, we discuss some common operations that a KP might involve in. This complements the KOS API documentation.

Interacting With KOS Plugin Services

A typical KOS service is implemented as a KOS Plugin Module. Many services are structured as a ``factory'' object shared by all clients. Clients interact with the factory only to create a personalized ``interface'' object with which they carry out all further interaction. Thus, there are the following steps:

1. factory = kos.lookup_service(type, name).Open()
Here, `kos' refers to the KOS bastion passed as an argument to the __main__() method; type is the service type, found in the service's documentation, a string of the form "module.interface"; name is the service name, a string (not containing slashes) also found in the service's documentation. Some services have multiple implementations, in which the service name can vary; the service type is normally always a constant, known ahead of time (if you don't know the type, you presumably don't know how to interact with it).

Note the .Open() call tacked on the end. The kos.lookup_service() call itself returns a descriptor for the service which is not directly usable. The Open() method on the descriptor returns an interface stub that can be used to interact with the service.

2. interface = factory.create_interface(arguments)
This step creates the interface object with which this KP will be interacting. The method name and the arguments it requires are to be found in the service's documentation.

3. interface.method(arguments) ... (repeat)
You're on your own with the service documentation now.

4. interface.close()
Many interfaces require that you call an explicit close() method when you're done, so that the service can deallocate resources (like a Unix subprocess!) associated with this instance of the interface. (ILU's distributed garbage collection (using objects marked COLLECTIBLE) will normally take care of this, but it is always considered good style to make an explicit close() call in the client when the service is no longer needed.)

Note that both the factory and interface objects are ILU surrogate objects. It is often the case that the interface object refers to a different server process than the factory process, though.

Interacting With Services Provided By Other KPs

This is generally done in the same way as interaction with plugin services, except that usually the KP service won't use an interface factory. So there are only two steps:

1. interface = kos.lookup_service(type, name).Open()

2. interface.method(arguments) ... (repeat)

Providing A Service To Other KPs

In the current KOS version, you can only provide a service if its ILU ISL file is present in the koe/interfaces directory and stubs for it have been made in the koe/interfaces/stubs directory. (We will eventually provide a way whereby the interface definition can be carried around in the suitcase of the KP instead.) Given an ISL file and an interface in it, here's how you create a service implementing the interface:

Step 1

Choose an interface. If you decide to create a new one, you have to make sure that its ISL file and stubs are present on every target KSS where you want to run your KP. It's easier to use an existing interface. Fortunately, there's a ``generic'' interface that lets you pass arbitrary lists of strings between client and server. See the source code for the GenericImpl module.

Step 2

Your KP should define a class that implements the interface. The class needn't inherit from anything, and you can pick the class name freely. The class should have methods implementing each of the methods of the chosen interface (the method names should correspond -- you cannot choose them freely). The arguments and return value of these methods should correpond to the arguments and return value of the corresponding method specified in the ISL file (this is the same as for ILU true servers, except those need to inherit from the __skel class; see the ILU documentation for details of the mapping, in particular concerning arguments of type OUT or IN OUT, and some more esoteric types such as ILU records).

Step 3

Pick a name for your service. Some requirements on the name are:

Step 4

Create an instance of your service class.

Step 5

Bind your instance to a service name and type. This is done using a statement like 

desc = kos.bind_service(name, instance, type)
where `kos' refers to the KOS bastion, and the arguments name, instance, type are the service name you chose, the instance you created, and the type name you chose (of the form "module.interface"), respectively. The return value (`desc') is a descriptor for the service, to be saved for later to unbind the service.

Step 6

Activate your service. This done using the statement 

kos.run()

This call starts the ILU main loop. It will not return control until the ILU main loop is stopped again by the statement 

kos.stop()

When a client uses your service, the ILU main loop will call your instance's methods. (The kos.stop() call could be executed by some method of your service, e.g., a call signalling that no client is in need of your service any more.) Note that an exception occurring in a method called from ILU does not exit the ILU main loop. ILU simply prints a traceback and continues servicing requests; the client that made the failing request receives an IluGeneralError exception.

Step 7 (optional)

Unbind your service. In general, it is not necessary to unbind your service if your KP is going to exit. You may explicitly unbind your service using the statement 

kos.unbind_service(desc)
where `desc' is the descriptor for the service, the saved return value of the kos.bind_service() call.

Notes

Carrying Large Amounts of Data -- The Suitcase

Every KP carries a ``suitcase'' along. This is a form of ``external'' or ``bulk'' storage which is stuctured and accessed roughly the same way as a file system. The suitcase is moved along with the KP on migration and copied into (but not shared between!) clones. It does not automatically persist when a KP terminates, but KP submission tools have a way to retrieve selected parts from the suitcase on termination. You can think of the suitcase as an individual, portable file system with a limited persistency.

The standard idioms for accessing the suitcase looks something like this: 

class KP:
    def __main__(self, kos):
        # Access the suitcase filesystem
        sfs = kos.get_suitcase()

	# Create a directory
	sfs.mkdir("/directory")

	# Write a file
	f = sfs.open("/directory/filename", "w")
	f.write("something\n")
	f.write("something else\n")
	f.close()

	# List a directory
	names = sfs.listdir("/directory")

	# Change current directory
	sfs.chdir("/directory")

	# Read a file
	f = sfs.open("filename", "r")
	while 1:
	    line = f.readline()
	    if not line: break
	f.close()

	# Remove a file
	sfs.remove("filename")

	# Remove a directory
	sfs.rmdir("/directory")

Complete documentation of the suitface filesystem interface can be found in the documentation for the Suitcase module.

Managing Clones

While migration is straightforward (see the tutorial), a program using clones generally needs to implement some form of communication between clones, or between clones and the original. There are a number of different communication patterns:

  1. Clones need to send data back to the ``master'' when they are done.
  2. Clones need to communicate regularly with the master during their operation.
  3. Clones need to communicate with each other.
  4. The master needs to broadcast some information to all clones at certain times.
  5. All of the above (or at least a combination of more than one).

In all cases, a problem is that you can only use kos.lookup_service() to request a service on your own KSS.

In case 1, the simplest solution may be to migrate back to the KSS where the master resides (assuming the master doesn't move), and initiate communication there. The clone can make the kos.lookup_service() call upon arrival at the original KSS.

In case 2, there's no point in migrating to do the communication, so the clone has to connect with the master which is residing on a different KSS. In this case, there are two solutions. Either the clone is initially sent to the master's KSS, where it makes the kos.lookup_service() call, stores the result as an instance variable on the KP instance, and then it migrates to its destination. Alternatively, it migrates directly to its destination, and uses the approach for case 3 to look up the master's service.

Case 3 really has to deal with the fact that you can only use kos.lookup_service() for local services. Fortunately, kos.lookup_service() mostly just does a namespace lookup, and you can do that yourself if you know the name of the KSS where the service resides. For example, 

import nstools
obj = nstools.WorldOpen(
    "/kos/kssname/services/servicename",
    "type")

Here, kssname is the name of the KSS where the service is running, servicename is the name of the service, and type is the service's type. The result is the interface object which allows interaction with the service (no separate .Open() call is needed when you use nstools.WorldOpen()).

Note that the nstools.WorldOpen() call can raise a nstools.BadPathError exception. This can happen when two clones travel with different speed to their destinations, and the client clone attempts to communicate with the server clone before the server clone has registered its service. It is best to write a loop with a delay (e.g., time.sleep(1)) that retries the operation if it fails. A timeout (e.g. a limit of 60 cycles through the loop, after which you give up) is also useful, in case the server clone was somehow terminated.

Case 4 can be solved the same way as case 3, but here the master looks up the clone's service. Because the travel time for clones is indeterminate, the loop-retry-timeout approach should be used.

Case 5 can be solved by combining the above techniques.

There's a caveat for all cases: a KP cannot simultaneously do useful work and operate a service. This is because in order to operate a service, kos.run() must be invoked, which means that the program is suspended, waiting for incoming service requests. The best solution is to design the communication patterns to avoid this -- e.g., some KP's can be servers and others can do useful work and be clients. It is possible to do some useful work first and then start operating a service, perhaps waiting for the cows, ehm, clones to come home. It is also possible to interrupt the work frequently to make a dummy RPC call.

Using Trigger Variables

Trigger variables provide an alternative means for communication between KPs on the same KSS. In the current system they are helpful because they provide a way for KPs to communicate with each other without registering a service. The trigger interface is not particularly useful, and its implementation in the kernel is unnecessary. Trigger variables are likely to be eliminated in a future version, in favor of a generic event interface that KPs can use.

Catching Exceptions

Catching exceptions raised by service invocations in a KP requires some additional effort, because the ILU stub module that defines the exception names cannot be imported from a KP (and exceptions must be referenced by their object identity, not by their string value). The KOS bastion provides an interface to access these exceptions. There are two versions: kos.exception("mod.name") returns the exception `name' defined in module `mod'; kos.all_exceptions("mod") returns a dummy module object containing all exceptions defined in module `mod'.

It is customary to store these exceptions in global variables at the start of the KP's __main__() method, so that the rest of the KP's code can use the familiar style of referencing exceptions. For example: 

class KP:

    def __main__(self, kos):
        global mod
        mod = kos.all_exceptions("mod")
	...other stuff...

    def some_other_method(self):
        try:
	    some_remote_operation()
	except mod.SomeException:
	    ...handle the exception...

The same procedure to access exceptions can be used by a KP that provides a service and needs to raise an exception defined in the service's interface definition.

In order to catch ILU exceptions, KPs can import the module ilu. which defines (only) the ILU exceptions IluGeneralError, IluProtocolError, and IluUnimplementedMethodError.

Table of Contents
[Contents] [Prev] [Next]
Copyright © 1998 by the Corporation for National Research Initiatives.