worldroot - Replicated root server for the distributed namespace used by the KOE.
worldroot [-f] [-h] [-n name] [-p port] [-q] [-s] [-v]
The worldroot server manages a local version of the top two levels of the distributed KOE namespace. A single worldroot server is generally shared between all KSSs at a local site. We will call a collection of KSSs that share a worldroot server a ``cluster.'' The worldroots of different clusters (called worldroot peers) exchange information about updates so they all share the same view of the world. Updates propagate asynchronously so clusters may temporarily have different views of the world. This is intentional, since synchronous updates wouldn't scale to a KOE with zillions of clusters.
For testing purposes, it is often useful to run a cluster in isolation (i.e. not linked to other clusters). By default, the worldroot server runs this way. To operate with a shared namespace, the worldroot must be configured with the names of peer worldroot servers.
The worldroot program supports the following command line options.
The worldroot server uses the following files. All filenames are relative to the $KOSROOT directory.
The file format is as follows: each line contains two whitespace-delimited fields. The first field is the name of a peer, the second is the last known SBH of its worldroot replicator. Blank lines and lines whose first non-blank character is a hash sign (comments) are ignored. Other lines that do not contain exactly 2 files are flagged as errors and ignored.
It is not necessary to specify all known worldroot servers as peers. It is sufficient that the peer-to-peer relationships define a bidirectional connected graph. For redundancy, it makes sense to have at least two peers so that a server is not isolated when its only peer goes down. Cycles in the graph are not a problem; updates received more than once are ignored and are not propagated further.
A cluster's worldroot needs to be running when a new KSS is started. Normally the koeboot utility starts the worldroot before starting any service stations.
The worldroot server only manages the top two hierarchical levels in the KOE namespace: the root context and the contexts immediately below it (for an explanation of the concept of a namespace context, see the nstools module). In the current implementation, the root context cannot be modified. It contains four standard contexts named "kos", "tools", "types" and "replicators", which are used as follows:
It is possible to restart the worldroot server after service stations have started, e.g. if the worldroot crashed. The namespace contexts run by the worldroot will be inaccessible while the server is being restarted. As a result, KP operations like migrate() will raise exceptions (because the "kos" context is unavailable). The service stations, however, will survive a temporary worldroot service interruption however.
Replicated operation is triggered by the presence of one or more peer servers in var/wr-peers.tab (described above). In replicated operation, any updates in the contexts managed by a worldroot server are asynchronously propagated to all its currently reachable peers. Updates received from one peer are propagated to all other peers.
Let's consider a simple example with two clusters, A and B. When a program in cluster A does a lookup of "kos/monty" relative to the worldroot context, its request will be processed by the worldroot server for cluster A; likewise in cluster B. The lookups are completely independent; if cluster B is unreachable, it doesn't affect the lookup in cluster A at all. Both lookups, however, will come up with an SBH for the same object. The object is not replicated; it might live in either cluster, or in a third cluster. The object must be alive and reachable in order to use it, but not to look it up.
Now let's look at an update made to a context managed by a worldroot server. Assume that a new KSS "repoman" is added to cluster A. When it is started, the KSS makes a call to its cluster's worldroot server to create a binding for "kos/repoman". When the worldroot server replies, it has added the binding to its local copy of the "kos" context and stored it in its current state database (var/wr-currentdb*, see below). The worldroot server propagates this update to all its peers, who propagate it too all their peers, and so on, until it has spread to the entire KOE. During the time it takes for the update to propagate, it is possible that a lookup of "kos/repoman" will fail on some clusters and succeed on others. This inconsistency is temporary: As long as all peers remain connected, they will receive the update eventually.
When a worldroot server is down, it can miss updates. Therefore, when a worldroot server starts up, it first reconstructs its state from its local database (var/wr-currentdb*, see above) and then connects to its peers and asks them to retransmit the updates that it has missed. Updates received during recovery are not propagated to other peers. It is therefore possible, though not very likely, that some updates are not propagated across the entire KOE. The KOE-wide sweep operation, described in the next section, can correct the resulting inconsistency.
A KOE-wide sweep operation is initiated by starting any worldroot server with the -s option. During a sweep, updates are collected from all known worldroot servers, processed, and then distributed back to all participating servers. After the sweep all participants are synchronized in their view of the world. The sweep operation does not just operate on the collection of peers found in var/wr-peers.tab; instead, it accesses all known worldroot servers. This collection is found in the "replicators" context of the KOE namespace. The sweep operation can only succeed when all participating servers are up. If there is a failure, the sweep operation is aborted and the initiating server transitions into normal service mode.
Sweeps can be expensive. Normally sweeps are not necessary because the recovery information exchanged with peers at server startup time (including after crashes) should bring a server's world view up to date. Pathological situations are possible, however, when updates are not propagated due to extended down time of several servers or network connections. Sweeps take care of recovery after such failures.