Promise

Computing the whole-model cross-entropy requires either a lot of patience, or distributed computation. For distributed programming, this toolkit has a Python implementaton of promises. A promise is essentially a function that can be evaluated (“fulfilled”). It may depend on other promises. This implicitly implies a tree of expressions that constitute the computation.

The term “function” is taken in the functional programming sense. A promise can declare what it needs as inputs, and it has a return value. It should not have any side effects, and each time it is evaluated with the same inputs, this must the same return value. These restrictions make it possible for a framework for distributed programming to be robust to failing nodes. The framework, not the promises, takes care of the loading results from and writing results to disk. This makes it possible to evaluate a promise on two machines at the same time, for example if one machine turns out to be slow, or if a job fails. Restricting functions to have no side effects therefore makes it possible for the framework to be inherently robust.

Interacting with the Promise framework involves setting up a structured promise (an expression tree, with promises that depend on other promises) in main_promise.py. It puts the main promise in the variable mainPromise. To evaluate the promise, ./dashboard.py starts a Python shell with the commands status(), run(), submit(). run() is useful in case of unlimited patience. submit() requires the Sun Grid Engine.

The results of promises are written to disk in a separate directory for each sub-promise. To allow constant-time identification and lookup, promises use hashing with consistentHash. The current implementation does not deal with hash collisions except by throwing an exception. It is therefore currently necessary to ensure that hash collisions do not occur.

Promises

Promises must be subclasses of Promise. It is possible to write one’s own subclasses, or to pass functions and values into ValuePromise, FunctionPromise, or BoundFunctionPromise. All functions and values that are passed in to promises need to have a consistent hash defined with consistentHash.

class promise.promise.Promise

Class that encapsulates a computable value. This can be thought of as a function. It may have parents, values that need to be computed before it can be computed itself. These can be thought of as arguments to the function.

This is an abstract class that needs to be subclassed.

computeHash()

Return the hash value. This must be a consistent hash value, e.g., using consistent_hash. This must take into account the parents as well. The return value is automatically memoised, which makes computing the hash value for large trees much faster.

This method must be implemented in each subclass.

evaluate(values)

Compute the value of this promise and return it.

This method must be implemented in each subclass.

Parameters:values – A dictionary mapping promises to values. Normally, all promises in parents() should be present.
getAll(predicate=None)

Yield all promises in the tree of promises exactly once, in order such that any promise only depends on previously-yielded ones. If a predicate is given, exclude promises for which predicate (promise) returns False, and their parent tree. predicate is called at most once for each promise.

getParentClasses()

Divide the promises in groups with the same parents and the same dependants.

Returns:A dictionary that maps parent classes (parents, dependants) to promises.
getShallow()
Returns:Potentially shallow version of this, which references

dependencies only by proxy. This reduces the pickled footprint of the promise. It may return self.

parents()

Return a frozenset of promises that fulfillment of this promise depends on immediately.

This method must be implemented in each subclass.

class promise.promise.ValuePromise(value)

A trivial Promise that encapsulates a fixed value.

Parameters:value – The value that the evaluation returns.
class promise.promise.FunctionPromise(function, parameters, friendlyName=None)

Promise that encapsulates a function. Its parameters have to be Promise’s.

Parameters:
  • function – The function to be evaluated to fulfill the promise.
  • parameters – List of Promise’s whose results form the parameters for the function. Note that it is possible for the same Promise to occur more than once.
  • friendlyName – Friendly name
class promise.promise.BoundFunctionPromise(function, parameters, friendlyName=None)

Promise that encapsulates a partially bound function.

Parameters:
  • function – The function to be evaluated to fulfill the promise.
  • parameters – List of arguments that form the parameters for the function. Each argument can either be a Promise, or a fixed values. Promises will be fulfilled before evaluating the promise; values will be passed into the function as is. Note that it is possible for the same Promise to occur more than once.
  • friendlyName – Friendly name
class promise.promise.PromiseProxy(promiseOrHash)

Promise that represents another promise, the relationship to which is defined through their hash values being the same. This is useful for pickling deeply nested promises.

This does not allow many operations. It does not remember its parents, or how to evaluate the promise. It can, however, be used to identify a dependency that must be retrieved from disk.

Parameters:promiseOrHash – The promise to be represented, or its hash value.

Fulfilling promises

It is possible to cache results of computation in memory, or on disk.

class promise.fulfill.MemoFulfill

Fulfill specialisation that remembers results in memory.

evaluate(promise)

Evaluate promise now, within the context of the remembered promises. The result is not remembered.

Pre :The parent promises must be remembered.
Returns:Result of the computation.
getUnrememberedPromises(mainPromise)

Yield all promises that are not remembered, in such order that they can be evaluated. This essentially returns the tree from the mainPromise up to the promises that are not remembered. This will yield nothing iff self.remembered (mainPromise) returns True.

remember(promise, value)

Store (remember) the result of the promise.

remembered(promise)
Returns:whether the result of the promise is remembered.
retrieve(promise)
Returns:the remembered result of the promise.
Throw :NotRememberedError if the result was not remembered.
class promise.fulfill_disk.FulfillDisk(mainPath, promiseNum=1)

Fulfill specialisation that remembers the results of promises in a directory structure on disk.

Parameters:
  • mainPath – the root path for evaluation and remembering results.
  • promiseNum – the rough number of promises. The framework aims to have about 64 subdirectories in one directory. This parameter determines the depth of the directory structure.
makePath(promise)

Prepare the directory. This also writes a file Promise.txt with a textual description of the promise. This is useful for debugging.

path(promise)
Returns:path in which and to which a promise is evaluated.
pickleableClone()
Returns:A clone of self without the cache.

Table Of Contents

Previous topic

Noise environment

Next topic

Utilities

This Page