[manual index][section index]


grid - peer-to-peer data distribution


load alphabet
typeset /grid
type /grid/endpoint


Grid is a typeset for alphabet (see sh-alphabet(1)) which enables allows direct interconnection of remote and local processing components. It defines one new type, endpoint, which represents a place in the network to which two parties can connect and exchange data.

In the following descriptions, if a type is not endpoint or a type defined in the root typeset (see alphabet-main(1)), it is assumed to be of type /string.

Modules currently provided within the /grid typeset include:

farm [-lnkavA] endpoint addr tasktype [arg...] -> endpoint
Farm connects to a grid labour exchange (see scheduler(4)) at addr, starts a new job of type workflow, and passes all the data read from endpoint to be processed by the currently available labour. The data is split into records, each one of which will be processed on a worker node by tasktype, with its associated arguments. Other than -A, which specifies unauthenticated access to the scheduler, the various options are all passed verbatim to workflow: -l causes it to split its input on newline-separated records; -n specifies that no record separation is necessary on output; -k specifies that intermediate data for failed tasks should be kept around; -a specifies that intermediate data for all tasks should be kept around, and -v specifies that workflow should produce a wordy description of what it is doing.
local endpoint -> /fd
Local reads everything from endpoint, and writes it to fd.
remote [-a addr] fd -> endpoint
Remote is the inverse of local: it reads data from fd and writes it to a newly created endpoint. If -a is given, addr specifies the network address of an endpoint server on which to create the new endpoint.
rexec [-A] endpoint addr cmd -> endpoint
Rexec connects to a remote execution server at addr (unauthenticated if -A is specified), and arranges to execute the alphabet expression cmd there. The expression should be compatible with usage fd -> fd. Data from the argument endpoint will be piped through this expression, and made available as the resulting endpoint endpoint.


The examples below that a local endpoint is available, and the following alphabet declarations:
	load alphabet
	typeset /grid
	type /string /endpoint /fd
	import /grid/local /grid/remote
	autoconvert fd endpoint remote
Set up a rendering pipeline:
	-{/read /tmp/somedata |
		remote |
		rexec tcp!node1!rexec "{(/fd); /filter $1 "{os render_stage1}} |
		rexec tcp!node2!rexec "{(/fd); /filter $1 "{os render_stage2}} |
		/create /tmp/somedata.result


/appl/alphabet/grid.b, /appl/alphabet/gridtypes.b


sh-alphabet(1), alphabet-main(1), sh(1)

ALPHABET-GRID(1 ) Rev:  Tue Mar 31 02:42:38 GMT 2015