NAME
-
grid - peer-to-peer data distribution
SYNOPSIS
-
load alphabet
typeset /grid
type /grid/endpoint
DESCRIPTION
-
Grid
is a typeset for
alphabet
(see
sh-alphabet(1))
which enables
allows direct interconnection of
remote and local processing components. It defines one new type,
endpoint,
which represents
a place in the network to which two parties can
connect and exchange data.
In the following descriptions, if a type
is not
endpoint
or a type defined in the root typeset (see
alphabet-main(1)),
it is assumed to be of type
/string.
Modules currently provided within
the
/grid
typeset include:
- farm [-lnkavA] endpoint addr tasktype [arg...] -> endpoint
- Farm
connects to a grid labour exchange (see
scheduler(4))
at
addr,
starts a new job of type
workflow,
and passes all the data read from
endpoint
to be processed by the currently available labour.
The data is split into records, each one of which will be
processed on a worker node by
tasktype,
with its associated
arguments.
Other than
-A,
which specifies unauthenticated access to the
scheduler,
the various options are all passed verbatim to
workflow:
-l
causes it to split its input on newline-separated records;
-n
specifies that no record separation is necessary on output;
-k
specifies that intermediate data for failed tasks should
be kept around;
-a
specifies that intermediate data for all tasks should
be kept around, and
-v
specifies that
workflow
should produce a wordy description of what it is doing.
- local endpoint -> /fd
- Local
reads everything from
endpoint,
and writes it to
fd.
- remote [-a addr] fd -> endpoint
- Remote
is the inverse of
local:
it reads data from
fd
and writes it to a newly created endpoint.
If
-a
is given,
addr
specifies the network address of an endpoint server
on which to create the new endpoint.
- rexec [-A] endpoint addr cmd -> endpoint
- Rexec
connects to a remote execution server at
addr
(unauthenticated if
-A
is specified), and arranges to execute the
alphabet
expression
cmd
there. The expression should be compatible with usage
fd -> fd.
Data from the argument
endpoint
will be piped through this expression, and
made available as the resulting
endpoint
endpoint.
EXAMPLES
-
The examples below that a local endpoint is available, and the following
alphabet
declarations:
load alphabet
typeset /grid
type /string /endpoint /fd
import /grid/local /grid/remote
autoconvert fd endpoint remote
Set up a rendering pipeline:
-{/read /tmp/somedata |
remote |
rexec tcp!node1!rexec "{(/fd); /filter $1 "{os render_stage1}} |
rexec tcp!node2!rexec "{(/fd); /filter $1 "{os render_stage2}} |
/create /tmp/somedata.result
}
SOURCE
-
/appl/alphabet/grid.b,
/appl/alphabet/gridtypes.b
/appl/cmd/grid/*.b
SEE ALSO
-
sh-alphabet(1),
alphabet-main(1),
sh(1)