Conversation with #inferno at Tue Mar 29 19:58:16 2011 on powerman-asdf@irc.freenode.net (irc)
(20:02:43) mjl-: GriffenJBS: i'm not really into js
(20:02:55) mjl-: but perhaps if the glue code really is small, it could be a fun project
(20:04:13) mjl-: heh, the node.js website is a bit funny
(20:04:22) GriffenJBS: how so?
(20:04:22) mjl-: Thread-based networking is relatively inefficient and very difficult to use. See: this and this. 
(20:04:29) mjl-: ok can give them the first
(20:04:44) mjl-: but then they say:  Node will show much better memory efficiency under high-loads than systems which allocate 2mb thread stacks for each connection. 
(20:04:49) mjl-: orly
(20:04:53) mjl-: duh :)
(20:05:41) GriffenJBS: yeah, think of how many green programmers you've met and their solution is 'new task = new thread'
(20:05:59) GriffenJBS: scaling past a few hundred starts to kill you in context switching
(20:08:42) mjl-: i tend to make new threads for new tasks
(20:08:44) mjl-: but that's in inferno
(20:08:48) mjl-: they don't have 2mb stacks
(20:09:00) mjl-: i'm pretty sure most pthread implementations don't have 2mb stacks either
(20:09:11) mjl-: though i was shocked a bit to see them set to 512kb by default
(20:09:38) GriffenJBS: even if they only have 8k stacks, context switching is a costly operation in most hardware
(20:09:40) mjl-: but hey, if i was a pthread author, i wouldn't want to hear people claiming the lib is crap because their code overflows stack all the time
(20:09:56) mjl-: true
(20:10:11) mjl-: but i don't know where exactly the limitations lie
(20:10:49) GriffenJBS: I know inferno uses channels, so they don't often have threading issues (am I wrong), that's a common error in windows and some linux apps
(20:11:32) mjl-: you can still create threading issues easily :)
(20:11:41) mjl-: but it's a lot easier with channels
(20:11:41) GriffenJBS: linux isn't less prone to those errors, they are just usually reviewed by more eyes and smarter users, so the problems get fixed
(20:12:03) GriffenJBS: no threading in node = no threading issues
(20:12:17) mjl-: channels give a somewhat higher level view of concurrency
(20:12:25) mjl-: i'm wondering about node.js about that
(20:12:33) mjl-: how do you write a server where you do need state?
(20:12:44) GriffenJBS: the state is in local objects, not global
(20:13:13) GriffenJBS: one local state is the listening socket, it passes handles to the other objects (via channels, so no global vars)
(20:13:38) GriffenJBS: and no blocking operations so no dead locking
(20:13:58) GriffenJBS: little strange to start with, but I've been really impressed with the results
(20:14:43) GriffenJBS: I still think mistakes make better students, no one learns from success
(20:14:50) mjl-: well, two node.js apps both reading on eachothers sockets, that sounds like a deadlock
(20:15:10) GriffenJBS: :-) the reading isn't blocking
(20:15:36) GriffenJBS: if there is no data, then there is nothing to do
(20:15:40) mjl-: no, but the processes don't make any progress.  that's a deadlock
(20:16:05) mjl-: the protocol between the two can be implemented wrongly.  that's the same with "traditional" threaded programs
(20:16:11) GriffenJBS: no it's a design error, that's like saying my auto-compete doesn't work before you type a letter
(20:17:09) GriffenJBS: the difference is two dead locked threads will never release, here there are no threads, nothing's blocked, it's idle
(20:17:13) mjl-: threading errors are pretty much design errors too then.  but probably easier te omake
(20:17:49) GriffenJBS: some threading errors happen with "funny" timing, since node doesn't use threads, those issues don't happen
(20:17:51) mjl-: heh, the state in node.js indicating something is waiting, that's the thread
(20:18:12) GriffenJBS: yes, there is one thread, the main loop
(20:18:42) GriffenJBS: at a basic level it's nothing new, but it pushes new programmers to a higher level and keeps them out of the lower level stuff
(20:18:49) mjl-: so how do you call the context of the thing that is waiting for network data to come in?
(20:18:52) GriffenJBS: like getting rid of pointer arthmatic
(20:19:24) GriffenJBS: it's flaged the OS to be signaled when data is available to a file/socket, or a timeout
(20:19:59) mjl-: but where is the stack?  with the functions that have been called?
(20:20:28) GriffenJBS: they didn't invent anything new, they used old ideas, with a common language and put it in one nice, neat, fast package (using v8)
(20:20:29) mjl-: normally, a "thread" is pretty much that call stack
(20:20:43) GriffenJBS: :-) the stack almost doesn't exist
(20:21:16) GriffenJBS: I think the deepest I've ever seen it is 6, that's only when you've foobard something and it throw an exception
(20:21:44) GriffenJBS: that's a problem they're working on now, is createing stack traces to help people with debugging
(20:22:42) GriffenJBS: when the system idles, the stack is empty
(20:24:19) GriffenJBS: btw you don't know of an 'learn limbo in 7 days' type site do you?
(20:24:41) GriffenJBS: the closest I've found has been at cat-v.org
(20:25:18) mjl-: nopes, i don't think there is such a site
(20:25:25) mjl-: i did once start writing some docs
(20:25:33) mjl-: but well...
(20:26:17) mjl-: i should test the limits of inferno & threads
(20:26:30) mjl-: e.g. run a high load against a network service
(20:26:36) GriffenJBS: does select(), or something like it exist in inferno?
(20:27:52) GriffenJBS: sorry, in limbo
(20:28:01) mjl-: no
(20:28:03) mjl-: there is alt
(20:28:13) mjl-: which is like a select on channels
(20:28:54) mjl-: if you want to do i/o on multiple fd's at a time in a single program, you normally spawn a thread, one for each fd you want to do i/o on (because it can block)
(20:29:24) mjl-: when the i/o operation completes, you send info about the action (e.g. data read, or error message or successful write) on a channel
(20:29:41) mjl-: then you can have a single thread that alt's on all channels that may be active
(20:29:50) GriffenJBS: so no non-blocking io in limbo
(20:29:51) mjl-: and handles the case
(20:29:54) mjl-: no
(20:30:59) mjl-: the inferno implementation (of the virtual machine) also does not use select/poll.  it creates a real OS processes (or thread in some cases, but it depends on the OS) if it has to do blocking i/o for a limbo thread
(20:32:10) GriffenJBS: ..? does inferno support select or poll?
(20:32:14) mjl-: channels are just within the virtual machine.  so if you send/receive from lots of processes, that's done with 1 OS process/thread.  but if you have many thread reading/writing to a network file descriptor, that will result in many OS processes/threads (assuming you're using emu)
(20:32:27) mjl-: no, no select or poll in inferno
(20:33:21) GriffenJBS: huh
(20:36:47) GriffenJBS: umm ... /os/port/qio.c
(20:43:37) vsrinivas: adding NBIO would be easy.
(20:44:20) vsrinivas: i
(20:44:59) vsrinivas: *i've felt the need for either select() or an async io interface or even vectored io.
(20:45:22) mjl-: vectored io i could probably put to good use
(20:45:29) mjl-: haven't had a need for select yet
(20:45:36) vsrinivas: its really silly in p9 and hosted inferno to spend so much on host procs for them to spend 90% of their time blocking.
(20:45:51) mjl-: well, that's what they are for :)
(20:46:02) mjl-: or do you mean it's a waste of memory?
(20:46:25) vsrinivas: waste of all involved! you can efficiently block without a PCB, kstack, user stacks, etc... :)
(20:46:41) mjl-: yeah, good point
(20:47:04) mjl-: it's still useful to be able to use host os procs for some calls
(20:47:13) vsrinivas: right, many things are fine.
(20:47:18) GriffenJBS: ..? vectored io, I don't know that term
(20:47:26) vsrinivas: readv/writev.
(20:47:37) mjl-: i don't have a list handy, but i'm sure there are syscalls that can block that you can't get events on with kqueue/epoll
(20:47:47) mjl-: like plenty of ioctls
(20:47:51) vsrinivas: i'd most prefer something like async_read/_write/... ; w/ async_collect().
(20:48:11) mjl-: but optimizing the typical read/write into epoll/kqueue will probably give 95% of the gain
(20:48:17) vsrinivas: all the async calls would return descriptors.
(20:48:31) vsrinivas: and then _collect would wait for the named set of descriptors to finish.
(20:48:36) vsrinivas: (or perhaps for at least one.)
(20:48:41) mjl-: vsrinivas: you want that in limbo?
(20:49:00) vsrinivas: on p9, in limbo (preferrably w/o a host thread per io request)
(20:49:15) vsrinivas: pretty much wherever :)
(20:49:42) mjl-: in limbo, you could implement those with an adt, channels and progs.  as long as you can make the underlying os reads/writes go into epoll/kqueue
(20:50:00) vsrinivas: yea. that'd be a reasonable start.
(20:50:32) mjl-: the dis thread overhead is really small
(20:50:34) Fish- [~Fish@9fans.fr] entered the room.
(20:50:47) vsrinivas: yes, the dis thread overhead isn't what bothers me when doing this in inferno.
(20:51:17) vsrinivas: its the release()/acquire(), plus the blocked host kproc. ; which there are host ways to do better.
(20:51:31) vsrinivas: ((i'm currently writing a fileserver on p9 and i dislike having so many Ioprocs around....)
(20:52:12) vsrinivas: anyway, </rant> :)
(21:06:10) Fish- left the room (quit: Quit: So Long, and Thanks for All the Fish).
(21:06:19) Fish- [~Fish@9fans.fr] entered the room.
(22:50:19) vsrinivas left the room (quit: Ping timeout: 252 seconds).
(22:50:44) GriffenJBS: so has anyone figured out how to map the delete key to something useful?
(22:50:56) vsrinivas [~me@c-68-48-11-23.hsd1.md.comcast.net] entered the room.
(22:55:41) vsrinivas: mjl-: any thoughts wrt better io interfaces in limbo?
(23:11:05) mjl-: vsrinivas: not really
(23:11:12) mjl-: i've found myself writing preadn a few times
(23:11:25) mjl-: and i've wished for vectored i/o a few times
(23:11:33) mjl-: especially for udp
(23:11:45) mjl-: where you _must_ do a single write
(23:12:15) mjl-: which is annoying (but not too hard) with multiple layers of protocol
(23:12:21) vsrinivas: ok.
(23:12:34) mjl-: perhaps i have to give it more thought :)
(23:12:37) vsrinivas: hrm, lets see if we can't figure out a good vectored io interface :)
(23:17:53) vsrinivas: unix has {read,write}v(int fd, char *buf[], int len[], int offset[], int niovs);
(23:18:06) vsrinivas: (approximately; they don't actually have offset, but w/e).
(23:19:46) vsrinivas: would those be close enough to what you wanted?
(23:20:32) mjl-: yes
(23:21:12) mjl-: there could be an iov array, each elem with data, len, offset
(23:21:34) vsrinivas: yea; and a source or dst offset to write to. (i was just expanding the unix one out)
(23:22:05) mjl-: yeah
(23:22:10) mjl-: having an offset is good
(23:22:17) mjl-: should reads work as readn?
(23:22:50) mjl-: without, calling readv for the remaining data is a bit annoying
(23:23:09) vsrinivas: agreed. hrm. is there any harm in that.... 
(23:24:06) mjl-: not sure.  i think you normally use readn if you know the input has a certain layout
(23:24:11) mjl-: i.e. nested protocols
(23:24:37) mjl-: otherwise you could as well use normal read (unless there is something with limited chunks of buffers)
(23:24:40) vsrinivas: also as a convenience; but readn() could be a bad idea with UDP fds, for example.
(23:24:55) mjl-: hmm, yeah
(23:25:20) mjl-: yes :)
(23:25:35) vsrinivas: so we could add a library readvn/writevn. or implement both readv/writev and readvn/writevn in the system...
(23:25:52) mjl-: have to check, but i don't think readv could know about udp and that a short read is just 1 packet
(23:25:59) mjl-: yeah
(23:26:13) mjl-: readvn sounds like it should be for later :)
(23:26:26) vsrinivas: yea, for now we can delay :)
(23:27:23) vsrinivas: ((hrm, need to merge inferno-os's changes into -npe again))
(23:27:38) vsrinivas: anyway. hrmmm -- if short returns are okay....
(23:27:52) vsrinivas: we could actually do readv(int fd[]) ... as well
(23:28:09) vsrinivas: for a merged form of select and readv....
(23:28:24) vsrinivas: (allow it to return if at least one readout.
(23:28:49) mjl-: hah
(23:28:53) mjl-: with a single buffer
(23:28:53) mjl-: ?
(23:29:02) mjl-: or one (set of) buffer(s) for each fd?
(23:29:03) vsrinivas: no, the full call as above.
(23:29:11) vsrinivas: one buffer for each fd is enough i think :)
(23:29:17) mjl-: yeah
(23:29:21) vsrinivas: you can fill in the fd array w/ the same fd if you want.
(23:29:22) mjl-: and it returns the buffer?
(23:29:41) vsrinivas: fills it in, just as in readv, yea :)
(23:30:24) mjl-: check
(23:30:38) mjl-: let's see, what about a version with and without offset?
(23:30:46) mjl-: plan 9 has dummies, preadv and readv
(23:30:58) mjl-: struct IOchunk
(23:30:58) mjl-: {
(23:30:58) mjl-:  void *addr;
(23:30:58) mjl-:  ulong len;
(23:30:58) mjl-: } IOchunk;
(23:31:06) mjl-: yuck, i bet that doesn't look good
(23:31:12) vsrinivas: why would you want one w/o offset?
(23:31:26) vsrinivas: i didn't know p9 had them , interesting.
(23:31:37) mjl-: i don't want them, i think
(23:31:49) mjl-: but for readv from network, what to fill in as offset?
(23:31:52) mjl-: 0 i suppose
(23:31:58) mjl-: it is ignored anyway
(23:32:02) vsrinivas: 0 is what you do w/ pread() atm.
(23:32:48) mjl-: but just preadv & pwritev would be enough
(23:33:17) mjl-: we could just have an array with those iov's.  IOchunk looks a bit ugly to me
(23:34:37) mjl-: long pwritev(int, IOchunk*, int, vlong);
(23:34:47) mjl-: long preadv(int, IOchunk*, int, vlong);
(23:34:58) mjl-: we don't need the third arg
(23:35:07) mjl-: and the IOchunk doesn't necessarily need a "len"
(23:35:11) mjl-: you can slice the array
(23:35:18) mjl-: don't know if that's wise
(23:35:54) mjl-: writev(fd, array[] of {data1[:10
(23:36:03) mjl-: ai, wasn't finished yet
(23:36:30) mjl-: pwritev(fd, array[] of {data1[:10], data2[:20]}, big 0);
(23:36:37) mjl-: but then perhaps with multiple fd's
(23:36:40) mjl-: well, for read :)
(23:36:48) mjl-: not for writev i think
(23:36:48) mjl-: ?
(23:38:24) vsrinivas: dunno about slicing... pread and pwrite do take a len atm...
(23:38:46) mjl-: yeah, that's right
(23:39:16) vsrinivas: oh .... this is going to be a 'fun' merge.
(23:39:17) mjl-: i was trying to avoid having to create an adt for the buffer :)
(23:39:27) mjl-: ☺
(23:39:43) mjl-: array[] of {(data1, 10), (data2, 10)}
(23:39:46) mjl-: or something like
(23:40:01) mjl-: array[] of {IOchunk(data1, 10), IOchunk(data2, 10)}
(23:42:47) vsrinivas: mjl-: why the iochunk adt at all?
(23:42:50) vsrinivas: preadv(fd: array of ref FD, bufs: array of array of byte, lens: array of int, off: array of big): int;
(23:42:57) vsrinivas: pwritev(fd: ref FD, bufs: array of array of byte, lens: array of int, off: array of big): int;
(23:43:54) mjl-: why have multiple offsets?
(23:44:30) vsrinivas: the idea was that was source offset; i should add dst offsets there.
(23:44:45) vsrinivas: (so you can read from all over a file)
(23:45:30) vsrinivas: that's probably not a good idea.
(23:45:46) vsrinivas: for read i can see the argument. for write, meh.
(23:45:46) mjl-: i'm wondering how common that is
(23:45:52) mjl-: yeah
(23:46:21) mjl-: for read, it may be annoying to modify three arrays for short reads
(23:46:39) mjl-: though it's probably manageable :)
(23:46:48) GriffenJBS: iirc the idea is to reduce copying, does a dis slice return a reference to the parent with new ofs, len, or a new array?
(23:48:03) GriffenJBS: should have thrown a ; in there for good measure
(23:49:00) mjl-: the slice references the same data, just potentially starting at different place and with different length
(23:49:05) mjl-: so it's pretty cheap
(23:49:28) mjl-: the original buffer is only freed when both references go away
(23:49:30) GriffenJBS: good, so passing an array of slices is effective
(23:50:06) mjl-: hmm, now i'm wondering if sliced strings are copied.  i have a vague feeling they are
(23:50:25) mjl-: yeah, an array of slices would not be ineffecient i think
(23:50:49) GriffenJBS: if strings are immutable, what would be the point?
(23:52:15) mjl-: strings are not immutable
(23:52:19) mjl-: you can append to them
(23:52:22) mjl-: s := "";
(23:52:25) mjl-: s[len s] = 'a'
(23:52:28) mjl-: now s contains "a"
(23:52:37) mjl-: you can only append 1 char at the time :)
(23:52:40) mjl-: it grows automatically
(23:53:01) mjl-: there may be operations where the string is copied
(23:53:10) mjl-: i tend not to rely on such behaviour because i can't remember the details :)
(23:53:43) vsrinivas: http://9fans.net/archive/2003/02/801 aha
(23:55:48) vsrinivas: okay -- one main weakness of unix's select() is that arrays need to be copied in/out of the kernel all the time; if slices are just refs, then this'll be better.
(00:00:40) vsrinivas: mjl-: hmm, like the IOchunk adt better? or the explicit arrays?
(00:03:35) mjl-: not sure yet
(00:03:48) mjl-: i think an anonymous tuple would be better than an IOchunk
(00:03:53) mjl-: that looks too bulky
(00:04:12) vsrinivas: okay. agreed, but never seen that interfaced to C code; is that possible?
(00:04:14) mjl-: but i don't know about 1 array with tuples vs multiple arrays
(00:04:29) mjl-: good point
(00:04:37) vsrinivas: (tuples would be nice though!)
(00:05:44) GriffenJBS: mjl-: wait s := ""; t := s; s[len s] = 's';
(00:05:46) GriffenJBS: what's t?
(00:06:00) mjl-: GriffenJBS: i think "s"
(00:08:06) GriffenJBS: :-) nope, t = "", just checked in with limbo/dis
(00:08:27) GriffenJBS: so strings are immutable
(00:09:05) mjl-: good to know :)
(00:09:10) mjl-: i should've known i guess
(00:09:25) mjl-: vsrinivas: limbo -a does not have a problem with an array of tuples
(00:09:31) mjl-: it makes it into an Array
(00:09:33) vsrinivas: neat.
(00:10:00) mjl-: not exactly sure that it expects the elems of that array to be
(00:14:25) GriffenJBS: it's not possible to catch Ctrl+C is it?
(00:15:06) GriffenJBS: I don't think I've seen any signals in inferno
(00:15:16) vsrinivas: no, its not;
(00:15:41) mjl-: there are no signals :)
(00:16:07) mjl-: at most there are exceptions that the OS can trigger, e.g. on write to orphaned pipe
(00:16:23) mjl-: or was it a read from an orphaned pipe
(00:16:34) mjl-: anyway, still feels a bit wrong
(00:17:08) mjl-: anyway, you can easily make a tk interface that catches eg Del and kills the program it is running.  should be possible to do for wm/sh
(00:17:11) mjl-: i've done it with wm/run
(00:17:41) vsrinivas: for now i'll use explicit arrays; i know how to read/write those from native code. :)
(00:17:42) mjl-: though processes can easily isolate themselves more than eg wm/sh can kill
(00:17:58) mjl-: vsrinivas: ok ☺
(00:22:51) GriffenJBS: is there an inferno wiki somewhere? I can read up and maybe add some of what I'm learning
(00:27:31) vsrinivas: don't think so; there is the space on google code....
(00:27:32) mjl-: there used to be one, but i think it is offline now
(00:33:39) GriffenJBS: :-/ makes it rather hard to get people to help
(00:33:42) mjl-: yeah
(00:33:49) mjl-: having a wiki would be good
(00:34:19) GriffenJBS: I have quiet a bit of IT experience and I'm finding this rather obscure
(00:34:35) vsrinivas: caerwyn used to run one; his was run on inferno itself.
(00:34:59) vsrinivas: i think it'd be alright for us to run a unix wiki program though. :)
(00:35:28) vsrinivas: we could use the google code one; or we could set one up....
(00:35:51) Fish- left the room (quit: Quit: So Long, and Thanks for All the Fish).
(00:37:55) mjl-: can we easily give anyone access to write to the inferno-npe wiki?
(00:38:36) vsrinivas: i think so...
(00:40:30) vsrinivas: oh hrm,,,,
(00:40:35) vsrinivas: "Only project owners and committers can add and edit wiki pages for a particular project. "
(00:43:43) GriffenJBS: what about google sites?
(00:44:17) vsrinivas: does it have a wiki?
(00:45:05) GriffenJBS: I know nothing of it, but it claims to
(00:45:17) GriffenJBS: only 10Gb of space
(00:45:55) GriffenJBS: :-) no matter how big storage gets porn can always fill it
(01:20:45) mjl-: the wiki functionality of google sites isn't very obvious to me
(01:20:53) mjl-: perhaps you have to login to see more about what the service is
(01:21:13) mjl-: if i can find how caerwyns wiki worked, i might be able to set one up
(02:11:29) btdn left the room (quit: Ping timeout: 240 seconds).
(02:11:57) btdn [~btdn@138.74.188.207] entered the room.
(03:06:06) GriffenJBS: mjl-: how is this for a start? https://sites.google.com/site/496e6665726e6f4f53/
(03:17:48) bugQ [~bug@c-67-171-127-76.hsd1.ut.comcast.net] entered the room.
(03:27:30) vsrinivas: GriffenJBS: need google account?
(03:36:50) GriffenJBS: yeah, keeps anon users from trashing site, google rule, not mine
(03:37:14) GriffenJBS: anyone can browse, account needed to edit
(04:25:37) KillerX left the room (quit: Quit: KillerX).
(04:50:52) bugQ left the room (quit: Ping timeout: 250 seconds).
(06:17:19) tensai_cirno left the room (quit: Ping timeout: 252 seconds).
(06:27:45) GriffenJBS left the room (quit: Read error: Operation timed out).
(06:31:15) GriffenJBS [~john@adsl-233-213-109.mia.bellsouth.net] entered the room.
(07:17:21) base2design [~base2desi@24-178-120-47.static.nwnn.ga.charter.com] entered the room.
(07:32:54) base2design left the room (quit: Remote host closed the connection).
(07:56:57) base2design [~base2desi@24-178-120-47.static.nwnn.ga.charter.com] entered the room.
(10:12:22) robot12 [~kazzhilki@proxy10.ts.fujitsu.com] entered the room.
(10:32:40) mjl-: morning
(10:33:05) mjl-: GriffenJBS: nice
(10:45:16) perdix [~mkhl@sxemacs/devel/perdix] entered the room.
(10:46:39) perdiy left the room (quit: Ping timeout: 246 seconds).
(12:14:41) arvindht [7c7cdbfa@gateway/web/freenode/ip.124.124.219.250] entered the room.
(12:27:00) mjl-: GriffenJBS: why is the name so obscure? :)
(12:27:08) mjl-: the hexadecimal in the url i mean
(12:29:00) mjl-: hmpf, something is weird.  if i click "sign in" i get an error from "google accounts" saying "the page you requested is invalid"
(12:29:37) mjl-: temporary failure it seems, works now
(12:29:48) mjl-: no, again
(12:33:22) mjl-: same with IE on win xp, i guess they really broke something.  or it is not available for edit by other uesrs (but then the reror message is very unclear)
(13:05:55) The account has disconnected and you are no longer in this chat. You will be automatically rejoined in the chat when the account reconnects.