Conversation with #inferno at Wed Dec  8 12:24:42 2010 on powerman-asdf@irc.freenode.net (irc)
(14:12:20) hotaru2k3 left the room (quit: Ping timeout: 245 seconds).
(15:05:13) jumzi left the room (quit: Read error: Operation timed out).
(15:49:57) f|NN [~f|NN@p4FE8825C.dip0.t-ipconnect.de] entered the room.
(16:39:41) bvalek2 left the room (quit: Quit: Page closed).
(16:55:27) Gegemon left the room (quit: Quit: Leaving.).
(18:48:53) Fish [~Fish@coss6.exosec.net] entered the room.
(19:51:24) anth [none@cpe-76-190-129-110.neo.res.rr.com] entered the room.
(20:25:02) bvalek2 [5985141d@gateway/web/freenode/ip.89.133.20.29] entered the room.
(20:56:09) powerman: is it possible to somehow find which one of several channels ready to receive data?
(20:56:44) powerman: i mean, not in case with fixed amount of channels
(20:56:51) mjl-: an array
(20:57:08) powerman: is it possible to write to array of chan?
(20:57:11) mjl-: a := array[10] of {* => chan of int};  and then alt
(20:57:19) mjl-: not sure
(20:57:21) powerman: different data for each channel?
(20:57:34) mjl-: i don't think so, but perhaps
(20:57:43) powerman: as far as I see in docs, array of chan is special case for "<-" operator
(20:58:45) mjl-: yeah
(20:59:14) mjl-: so, you probably cannot write to an array of chan
(20:59:40) powerman: anyway. I've FIFO queue of data to send, and have to distribute this data to several peers. so I run one thread per each peer, which should receive "next elem from FIFO queue" someone (using chan to main thread), then block in sys->write() of this elem to peer, then repeat
(21:00:20) powerman: of course, speed of peers is different, and one peer may receive 5 elem while another just 1
(21:00:55) powerman: so, how to better implement this sort of queues in limbo?
(21:01:09) mjl-: and i suppose there can be a variable number of peers
(21:01:13) powerman: yep
(21:01:35) powerman: looks like only native FIFO queue in limbo is channels itself
(21:02:29) powerman: but I've high enough data rate, and I afraid creating chan[100000] of string per each peer isn't good enough idea
(21:02:34) mjl-: neh :)
(21:03:12) mjl-: the data that has to be sent to the peers, is there any push back mechanism?  so that the main thread doesn't receive 100000 things to send to a peer...
(21:05:20) powerman: no, I don't think so. this is sort of pipe, data received by main thread 24x7, and peers are doing sort of analyse for that data. some peers on some data may be slow, in this case they should be disconnected and re-connected again (so they lose some part of data but continue analyse from current data item)
(21:05:59) mjl-: oh
(21:06:11) mjl-: couldn't you let the peers send a channel to the main thread when they are ready to receive data?
(21:06:32) powerman: of course, I can add "extra" push back mechanism - each thread after successful sys->write() to peer can send "1" to main thread by channel and then block reading on another channel from main thread
(21:06:56) mjl-: then the main thread can do for each peer:  keep track of data to send,  and a response channel from the peer to send it on.  then when data or a response channel comes in, the main thread can send the data on the response channel
(21:06:56) powerman: but this looks overcomplicated. and probably add extra latency.
(21:07:18) mjl-: i think that is how i usually do it
(21:12:21) powerman: hm
(21:12:37) powerman: chan[100000] of string looks like simpler
(21:13:14) powerman: in most cases, peers will be fast enough to process data with same speed as this data received from pipe by main thread
(21:13:52) powerman: so, in most cases these chans will use int*100000=~0.5MB per peer of memory to keep empty pointers to string
(21:14:18) powerman: with 5-10 peers that's not a problem
(21:15:50) powerman: and in this case main thread can just do "for(i ...) alt{ c[i]<-=data[i] => ; * => kill_overloaded_chan(i) }"
(21:32:31) hotaru2k3 [debian-tor@gateway/tor-sasl/hotaru2k3] entered the room.
(21:34:13) powerman: btw
(21:35:30) powerman: looks like easiest way to implement local FIFO queue in single thread is create buffered channel which will be read and written only by that thread :)
(21:36:24) powerman: and use alt{} with "*" case while both reading and writing to detect queue overload and empty queue :)
(22:08:50) f|NN left the room (quit: Quit: leaving).
(22:13:48) f|NN [~f|NN@p4FE8825C.dip0.t-ipconnect.de] entered the room.
(00:27:33) bvalek2 left the room (quit: Quit: Page closed).
(02:28:12) f|NN left the room (quit: Quit: leaving).
(03:09:28) hotaru2k3 left the room (quit: Remote host closed the connection).
(03:10:19) hotaru2k3 [debian-tor@gateway/tor-sasl/hotaru2k3] entered the room.
(07:24:38) anth: took another look at my json project today. turns out the other end's more involved than i realized.
(07:25:07) anth: in my stuff, though, there's something i don't get: the json has two garbage characters prepended.
(07:25:19) anth: for example, i get:
(07:25:35) anth: ��{"device-token":"e87ffd7751... (and then correct json)
(08:59:56) anth: Hadn't correctly set up ssl.
(10:00:35) Gegemon [~ynv@mx1.airis.ru] entered the room.
(10:50:20) bvalek2 [c3e41404@gateway/web/freenode/ip.195.228.20.4] entered the room.
(13:36:22) Fish left the room (quit: Quit: So Long, and Thanks for All the Fish).
(15:01:36) Fish [~Fish@coss6.exosec.net] entered the room.
(15:31:31) powerman: is inferno expected to operate well after "out of memory: heap" errors?
(15:34:23) powerman: I've some thread which read events added/changed/deleted with some data chunks in json from external source and keep current data state in memory. So, if external source provide too many "added" and too few "deleted" events, this thread may use a lot of memory, and that's correct behaviour. I've set "heap" poll to 96MB, but sometimes this isn't enough and I still got "out of memory" exception.
(15:37:07) powerman: After that exception this thread dies (and so frees all used memory), then manager thread start this thread again, and I expect everything to continue working in usual way. But this isn't the case - looks like everything stopped after "out of memory" error. Only working process in this emu left is rstyx, so I'm able to do some analyse now…
(15:39:03) powerman: btw, emu output this line: arena heap too large: "size 15238656 cursize 68867104 arenasize 98049920 maxsize 100663296" - it's pretty clear what "maxsize" is (my 96MB heap poll), and I suppose "arenasize" is amount of currently used memory in that poll, but can someone explain to me what "size" and "cursize" values mean?
(15:42:50) powerman: ; cat '#c/memory'
(15:42:50) powerman: 1515712 33554432 1568736 294497783 294491615 4 32038708 main
(15:42:50) powerman: 4968064 100663296 84769376 641163149 641127445 56 95695220 heap
(15:42:50) powerman: 0 33554688 0 0 0 0 33554676 image
(16:11:30) powerman: ok, looks like inferno itself in general is ok. issue was in hang Cat process reading from thread serving file2chan which died because of "out of memory", after killing it normal operation of this emu was restored.
(16:12:36) powerman: but… isn't that Cat should got EOF from '#s' when that thread was killed?
(16:13:58) powerman: looks like it's time to write some small test… and pray this isn't race condition in emu again
(16:45:28) f|NN [~f|NN@p4FE88269.dip0.t-ipconnect.de] entered the room.
(16:55:09) Gegemon left the room (quit: Quit: Leaving.).
(17:00:35) powerman: so, looks like if thread accept file2chan's read request and exit without replying, other side will hang in sys->read() because '#s' doesn't detect thread exit and doesn't send Rread
(17:05:11) powerman: actually, '#s' do detect thread exit (and remove files created by that thread) but doesn't send pending Rread
(17:05:24) powerman: so, this looks like a bug in '#s'
(17:13:19) bvalek2 left the room (quit: Quit: Page closed).
(17:20:53) powerman: reported http://code.google.com/p/inferno-os/issues/detail?id=244 let's see what Charles will reply…
(18:12:24) f|NN left the room (quit: Quit: leaving).
(19:36:23) Fish left the room (quit: Quit: So Long, and Thanks for All the Fish).
(20:01:51) bvalek2 [5985141d@gateway/web/freenode/ip.89.133.20.29] entered the room.
(20:56:12) bvalek2 left the room (quit: Quit: Page closed).
(21:29:24) The account has disconnected and you are no longer in this chat. You will be automatically rejoined in the chat when the account reconnects.