Conversation with #inferno at Sun Apr 24 19:12:21 2011 on powerman-asdf@irc.freenode.net (irc)
(19:13:06) Fish- [~Fish@9fans.fr] entered the room.
(19:13:57) powerman-asdf1 left the room (quit: Ping timeout: 250 seconds).
(00:56:33) Fish- left the room (quit: Quit: So Long, and Thanks for All the Fish).
(02:26:36) ZNC [patty@unaffiliated/yutaka] entered the room.
(02:27:09) ZNC left the room ("Part").
(03:34:41) hotaru2k3 [debian-tor@gateway/tor-sasl/hotaru2k3] entered the room.
(04:20:17) dreadlorde [~dreadlord@c-24-11-39-160.hsd1.mi.comcast.net] entered the room.
(05:17:28) hotaru2k3 left the room (quit: Quit: leaving).
(06:13:13) dreadlorde left the room (quit: Ping timeout: 252 seconds).
(08:33:59) josephholsten [~josephhol@ip70-185-215-221.ok.ok.cox.net] entered the room.
(09:29:53) robot12 [~kazzhilki@proxy10.ts.fujitsu.com] entered the room.
(09:50:04) bvalek2 [5e152df5@gateway/web/freenode/ip.94.21.45.245] entered the room.
(10:23:26) josephholsten left the room (quit: Quit: josephholsten).
(10:33:06) Fish- [~Fish@9fans.fr] entered the room.
(11:21:58) powerman: if anyone interested… https://code.google.com/p/inferno-cjson/ http://powerman.name/Inferno/man/2/cjson.html
(11:39:21) powerman: it's about 5 times faster than json(2). pure limbo module with same interface is about 3 times faster than json(2).
(11:40:29) powerman: btw, while doing a lot of benchmarking in last days, and comparing same code both in Limbo and C, I was really surprised how fast Limbo (with JIT) work! rewriting in C usually gives at most 1.5 times speedup.
(12:39:20) bvalek2 left the room (quit: Quit: Page closed).
(13:52:53) base2design left the room (quit: Remote host closed the connection).
(14:10:45) base2design [~base2desi@68-190-40-142.dhcp.athn.ga.charter.com] entered the room.
(15:07:16) powerman: mjl-: is there exists top-like tool for inferno? wm/task doesn't show cpu usage
(15:22:06) powerman: in theory, /prog/*/status contain CPU time field, and with manual diff every n seconds we can find out cpu usage by process. but in practice this field always contain 0:00.0 ☹
(16:01:47) powerman: mjl-: do you've vim syntax highlight config for /dis/sh files?
(16:18:25) vpm left the room (quit: Quit: Take me to your leader!).
(16:19:39) vpm [~vpm@reverse-94.fdn.fr] entered the room.
(17:35:00) josephholsten [~josephhol@216.16.128.242] entered the room.
(18:03:18) robot12 left the room (quit: Quit: Leaving.).
(18:12:55) josephholsten left the room (quit: Read error: Connection reset by peer).
(18:13:11) josephholsten [~josephhol@216.16.128.242] entered the room.
(18:14:39) josephholsten left the room.
(18:16:18) btdn [~btdn@138.74.188.207] entered the room.
(18:28:23) anth_x: powerman-asdf: iirc, cpu usage is 0 if you're running jit, but has content if you're interpreted.
(18:28:28) anth_x: that could be outdated info, though.
(18:29:12) anth_x: historically, we've always said jit'd dis resulted in about a 2x performance hit. it's nice to hear that's still roughly true, maybe better.
(18:30:12) powerman: no, it doesn't work both with and without jit
(18:30:18) powerman: at least on linux
(18:36:55) josephholsten [~josephhol@216.16.128.242] entered the room.
(18:37:46) josephholsten left the room.
(20:15:11) jibun left the room (quit: Read error: Operation timed out).
(20:34:17) vpm left the room (quit: Quit: Lost terminal).
(20:36:25) vpm [~vpm@reverse-94.fdn.fr] entered the room.
(23:43:01) jibun [~jibun@dsl-trebrasgw1-fff0c300-37.dhcp.inet.fi] entered the room.
(00:28:06) Fish- left the room (quit: Quit: So Long, and Thanks for All the Fish).
(02:43:08) hotaru2k3 [debian-tor@gateway/tor-sasl/hotaru2k3] entered the room.
(05:00:16) hotaru2k3 left the room (quit: Quit: leaving).
(05:11:41) jibun left the room (quit: Ping timeout: 240 seconds).
(06:26:23) powerman: i'd like to simplify file2chan's server by ignoring fids because there are should be only one reader from that file. it's possible to set DMEXCL on file2chan's file, and thus guarantee two readers doesn't open this file simultaneously by occasion.
(06:26:32) powerman: but there are race condition between file2chan() and wstat() calls - two clients may open file2chan()'s file between these calls. is it possible to avoid it somehow?
(06:32:16) powerman: probably setting DMEXCL should detach all current clients but one
(06:33:16) powerman: or there are should exists (umask-like?) way to provide initial permissions to file2chan()
(08:44:35) robot12 [~kazzhilki@proxy10.ts.fujitsu.com] entered the room.
(10:01:34) mjl-: powerman-asdf: i'll have a look at your json code later.  good to hear you get a speed improvement :)
(10:07:12) mjl-: hopefully i'll even get around to running a few benchmarks
(10:07:32) mjl-: powerman-asdf: there's no top-like afaik, also no useful cpu information
(10:07:48) powerman: i had to implement custom ugly replacement for patch tool :(
(10:08:10) mjl-: i would like _some_ information about threads...  perhaps number of context switchs.  a program that sorts programs by state (ready first) could help already
(10:08:30) powerman: because standard patch won't support patching same mkfiles in same places for adding several libinterp/ modules
(10:08:34) mjl-: yeah, the path tool is for modifying libinterp/mkfile right?
(10:09:09) powerman: http://code.google.com/p/inferno-cjson/source/browse/patch.cjson
(10:09:48) powerman: looks like we've to include something like that with each C module ☹
(10:10:43) powerman: ideally, charles should rework somehow structure of files to allow adding C modules at least as simple and with /opt. but we can wait until this happens several years
(10:16:51) mjl-: :)
(10:17:00) mjl-: must build it ourselves :)
(10:17:08) mjl-: loadable modules would be pretty neat
(10:17:09) mjl-: at runtime
(10:17:16) mjl-: but that's yet another step
(10:19:14) powerman: loadable C modules from /opt solve issue with both ugly patching libinterp/mkfile etc and rebuilding emu each time. and made developing them much ease. so, if you ask me, then this is only right way to go, if there are no technical limitations against loading dynamic code into inferno kernel
(10:23:45) powerman: Limbo code maps to C nearly 1-to-1. I havn't checked yet how to spawn processes and communicate with channels in C, but everything else is really easy, and that was a surprise for me. C which I remember from my college time was much worse. I even prefer to use Asm for my diploma project than C. ☺
(13:02:53) powerman: http://pastebin.ca/2050809 that looks overcomplicated, but looks like simpler implementations to guarantee there only one reader for our file2chan (and so we can safely ignore fid management) are just buggy
(13:08:58) powerman: so, to avoid this complexity, I'd prefer to use DMEXCL + renaming file to avoid races: http://pastebin.ca/2050815
(13:09:40) powerman: this should be good enough protection against running occasional second reader
(15:58:00) jibun [~jibun@dsl-trebrasgw1-fff0c300-37.dhcp.inet.fi] entered the room.
(18:23:57) robot12 left the room (quit: Quit: Leaving.).
(18:36:58) josephholsten [~josephhol@216.16.128.242] entered the room.
(18:38:47) josephholsten left the room.
(19:40:19) powerman: imagine two services: A and B. B require A for work, so A should be started first, B next. easiest way to guarantee this - A should initialize (create file2chan, for example), and then exit, leaving thread serving that file2chan. this way we can just run in sh: "A; B" and B guaranteed to see A's file2chan when started
(19:40:38) jibun left the room (quit: Ping timeout: 240 seconds).
(19:40:57) powerman: now, we also need to monitor both services, and if any one of them crash - restart emu
(19:42:36) powerman: if there are no dependencies between these services, then they won't need to exit after initialization. and we can monitor them in sh: "{A; shutdown -h}& {B; shutdown -h}&"
(19:43:24) powerman: sadly, but I need both dependency between services and monitoring them for crash ☹
(19:43:49) powerman: anyone have idea how to implement this in simple way?
(19:45:22) powerman: my best idea is embed supervisor inside services and have dependency resolved in first way (service initialize and then exit leaving working thread)
(19:46:37) powerman: but I'd like to see supervisor implemented independently (like runit/daemontools), which able to supervise any service, not only specially developed with supervising support in code
(19:48:28) powerman: if process will print to stdout pid of working thread which it leave running when exiting after initialization, we can do something like: "supervise `{A}; supervise `{B};"
(19:49:43) powerman: that looks like a solution, but there are race condition between printing pid from A and attaching to that pid's /prog/pid/wait in supervise - there are no guarantee another process wont reuse that pid in between
(19:52:29) powerman: and I didn't really like printing pids to stdout from aesthetic view, but that's not really important
(20:02:28) josephholsten_ [~josephhol@216.16.128.242] entered the room.
(20:03:51) josephholsten_ left the room.
(21:12:32) Fish- [~Fish@9fans.fr] entered the room.
(21:45:21) josephholsten [~josephhol@216.16.128.242] entered the room.
(21:45:35) josephholsten left the room.
(22:34:44) jibun [~jibun@dsl-trebrasgw1-fff0c300-37.dhcp.inet.fi] entered the room.
(23:07:12) josephholsten [~josephhol@216.16.128.242] entered the room.
(23:07:55) josephholsten left the room (quit: Read error: Connection reset by peer).
(23:08:06) josephholsten_ [~josephhol@216.16.128.242] entered the room.
(23:08:07) josephholsten_ is now known as josephholsten
(23:09:39) josephholsten left the room.
(23:14:34) KBme left the room (quit: Read error: Operation timed out).
(23:23:38) KBme [~KBme@2001:470:1f13:94::3] entered the room.
(00:20:03) josephholsten [~josephhol@216.16.128.242] entered the room.
(00:22:13) josephholsten left the room.
(00:27:25) Fish- left the room (quit: Quit: So Long, and Thanks for All the Fish).
(00:54:18) KBme left the room (quit: Read error: Operation timed out).
(01:00:38) KBme [~KBme@2001:470:1f13:94::3] entered the room.
(01:49:04) GriffenJBS left the room (quit: Ping timeout: 248 seconds).
(01:51:10) GriffenJBS [~john@adsl-32-34-92.mia.bellsouth.net] entered the room.
(02:15:46) josephholsten [~josephhol@216.16.128.242] entered the room.
(02:20:57) KBme left the room (quit: Read error: Operation timed out).
(02:26:09) KBme [~KBme@2001:470:1f13:94::3] entered the room.
(02:42:28) josephholsten left the room (quit: Read error: Connection reset by peer).
(02:42:51) josephholsten [~josephhol@216.16.128.242] entered the room.
(03:06:19) KBme left the room (quit: Read error: Operation timed out).
(03:08:54) josephholsten left the room (quit: Quit: josephholsten).
(03:22:56) jibun left the room (quit: Ping timeout: 248 seconds).
(03:30:49) KBme [~KBme@2001:470:cabe:666:666:666:666:666] entered the room.
(03:43:33) josephholsten [~josephhol@ip70-185-215-221.ok.ok.cox.net] entered the room.
(03:53:16) josephholsten left the room (quit: Quit: josephholsten).
(05:48:30) powerman: ok, looks like I've figured out how to implement watchdog. usual processes which doesn't put themselves to background can be run as "watchdog someservice". in case we must monitor already started process we can use "watchdog -p pid", so "someservice & watchdog -p $apid" is equivalent for previous command (with very unlikely but unavoidable race)
(05:49:44) powerman: processes which put themselves to background will load Watchdog module (same .dis with another interface) and do "watchdog->me();" or "watchdog->pid(pid);" to add themselves to watchdog
(05:52:42) powerman: that have sense because such processes may consists of several groups of processes, each in own pgrp, and each must be monitored by own watchdog. moreover, they may spawn such groups of processes at any time later. in case such groups of processes spawned later and may legally exit when done their work we can have them to do "watchdog->stop();" to avoid executing watchdog's command when they exit
(05:55:18) powerman: actual command to be executed by watchdog can be defined in environment $watchdog. this allow us to define it once for all services (most used case), but also redefine for some services (replace shutdown -h with service restart command for example). in case service is going to spawn groups of processes later and add them to watchdog at that time it should do FORKENV to protect $watchdog from changes while it running
(06:20:45) base2design left the room (quit: Quit: Tclunk).
(08:42:27) robot12 [~kazzhilki@proxy10.ts.fujitsu.com] entered the room.
(08:55:06) wooster [~bwooster@prolapse.analapocalypse.com] entered the room.
(08:55:24) wooster: ./mkconfig: 33: cannot open /home/bobo/dev/inferno/mkfiles/mkfile-Linux-: No such file
(08:55:25) wooster: removing old libraries and binaries
(08:55:25) wooster: gcc -c -I/home/bobo/dev/inferno/Linux//include -I/home/bobo/dev/inferno/include -I/home/bobo/dev/inferno/utils/include regaux.c regcomp.c regerror.c regexec.c regsub.c rregexec.c rregsub.c
(08:55:25) wooster: regaux.c:1:18: error: lib9.h: No such file or directory
(08:55:25) wooster: In file included from regaux.c:2:
(09:02:11) KBme left the room (quit: Read error: Operation timed out).
(09:06:29) wooster: que es esto
(09:07:08) KBme [~KBme@2001:470:cabe:666:666:666:666:666] entered the room.
(09:48:14) josephholsten [~josephhol@ip70-185-215-221.ok.ok.cox.net] entered the room.
(09:48:40) josephholsten left the room.
(09:50:06) powerman: wooster: check mkconfig
(09:50:30) powerman: probably you forget OBJTYPE=386
(10:09:39) wooster: k
(10:09:58) wooster: no help
(10:10:03) wooster: i have it set to Linux and 386
(10:16:40) powerman: wooster: these errors mean you didn't set OBJTYPE
(10:16:55) mjl-: usually there is still a line OBJTYPE=$objtype after the line setting it to 386
(10:17:12) wooster: OBJTYPE=386         # target system object type (eg, 386, arm, mips, power, s800, sparc)
(10:17:12) wooster: OBJTYPE=$objtype
(10:17:27) powerman: yeah, just comment out second OBJTYPE line
(10:17:33) powerman: it override first value
(10:17:56) wooster: k, thanks
(10:18:01) wooster: least friendly build system of all time
(10:18:33) mjl-: least friendly defaults in mkconfig perhaps
(10:18:45) mjl-: plus, at least it builds fast :)
(10:21:55) mjl-: powerman-asdf: does DMEXCL work for file2chan's?
(10:22:40) powerman: mjl-: yep
(10:22:54) mjl-: powerman-asdf: and, there isn't a pid race in inferno.  iirc, pids aren't even reused.  emu just stops when all pids have been used once.  and they are allocated sequentially
(10:23:17) powerman: "just stops" sounds scary :)
(10:23:49) mjl-: that's what i thought
(10:23:52) mjl-: hasn't happened to me
(10:24:03) mjl-: but a long running instance with lots of progs, it could happen i suppose
(10:24:08) mjl-: see emu/port/dis.c
(10:24:16) mjl-: :/^newprog(
(10:24:27) mjl-: there is a static int pidnum;
(10:24:35) mjl-:         n->pid = ++pidnum;
(10:24:35) mjl-:         if(n->pid <= 0)
(10:24:35) mjl-:                 panic("no pids");
(10:24:43) mjl-: it checks for wrapping
(10:25:29) powerman: i see
(10:26:01) powerman: and n->pid is int
(10:26:33) powerman: ok, panic() is ok for me, I always run emu under supervisor, so it just restarts
(10:27:33) mjl-: :)
(10:28:09) mjl-: i think your watchdog makes sense.  nice that it just needs to check a pid and simpley reboots when triggered
(10:28:55) powerman: mjl-: watchdog already implemented, I've just finished testing it
(10:29:02) powerman: it's clever :)
(10:29:46) powerman: but if I'll create google code project per each single small app like watchdog i'll quickly run out of projects limit
(10:30:59) mjl-: oh, google has a limit for that?
(10:31:02) mjl-: that's silly
(10:31:32) powerman: they afraid… 6 billion people vs just one small google…
(10:32:25) powerman: and 5 billions are malicious spammer, hackers and cookies :)
(10:34:13) powerman: hm. is hg able to clone project's subdir?
(10:35:00) mjl-: hehe
(10:35:06) mjl-: no, hg needs the full repo
(10:35:26) mjl-: well, there is some subrepo extension i think, or partial repo extension, but i don't know if/how it works
(10:35:39) mjl-: you could try bitbucket, i don't think they have limits
(10:42:00) powerman: actually, I probably shouldn't bother about google limits. if i reach it, they either increase it for me after checking i'm not a spammer, or I'll use another google account for new projects, or I'll switch to bitbucket/github/anything. and until that happens it's better to keep everything in same place
(10:44:44) mjl-: yes
(10:51:44) powerman: mjl-: btw, checking for pid isn't really reliable - is someone else open same wait file (i'm not sure, but i'm afraid it even can be another wait file!) and receive pid which watchgod waited for - it doesn't notice pid exit
(10:52:20) powerman: and implement polling for checking existence of directory in /prog/ is ugly
(12:20:24) powerman: /opt/setup.sh at http://code.google.com/p/inferno-os/issues/detail?id=261 (don't really believe charles will accept it, but let's try…)
(12:28:24) pension [~kji@173.242.126.219] entered the room.
(13:16:59) powerman: mjl-: http://code.google.com/p/inferno-contrib-watchdog/
(15:38:25) powerman: mjl-: what you think about bufio(2)?
(15:39:13) powerman: i'm trying to use it, but looks like it's not really suitable for most of my tasks
(15:39:25) powerman: for example, simple reading lines from file
(15:40:02) powerman: gets() is nice, but it doesn't return error, which may happens (bad blocks, file mounted from network, other errors like out of memory)
(15:41:07) powerman: and to check error I need two extra operations - clean werrstr() before calling gets(), and check is sprint("%r") is nil to distinguish between EOF and ERROR when gets() will return nil
(15:41:36) powerman: (and then I need to call sprint(%r) once again to print error to log/stdout
(15:43:05) mjl-: powerman-asdf: yeah, bufio isn't too great
(15:43:11) powerman: another example: for network protocols (and json) it's usually better to work with array of byte, not string. but network protocol often is line-separated, like HTTP (and in most cases json records also separated by \n for ease reading)
(15:43:19) mjl-: i don't thing there is a proper way to find errors from gets
(15:43:50) mjl-: not getting errors is the worst problem
(15:44:14) mjl-: i think most of my use of bufio just treats gets() == nil as eof... it has bitten me already
(15:44:37) mjl-: i thing that sometimes i do a getc() after a gets()==nil, and check for EOF or ERROR
(15:44:40) mjl-: but i'm not sure that's even vali d:)
(15:44:42) mjl-: valid*
(15:45:38) powerman: i've implemented small module ReadLine, which effectively return strings from fd, but as array of byte (and it doesn't convert array of byte to strings itself, it just check for \n as separator). so if you didn't need string, you don't waste cpu/memory for c2string(). also it's very effective in reusing same buffer for reading just like bufio did.
(15:46:03) powerman: question is, is it have sense to make it somehow compatible with bufio?
(15:46:33) mjl-: that's probably hard to do, making it compatible
(15:47:07) powerman: or just forget about bufio? thing is, if we just go this way, one day we realize we had rewritten bufio.
(15:47:23) powerman: (and ReadLine raise on errors, of course)
(15:47:27) mjl-: another thing i would like having is to get the \n (or other separator) stripped from the returned string by gets().  now that's not useful because empty strings would be nil...
(15:48:21) powerman: array[0] of byte and nil are different things, so it's possible to strip \n with such interface
(17:22:27) pension left the room (quit: Quit: quitting irc forever).
(17:31:43) jibun [~jibun@dsl-trebrasgw1-fff0c300-37.dhcp.inet.fi] entered the room.
(17:53:00) robot12 left the room (quit: Read error: Connection reset by peer).
(20:06:02) dreadlorde [~dreadlord@c-24-11-39-160.hsd1.mi.comcast.net] entered the room.
(20:24:03) jibun left the room (quit: Ping timeout: 240 seconds).
(20:35:26) Fish- [~Fish@9fans.fr] entered the room.
(21:20:26) mjl-: anth_r: the links in your 9fans mail on gsoc cannot be viewed by the public
(21:20:31) mjl-: This proposal is not made public, and you are not the student who submitted the proposal, nor are you a mentor for the organization it was submitted to.
(21:20:35) mjl-: that's what i get at least
(21:28:14) Fish-: yes
(21:33:42) dreadlorde left the room (quit: Read error: Operation timed out).
(00:10:18) __20h__ left the room (quit: Ping timeout: 240 seconds).
(00:11:01) __20h__ [~some_one@r-36.net] entered the room.
(00:35:01) Fish- left the room (quit: Quit: So Long, and Thanks for All the Fish).
(05:20:16) goozbach left the room (quit: Ping timeout: 240 seconds).
(05:24:26) goozbach [~goozbach@kessel.friocorte.com] entered the room.
(06:31:12) anth_x: mjl-: yeah, i know; only folks signed up as mentors in melange will, which is like a dozen people.
(06:31:38) anth_x: i wasn't going to try and recap the proposal's content there.
(09:06:29) bvalek2 [50bbcc06@gateway/web/freenode/ip.80.187.204.6] entered the room.
(09:15:19) robot12 [~kazzhilki@proxy10.ts.fujitsu.com] entered the room.
(12:53:19) powerman: mjl-: here is proposed interface for bufio(2) replacement: http://pastebin.ca/2051481
(12:54:39) powerman: i'm going to implement it today/tomorrow, so if you've any ideas about this it's better to discuss it earlier :)
(14:49:21) bvalek2 left the room (quit: Ping timeout: 252 seconds).
(15:35:52) KBme left the room (quit: Read error: Operation timed out).
(15:37:46) KBme [~KBme@2001:470:cabe:666:666:666:666:666] entered the room.
(18:58:59) jas: I've been using bufio mostly for writing or when using sexp
(19:03:21) jas: but typically I find that I need to have a separate reader spawned off for input filtering anyway. and for that it's just reading single bytes and using bufio hasn't been as beneficial
(19:26:09) robot12 left the room (quit: Quit: Leaving.).
(19:35:57) powerman: ok, i've implemented it http://code.google.com/p/inferno-contrib-iobuf/ and looks like it works, but i'm going to write more unit tests now to thoroughly test all features
(20:48:52) Fish- [~Fish@9fans.fr] entered the room.
(21:43:34) josephholsten [~josephhol@216.16.128.242] entered the room.
(21:48:16) josephholsten left the room (quit: Client Quit).
(23:57:24) jibun [~jibun@dsl-trebrasgw1-fff0c300-37.dhcp.inet.fi] entered the room.
(00:25:28) Fish- left the room (quit: Quit: So Long, and Thanks for All the Fish).
(05:33:05) The account has disconnected and you are no longer in this chat. You will be automatically rejoined in the chat when the account reconnects.