Conversation with #inferno at Sun Apr 10 14:38:19 2011 on powerman-asdf@irc.freenode.net (irc)
(14:38:25) powerman: Now, we have to develop complex and flexible data filtering service, which I plan to implement in Limbo, using ~10-15 independent processes/modules, which will do parallel data analyse with different algorithms, produce some "expert knowledge" (of different types) as result of this analyse, keep these "knowledge" in databases (one per each type of "knowledge"), and use these databases to filter data items.
(14:40:09) powerman: Initial idea was run all these ~10-15 parts of this filtering service as independent processes, using file2chan for communications and json for serialization - as usually in other our network services, with only difference is file2chan instead of tcp sockets.
(14:41:17) powerman: But with current Limbo's json speed (with jit) our benchmarks show we'll spend about 50 seconds just on packing/unpacking json every minute, so there will be only about 10 second to do actual work (analyse/filter data).
(14:45:28) powerman: So, current options are:
(14:45:28) powerman: a) Run these ~10-15 parts of this service as threads in same process, and use channels for communications - this mean we'll have to pack/unpack json just once, on input and output of this service, so it will took 5 seconds every minute instead of 50. But this also mean we'll have more strong coupling and won't be able to run some parts (which may require a lot of CPU for data analyse) on different servers.
(14:48:58) powerman: b) Implement Inferno driver module for packing/unpacking json in C. This may be non-trivial because of needs to create  JValue (which is pick adt and I've no idea is it possible to work with pick adt's in C) and needs to use pure Limbo module Bufio from C level. This probably should speed up json in 10 times and make it comparable in speed with Perl's JSON::XS.
(14:50:24) powerman: c) Replace JSON with some another format, probably binary TLV, which can be packed/unpacked in pure Limbo much faster.
(14:55:36) powerman: I tend to go with "a)", but I'll be glad to hear any ideas about this situation.
(16:47:23) powerman: btw, I've tried to profile json module a little, and it looks like string concatenation "on the fly" took surprisingly much time!
(16:48:16) powerman: for example, in quote() last operation is:
(16:48:16) powerman:   return ns+"\"";
(16:49:53) powerman: replacing it with two operations:
(16:49:53) powerman:   ns[len ns] = '"';
(16:49:53) powerman:   return ns;
(16:49:53) powerman: result in speedup from 5.35 sec to 4.85 sec! and that's for complex benchmark which involve bufio etc. i.e. not just calling quote()
(16:50:59) powerman: if I use
(16:50:59) powerman:   ns += "\"";
(16:50:59) powerman: instead of
(16:50:59) powerman:   ns[len ns] = '"';
(16:50:59) powerman: then test time become 4.92 sec instead of 4.85 sec, which is quite impressive too.
(17:20:34) i__ left the room (quit: Quit: leaving).
(17:21:39) i__ [~none@unaffiliated/i--/x-3618442] entered the room.
(18:17:12) Fish [~Fish@9fans.fr] entered the room.
(19:08:54) __20h__ left the room (quit: *.net *.split).
(19:08:55) Fish left the room (quit: *.net *.split).
(19:08:56) Fish- left the room (quit: *.net *.split).
(19:08:56) eno left the room (quit: *.net *.split).
(19:08:56) mjl- left the room (quit: *.net *.split).
(19:08:56) yiyus left the room (quit: *.net *.split).
(19:08:58) vpm left the room (quit: *.net *.split).
(19:08:58) fgudin left the room (quit: *.net *.split).
(19:08:58) uriel left the room (quit: *.net *.split).
(19:09:02) goozbach left the room (quit: *.net *.split).
(19:09:02) jas left the room (quit: *.net *.split).
(19:09:02) mycroftiv left the room (quit: *.net *.split).
(19:09:19) KBme left the room (quit: *.net *.split).
(19:09:19) tensai_cirno left the room (quit: *.net *.split).
(19:09:19) GriffenJBS left the room (quit: *.net *.split).
(19:09:19) btdn left the room (quit: *.net *.split).
(19:09:19) vsrinivas left the room (quit: *.net *.split).
(19:09:19) i__ left the room (quit: *.net *.split).
(19:09:19) sl left the room (quit: *.net *.split).
(19:09:19) perdiy left the room (quit: *.net *.split).
(19:09:19) base2design left the room (quit: *.net *.split).
(19:11:26) Fish [~Fish@9fans.fr] entered the room.
(19:11:26) i__ [~none@unaffiliated/i--/x-3618442] entered the room.
(19:11:26) KBme [~KBme@2001:470:1f13:94::3] entered the room.
(19:11:26) perdiy [~mkhl@sxemacs/devel/perdix] entered the room.
(19:11:26) tensai_cirno [~cirno@77.232.15.216] entered the room.
(19:11:26) base2design [~base2desi@68-190-40-142.dhcp.athn.ga.charter.com] entered the room.
(19:11:26) Fish- [~Fish@exo3753.pck.nerim.net] entered the room.
(19:11:26) eno [~eno@nslu2-linux/eno] entered the room.
(19:11:26) vsrinivas [U2FsdGVkX1@batman.acm.jhu.edu] entered the room.
(19:11:26) GriffenJBS [~john@adsl-8-186-128.mia.bellsouth.net] entered the room.
(19:11:26) __20h__ [~some_one@r-36.net] entered the room.
(19:11:26) mycroftiv [~ircguy@h69-128-47-242.mdsnwi.dedicated.static.tds.net] entered the room.
(19:11:26) jas [~jas@adsl-69-215-39-41.dsl.chcgil.ameritech.net] entered the room.
(19:11:26) sl [none@sp.inri.net] entered the room.
(19:11:26) vpm [~vpm@reverse-94.fdn.fr] entered the room.
(19:11:26) mjl- [~none@berteun.nl] entered the room.
(19:11:26) goozbach [~goozbach@fedora/goozbach] entered the room.
(19:11:26) yiyus [1242712427@je.je.je] entered the room.
(19:11:26) uriel [~uriel@li43-28.members.linode.com] entered the room.
(19:11:26) fgudin [~none@digi00070.digicube.fr] entered the room.
(19:11:26) btdn [~btdn@138.74.188.207] entered the room.
(19:20:40) perdiy left the room (quit: Ping timeout: 246 seconds).
(19:21:05) perdix [~mkhl@sxemacs/devel/perdix] entered the room.
(19:51:01) anth [~a@c-98-244-54-221.hsd1.ca.comcast.net] entered the room.
(20:13:24) anth left the room (quit: Quit: anth).
(20:38:11) perdix left the room (quit: Remote host closed the connection).
(21:26:20) perdix [~mkhl@sxemacs/devel/perdix] entered the room.
(00:32:04) Fish left the room (quit: Quit: So Long, and Thanks for All the Fish).
(01:15:28) i__ left the room (quit: Remote host closed the connection).
(01:17:12) i__ [~none@unaffiliated/i--/x-3618442] entered the room.
(02:45:03) acmeuser [~acmeuser@ip-81-210-147-233.unitymediagroup.de] entered the room.
(02:48:20) acmeuser left the room (quit: Remote host closed the connection).
(04:02:16) acmeuser [~acmeuser@186.221.137.183] entered the room.
(04:02:19) acmeuser left the room (quit: Remote host closed the connection).
(04:26:14) bugQ [~bug@c-71-195-207-98.hsd1.ut.comcast.net] entered the room.
(04:33:15) btdn left the room (quit: Quit: leaving).
(04:37:25) bugQ left the room (quit: Ping timeout: 276 seconds).
(06:18:34) btdn [~btdn@138.74.188.207] entered the room.
(06:24:49) tensai_cirno left the room (quit: Ping timeout: 246 seconds).
(07:17:21) tensai_cirno [~cirno@194.154.66.189] entered the room.
(07:44:49) tensai_cirno left the room (quit: Read error: Connection reset by peer).
(07:45:37) tensai_cirno [~cirno@194.154.66.189] entered the room.
(07:46:20) base2design left the room (quit: Quit: Tclunk).
(08:03:25) tensai_cirno left the room (quit: Read error: Connection reset by peer).
(08:20:58) tensai_cirno [~cirno@194.154.66.189] entered the room.
(08:43:46) tensai_cirno left the room (quit: Ping timeout: 246 seconds).
(08:46:55) robot12 [~kazzhilki@proxy10.ts.fujitsu.com] entered the room.
(09:09:09) perdix left the room (quit: Ping timeout: 252 seconds).
(09:15:57) perdix [~mkhl@sxemacs/devel/perdix] entered the room.
(09:35:28) tensai_cirno [~cirno@77.232.15.216] entered the room.
(11:18:26) perdix left the room (quit: Read error: Connection reset by peer).
(11:18:47) perdix [~mkhl@sxemacs/devel/perdix] entered the room.
(12:13:07) tensai_cirno left the room (quit: Quit: Leaving).
(13:43:13) i__ left the room (quit: Ping timeout: 252 seconds).
(14:37:20) The account has disconnected and you are no longer in this chat. You will be automatically rejoined in the chat when the account reconnects.