Does ClickOS-ctl work with lixs?

[stevepp]

Recently, on my laptop with an eight core CPU and 16GB RAM, I created 200 ClickOS VMs and used ClickOS-ctl to install the router instances into it. For my needs then I tried to update the routers installed into each ClickOS instance by first removing the old router and then install a new router. It takes quite a lot of time (i.e. several tens of seconds) to finish updating the routers installed into the 200 ClickOS VMs.

Then I wrote a python script which uses the multiprocessing module to create multiple processes and each process is responsible for updating routers installed in a fraction of the 200 ClickOS VMs. Of course, each process still utilizes the ClickOS-ctl tool. This time the overall time taken is ridiculously high (in the orders of tens of minutes).

I am wondering what might cause such slowdown. Does the XenStore have good support for concurrently read/write using ClickOS-ctl? Moreover, I am using the regular XenStore library instead of the LIghtweight XenStore (lixs). Will it help if I use the lixs? I am just wondering whether the ClickOS-ctl tool will work with the lightweight xenstore.

[fmanco]

Hi

There is a transaction mechanism on the xenstore. I'm fairly sure the slowdown is cause by conflicting transactions on the concurrent case.

What xenstore exactly are you using? There are two upstream implementations xenstored (C implementation) and oxenstored (ocaml implementation). The former has particularly poor transaction handling and therefore poor performance and is the default.

LiXS is supposed to be a drop-in replacement for upstream xenstore implementations so it should work directly with ClickOS-ctl. LiXS doesn't implement all xenstore functionality yet but is enough to work with ClickOS (and for most use-cases for that matter). LiXS is not particularly great at dealing with concurrent transactions because the merging algorithm is fairly conservative. However you should still see performance gains on both the sequential and concurrent cases. Maybe you can give LiXS a try.

Another alternative is to use oxenstored from xen 4.7. There are performance improvements related to transaction handling that dramatically speed things up in some cases.

Best

[stevepp]

@fmanco Thanks so much for the suggestions. That's very helpful. Yes, I am using the default XenStore daemon shipped with Xen 4.4.2. Because I am running both ClickOS VMs and the regular VMs with Linux kernels on the same host, I guess LiXS might not be a good option to try. I will re-build my system using Xen 4.7 and see whether oxenstored can offer some speedup.

[fmanco]

LiXS works with any guest, there are only a few operations that are not implemented (cnplab/lixs#1) but this should only affect stub-domains and maybe suspend/resume. However LiXS is indeed less stable than the upstream alternatives. I'll close the issue then. Would be curious to know whether Xen 4.7 helped.

Best

[stevepp]

Sorry that I need to re-open this issue. I followed the instructions in the LiXS github page to compile it. But I can not figure out how to configure xen to use LiXS. In the README it said we need to modify the file /etc/default/xencommons. However, I can not find this file on my host OS. My Dom0 runs Ubuntu 14.04 with kernel version 3.13. Is it because I am using wrong version of OS?

[fmanco]

No problem, but I don't have an Ubuntu installation at hand to analyse it. Are you running systemd? if so you might need to analyse the service file (the service should be called xenstored or similar) to see where it is getting the configurations from.

[stevepp]

No. Ubuntu 14.04 does not use systemd. Do you mean I need to find the configuration which sets what version of xenstored will be used during system boot? I found a file /etc/init.d/xen and it contains some settings related to Xen daemons. BTW, which OS does the LiXS install instruction target at?

[fmanco]

Hi. So I guess it uses upstart (which I know nothing about), but you're on the right track, you need to find some setting of the path to the xenstore daemon binary. LiXS instructions target Debian Wheezy.

[stevepp]

Just want to report some results using LiXS. There is obvious slowdown (around two times) using LiXS when updating the router instances installed into each ClickOS. Not quite sure why LiXS does not give performance benefit.

[fmanco]

Thanks for the effort of trying LiXS. Indeed this is weird, however since I'm not actively developing LiXS, for the time being I would suggest you just go for oxenstored. It would also be interesting if you could report whether the new version of oxenstored on Xen 4.7 increased performance.