Tuesday, May 09, 2006

Distcc and increasing compiling speed

Distcc allows you to easily distribute your compilation jobs over a number of machines, so even if u have an outdated system still u can compile huge codes pretty fast if u have hooked up ur system with some other using distcc.

The requirments are minimal:

  • Install GCC and distcc on all the machines you wish to use for the "compile farm".

As you can see there are no onerous requirements, common with distributed work. Specifically:

  • You don't need to have the same libraries and development packages on all systems.
  • You don't need to have a single filesystem which all machines can access.
  • The clocks on all the machines don't need to be in sync.
Use apt-get 2 install distcc:

apt-get install distcc

When the package is installed you'll be asked two questions :

  • Should distcc be started on boot?
    • Answer yes if think you will wish to use the package often, or no if not. (No is the default)
  • Which machines should be allowed to connect to the distcc server?
    • Asnwer with the machines you wish to allow, and the localhost

The answers you give will be stored away in the configuration file /etc/default/distcc.

Install the package on all the other machines you wish to use and make sure that each one is allowed to connect to the other.

There are several ways you can specify the hosts which should be used to perform the compilation:

  • Via the environmental variable "DISTCC_HOSTS"
  • Via the a per-user configuration file ~/.distcc/hosts

Assuming that you have two machines appy and abracadabra and wish to compile a job you could run using distcc as follows:

soumya@abracadabra:~/tmp$ export DISTCC_HOSTS="appy abracadabra"
soumya@abracadabra:~/tmp$ make CC=distcc

This specifies the two machines you wish to use for the compilation, and runs make telling it to use distcc as the compiler.

Once you do this you should find that the jobs are spread fairly evenly across the two machines.

Once you've installed the distcc program on each machine you must:

  • Tell the local instance which machines it can contact to compile files upon remotely.
  • Make sure you use "distcc" instead of "gcc" as the compiler command in your Makefile / build system.

Tats it done, now u need not go for coffee while compiling codes :)