Using the Xeon Phi coprocessor

For development work using the Xeon Phi, login to 'knot-gpu' - this node has both NVIDIA GPUs and an Intel Xeon Phi coprocessor.  This system is meant for development work, for production runs, GPU jobs are run in the 'gpu' queue, and Xeon Phi jobs are run on 'phi-node'.

There is quite a bit to learn and understand about this coprocessor - some good links:

• Intel quick start Xeon Phi guide
• Slides from training at XSEDE/TACC
• Dr.Dobbs Journal jumpstart to programming the Phi
• Colfax International Xeon Phi research pages

First, let Fuz or Paul know that you want to use the Phi so we can enable your account on it.  The easy way to think about the Phi is that it is a separate node (with a lot of cores). 

One consequence of this is that the Phi has its own filesystem, so your files in /home aren't visible - they need to be copied there via SCP first (And results copied back out). 

For a very simple example:

#include <stdio.h>
#include <unistd.h>

// This is the example from Colfax

int main(){
printf("Hello world! I have %ld logical cores.\n",
sysconf(_SC_NPROCESSORS_ONLN ));

}

you can compile/run it on the host computer (knot-gpu), by

burak@node139:~/MIC/hello$ icpc hello.cc -o hello.XEON
burak@node139:~/MIC/hello$ ./hello.XEON
Hello world! I have 12 logical cores.

You can then use the Intel compiler to create code for the coprocessor by adding the flag '-mmic'

icpc -mmic hello.cc -o hello.MIC

However, if you try to run it, you'll get an error

burak@node139:~/MIC/hello$ ./hello.MIC
-bash: ./hello.MIC: cannot execute binary file

since indeed, the knot-gpu system is not a 'Phi' - you need to login to the Phi coprocessor and run it.  However, you first need to copy the file there since /home isn't available on the Phi.  Then you run it by logging in (here all on one line)

burak@node139:~/MIC/hello$ scp hello.MIC mic0:~/hello.MIC
hello.MIC                                         100%   11KB  10.6KB/s   00:00  
burak@node139:~/MIC/hello$ ssh mic0 ./hello.MIC
Hello world! I have 228 logical cores.

Alternatively, you can run the binary executable on the coprocessor using micnativeloadex :

burak@node139:~/MIC/hello$ micnativeloadex hello.MIC 
Hello world! I have 228 logical cores.

Linking libraries: 

The object files need to be made visible to Xeon Phi. Either you can scp these files or use micnativeloadex. For example, suppose you are trying to link OpenMP libraries during compilation:

burak@node139:~/MIC/hello$ icpc -mmic -openmp hello.cc -o hello-openmp.MIC

When you try to run on Xeon Phi, you will get an error:

burak@node139:~/MIC/hello$ scp hello-openmp.MIC mic0:~/
hello-openmp.MIC                    100%   11KB  11.3KB/s   00:00    
burak@node139:~/MIC/hello$ ssh mic0 ~/hello-openmp.MIC
/home/burak/hello-openmp.MIC: error while loading shared libraries: libiomp5.so: cannot open shared object file: No such file or directory

One option is to scp the libiomp5.o to mic0. The better option is to set yout environment variable SINK_LD_LIBRARY and use micnativeloadex:

burak@node139:~/MIC/hello$ export SINK_LD_LIBRARY_PATH=$MIC_LD_LIBRARY_PATH
burak@node139:~/MIC/hello$ micnativeloadex hello-openmp.MIC
Hello world! I have 228 logical cores.