Nautilus for TensorFlow example

See Fuz' page about general info about how to set up and run Nautilus jobs.  This page will describe a way to use Nautilus to run production TensorFlow jobs as an example, and how to transfer the data back and forth to your local systems.  This will create a persistent storage for you, and then it will transfer your files via rclone to google.  The Ceph system is probably a better model, but until we work that out and write it up, this works pretty well.

First, create some persistent storage as described in the intro document.  In my case, it's called 'pcwvol' - once you've successfully done it, you can see it via kubectl get pvc

kubectl get pvc
NAME                            STATUS    VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
fuzvol                          Bound     pvc-f9434cf7-15bd-11e9-b6d3-0cc47a6be994   2Gi        RWO            rook-block     224d
pcwvol                          Bound     pvc-b65a6c1c-48dd-11e9-86c5-0cc47a6be994   4Gi        RWO            rook-block     159d

Now create a simple job that connects back to it, say called Paul-interactive-pod.yaml.  This will mount your volume at /pcwvol (name it what you want!) and allow the pod to run for 600 seconds for you to get in and set things up (on the volume).

apiVersion: v1
kind: Pod
metadata:
  name: gpu-pod
  namespace: ucsb-csc
spec:
      restartPolicy: Never
      containers:
      - name: fs-container
        image:  nginx:1.9.1
        command: ["/bin/bash"]
        args: ["-c", "chmod 777 /pcwvol; sleep 600;"]
        resources:
          limits:
            nvidia.com/gpu: 1
            memory: "64Mi"
            cpu: "250m"
        volumeMounts:
        - name: pcwvol
          mountPath: /pcwvol
      volumes:
        - name: pcwvol
          persistentVolumeClaim:
           claimName: pcwvol

Launch this pod (basically, think booting up a system) with kubectl create -f Paul-interactive-pod.yaml.  You can see it's running with

$ kubectl get pods
NAME      READY     STATUS    RESTARTS   AGE
gpu-pod   1/1       Running   0          17s

You can now connect to it with kubectl exec -it gpu-pod -- /bin/bash  which will open a shell to it.  You can then 'cd /pcwvol' to see your files on the persistent filesytem (there won't be any yet).

Create a file in /pcwvol named setup.sh with the following (but change the /pcwvol to whatever your volume is named):

export RCLONE_CONFIG=/pcwvol/.rclone/rclone.conf
export PATH=/pcwvol/bin:$PATH

Exit, and kill the pod with 'kubectl delete pod gpu-pod'.  Then restart it (the 'create' command above).  Login (with the 'exec bash' line) -  check in /pcwvol to see your file is still there.  If you type 'history' you won't have a history - since it's rebooted to create a new system (pod) - but the persistent storage has saved your files.

Now, we'll finish setting up, by getting a copy of the 'rclone' program, and also set it up to see your Google Drive.

cd /pcwvol/bin

wget http://admin3.cnsi.ucsb.edu/kube/rclone

.  /pcwvol/setup.sh

rclone config

At this point it will ask you a bunch of questions.  The important ones are, start with 'n' (new remote), say named 'Google'.  Pick #11 (google drive).  Give it 'scope 1'.  Just click return for all the other questions you don't know about (service_account, etc.).  Don't do advanced config.  When it as's 'use auto config' say N (since you're on a 'headless' system).  It will give you a long URL that you cut and paste into a browser, the browser gives a code back that you enter into 'rclone' (it will be sitting and waiting for it).  Once you do that, you're set!

You can test it with (say to show directories)

root@gpu-pod:/pcwvol# rclone lsd Google:
          -1 2019-05-28 18:59:34        -1 AWS-accounting-data
          -1 2017-12-14 00:36:16        -1 CI-committee
          -1 2018-02-28 19:42:23        -1 GlobusFiles
          -1 2017-11-27 19:57:17        -1 IntelCompilers
          -1 2019-06-11 17:00:19        -1 KV

I like to make a separate directory that all my Nautlius stuff lives in at Google - I don't want my script having a typo and accidentally deleting all my google files!

rclone mkdir Google:Nautilus

then, to look at a listing of this directory

rclone lsl Google:Nautilus

Set up 'rclone' on your own computer too that you're running 'kubectl' on  (or just use google drive through the browser).  Go ahead and shut down this pod that was really just for setting up everything.  kubectl delete pod gpu-pod.

What we'll do next is a 'job' - basically it will boot up a system and run a job (script) for us.  You don't want to just launch 'pods' and leave them running with a long sleep command, since if you're not using it, that ties up the hardware(e.g. a GPU) so that someone else can't use it.  Using a 'job' will start the 'pod' (a system) and then when it's done, shut down that pod, freeing up the resources again.

Now with our volume what can happen is you just drop files in your Google drive, Nautilus folder, and the first step in your YAML file is you can sync it.  So now let's try Fuz' example of doing a tensorflow job.  First - in google drive, drop the files you need for tensorflow, e.g. 'classify_image.py' and some jpg files.  Then you can just use the following YAML to run TF on say 4 different jpgs.   Note that you chain all of your commands together - and don't forget the semicolon on the end of each line!!!!  This will clone your google directory, run the jobs, and then clone it back to google.

When it's done, be sure to clean up the job, i.e. kubectl delete job gpu-job

apiVersion: batch/v1
kind: Job
metadata:
  name: gpu-job
  namespace: ucsb-csc
spec:
  template:
    spec:
      restartPolicy: Never
      containers:
      - name: fs-container
        image: fuzzrence/tf-gpu2
        command: ["/bin/bash", "-c"]
        args:
          -  echo start;
             . /pcwvol/setup.sh;
             chmod 777 /pcwvol;
             echo 'helloworld' > /pcwvol/test4.txt;
             nvidia-smi >> /pcwvol/test4.txt;
             cd /pcwvol/tf;
             echo 'getting ready to rclone' >> /pcwvol/test4.txt;
             rclone sync Google:Nautilus . >> /pcwvol/test4.txt;
             python ./classify_image.py --image imagenet/basic-cat.jpg > log.basic.cat;
             python ./classify_image.py --image hal.jpg  > log.hal;
             python ./classify_image.py --image dog1.jpg  > dog1.log;
             python ./classify_image.py --image dog2.jpg  > dog2.log;
             python ./classify_image.py --image IMG_6216.jpg > whitney.log;
             echo 'done' >> /pcwvol/test4.txt;
             /bin/cp /pcwvol/test4.txt /pcwvol/tf/test4.txt;
             rclone sync . Google:Nautilus > /pcwvol/sync.log;
             sleep 30;
        resources:
          requests:
            nvidia.com/gpu: 1
            memory: "1024Mi"
            cpu: "1000m"
          limits:
            nvidia.com/gpu: 1
            memory: "2048Mi"
            cpu: "1000m"
        volumeMounts:
        - name: pcwvol
          mountPath: /pcwvol
      volumes:
        - name: pcwvol
          persistentVolumeClaim:
            claimName: pcwvol