y3VqWAqB

1**Table of Contents**
2- [Installation](#installation)
3  - [load a C11 compiler and anaconda](#load-a-c11-compiler-and-anaconda)
4  - [prepare an anaconda environment](#prepare-an-anaconda-environment)
5  - [check pip](#check-pip)
6  - [clear any previous attempts to build projectq](#clear-any-previous-attempts-to-build-projectq)
7  - [install pybind11](#install-pybind11)
8  - [build ProjectQ](#build-projectq)
9  - [check installation was successful](#check-installation-was-successful)
10  - [possible errors](#possible-errors)
11- [Usage](#usage)
12
13
14# Installation
15
16These are instructions for installing projectq and using its fast C++ backend on ARCHER or ARCUS-B, accessible by SLURM / PBS jobs.
17
18## load a C11 compiler and anaconda
19
20```bash
21module load gcc/5.3.0
22
23# if you're on ARCHER:
24module load anaconda-compute/python3
25module unload xalt
26
27# if you're on ARCUS-B:
28module load binutils python/anaconda3/4.3.0
29```
30
31Notice on ARCHER we loaded `anaconda-compute` rater than `anaconda`, so that our running jobs can access python.
32Notice on ARCUS-B we loaded some additional compiler instructions through `binutils`.
33
34To see what compiler and anaconda versions are available, call
35```bash
36module avail gcc
37module avail anaconda
38```
39
40You can clear your loaded modules with `module purge`, and view available modules with `module avail`, and view modules currently loaded with `module list`.
41
42> on ARCHER, calling `module purge` will limit which modules you can subsequently load, requiring you restart your SSH session 
43
44The above modules will need to be loaded each time you SSH into either ARCHER or ARCUS-B.
45
46## prepare an anaconda environment
47
48Here we create and activate an environment (and if on ARCHER, get some C++ symbols we'll need later).
49
50**On ARCUS-B**
51```bash
52conda create --name projqenv python=3.5 anaconda
53
54source activate projqenv
55```
56
57This environment will need to be activated (`source activate projqenv`) each time after SSH'ing in to ARCUS-B.
58
59Enter `conda info --env` to see a list of your conda environments, `source deactivate` to *leave* an environment and delete the environment with
60```bash
61conda remove --name projqenv --all
62```
63
64**On ARCHER**
65
66Care must be taken to put your conda environment in the *work* (accessible by the compute nodes) and not the default *home* directory.
67Replace `PROJNAME` and `USERNAME` below with your project code and username (which you SSH in with) respectively.
68
69```bash
70cd /work/PROJNAME/PROJNAME/USERNAME/
71
72rm -rf .condarc .conda .local .cache
73mkdir -p .conda .local .cache
74
75export CONDARC=$PWD/.condarc
76export PYTHONUSERBASE=$PWD/.local
77
78conda config --add envs_dirs $PWD/.conda/envs
79conda create --copy --name projqenv python=3.5 anaconda
80source activate projqenv
81
82conda install --copy libgcc
83```
84
85This environment will need to be activated (`source activate projqenv`) each time after SSH'ing in to ARCHER.
86
87
88## check pip
89
90Anaconda has a habit of giving you an old pip version, or a handle to the global pip (rather than your *projqenv* environment's pip).
91
92Check that
93```bash
94which pip
95```
96features 
97```
98~/.conda/envs/projqenv/bin/pip
99```
100and if not, call
101```
102conda install pip
103```
104
105Also ensure pip is up-to-date:
106```
107pip install --upgrade pip
108```
109
110## clear any previous attempts to build projectq
111
112```bash
113pip uninstall pybind11
114pip uninstall projectq
115rm -r ~/.cache/pip
116```
117
118This clears any built wheels, so we can force pip to (re)build projectq correctly
119
120## install pybind11
121
122Even the latest version of pip can goof up installing projectq's dependencies in the correct order.
123Give it a hand by installing pybind11 yourself first, though *not* the notoriously bugged 2.0.0 version!
124
125```bash
126
127# on ARCUS-B
128python -m pip install pybind11!=2.2.0
129
130# on ARCHER
131pip install --cache-dir $PWD/.cache pybind11!=2.2.0
132```
133
134On ARCHER, we were careful to install pybind in our cache in the /work directory.
135
136## build ProjectQ
137
138Let's now (force pip to) build projectq.
139
140```
141# ARCUS-B:
142env LANG=en_GB.UTF-8 CC=gcc CXX=g++ python -m pip install --user --force-reinstall --ignore-installed projectq
143
144# ARCHER:
145env LANG=en_GB.UTF-8 CC=gcc CXX=g++ pip install --cache-dir $PWD/.cache --user --force-reinstall --ignore-installed projectq
146```
147
148Here, we've
149- warned pip that projectq's README file contains non-ASCII characters (through `LANG`)
150- pointed pip to our C and C++ compilers (`gcc` and `g++`)
151- forced projectq to build, regardless of any previous builds (e.g. ones which failed to build the C++ backend)
152- on ARCHER, forced projq to install on the /work directory
153
154## check installation was successful
155
156All done (hopefully)! Open `python`, and call
157```python
158from projectq import MainEngine
159MainEngine()
160```
161If the following notice doesn't appear:
162```(Note: This is the (slow) Python simulator.)```
163then installation of projectq (with the C++ backend) was successful, at least on the front-end.
164
165On ARCHER, further check that
166```python
167import projectq
168projectq.__file__
169
170import pybind11
171pybind11.__file__
172```
173are located inside the /work dir, and not /home, so as to be accessible by the compute nodes.
174
175## possible errors
176
177- *Assembler Error: no such instruction ...*:  
178  you're missing some assembler instructions. On ARCUS-B, load these with `module load binutils`
179  
180- *(some generic compilation error)*:  
181  your C/C++ compilers mightn't support C11! Get a compatible one using (for example) `module load gcc/5.3.0`; view available compilers using `module avail gcc`
182  
183- *PermissionError: [Errno 13] Permission denied: '.../site-packages/projectq'*:  
184  you mightn't be using your conda environment's pip, but the global one instead. Check this which `which pip` (the direc should feature your environment name), and correct it with `conda install pip` then `pip install --upgrade pip`
185  
186- *FileNotFoundError: [Errno 2] No such file or directory: '.../.conda/envs/.pkgs/qt-5.6.2-4/info/index.json'*:  
187   You might be using an incompatible compiler version. Try a newer/older compiler using (for example) `module load gcc/5.3.0`; view available compilers using `module avail gcc`
188   
189- *ImportError: No module named 'pybind11' --- Failed building wheel for projectq*:  
190  pip didn't build projectq's dependencies in the correct order; this can still happen with the latest pip! Install pybind11 yourself first before retrying (`python -m pip install pybind11!=2.2.0`)
191  
192- Successful installation, but `MainEngine()` gives *(Note: This is the (slow) Python simulator.)*:  
193  Check building projectq was successful by checking `.../site-packages/projectq/backends/_sim/_cppsim...so` exists.
194  If not, force rebuilding as above. If so, you might need extra C symbols which you can get with `conda install libgcc`.
195  To better diagnose, edit `site-packages/projectq/backends/_sim/_simulator.py`. You can find `site packages` with `python -m site --user-site`. In `_simulator.py`, add `from ._cppsim import Simulator as SimulatorBackend` above the `try/except`. Then open `python` and call `from projectq import MainEngine` and you may see a more descriptive error message.
196  
197- Successful installation, `MainEngine()` gives no warning, but ARCHER PBS jobs give error `No module named projectq`.
198  Check `projectq` and `pybind11` are in the /work dir, by in `python`: `import projectq; projectq.__file__`.
199  If so, ensure you `CONDARC` and `PYTHONUSERBASE` in your submission script and pass `-b` to aprun (see below).
200
201# Usage
202
203## In the frontend
204
205Now that you've set projectq up, the next time you SSH into ARCHER or ARCUS-B, in order to access projectq
206on the submit machine, you'll need to call...
207
208**on ARCHER**:
209```bash
210module load gcc/5.3.0 anaconda-compute/python3
211module unload xalt
212source activate projqenv
213```
214**on ARCUS-B**:
215```bash
216module load gcc/5.3.0 python/anaconda3/4.3.0
217source activate projqenv
218```
219
220You can put the above code into a script...
221```
222nano myscript.sh
223< paste above code >
224chmod +x myscript.sh
225```
226which you can then call via `source myscript.sh` after SSH'ing in, or inside your SLURM or PBS submission scripts.
227
228Recall you can enter `conda info --env` to see a list of your conda environments, and `source deactivate` to *leave* an environment.
229
230## In a job
231
232### ARCUS-B:
233
234In your SLURM submit script, simply include
235```bash
236module load gcc/5.3.0 python/anaconda3/4.3.0
237source activate projqenv
238
239export OMP_NUM_THREADS=16
240export OMP_PROC_BIND=spread
241
242python my_projq_script.py
243```
244
245### ARCHER:
246
247Your PBS submit script should include (replacing `PROJECTNAME` and `USERNAME`)
248```bash
249
250mydir=/work/PROJECTNAME/PROJECTNAME/USERNAME
251
252module load anaconda-compute/python3
253module load gcc/5.3.0
254module unload xalt
255
256export CONDARC=${mydir}/.condarc
257export PYTHONUSERBASE=${mydir}/.local
258source activate projqenv
259
260export OMP_NUM_THREADS=24
261export OMP_PROC_BIND=spread
262
263cd "$PBS_O_WORKDIR"
264
265aprun -b -n 1 -d 24 python my_projq_script.py
266```
267
268Take note of the `-b` argument to aprun.
269
270We recommend creating a script `/work/PROJECTNAME/PROJECTNAME/USERNAME/prepq.sh` with contents
271```bash
272mydir=/work/PROJECTNAME/PROJECTNAME/USERNAME
273
274module load anaconda-compute/python3
275module load gcc/5.3.0
276module unload xalt
277
278export CONDARC=${mydir}/.condarc
279export PYTHONUSERBASE=${mydir}/.local
280source activate projqenv
281
282export OMP_NUM_THREADS=24
283export OMP_PROC_BIND=spread
284```
285and giving it permission to run (`chmod +x prepq.sh`) and replacing the above code in your PBS submission script with
286```bash
287
288cd "$PBS_O_WORKDIR"
289source .../path/to/prepq.sh
290
291aprun -b -n 1 -d 24 python my_projq_script.py
292
293```
294
295
296------------------
297
298An enormous thankyou to the ARCHER CSE team, Andrew Gittings of ARCUS and Thomas Haener of ProjectQ for their considerable help and patience!