# Installing AraSim on OSC

Readding this because I realized it was deleted when I went looking for it

Quick Links:
- https://github.com/ara-software/AraSim # AraSim github repo (bottom has installation instructions that are sort of right)
- Once AraSim is downloaded: AraSim/UserGuideTex/AraSimGuide.pdf (manual for AraSim) might have to be downloaded if you can't view pdf's where you write code

Step 1: 
We need to add in the dependancies. AraSim needs multiple different packages to be able to run correctly. The easiest way on OSC to get these without a headache is to add the following to you .bashrc for your user.

cvmfs () {
    module load gnu/4.8.5
    export CC=`which gcc`
    export CXX=`which g++`
    if [ $# -eq 0 ]; then
        local version="trunk"
    elif [ $# -eq 1 ]; then
        local version=$1
    else
        echo "cvmfs: takes up to 1 argument, the version to use"
        return 1
    fi
    echo "Loading cvmfs for AraSim"
    echo "Using /cvmfs/ara.opensciencegrid.org/${version}/centos7/setup.sh"
    source "/cvmfs/ara.opensciencegrid.org/${version}/centos7/setup.sh"
    #export JUPYTER_CONFIG_DIR=$HOME/.jupyter
    #export JUPYTER_PATH=$HOME/.local/share/jupyter
    #export PYTHONPATH=/users/PAS0654/alansalgo1/.local/bin:/users/PAS0654/alansalgo1/.local/bin/pyrex:$PYTHONPATH
}

If you want to view my bashrc
- /users/PAS1977/jacobweiler/.bashrc

Reload .bashrc
- source ~/.bashrc

Step 2:
Go to directory that you want to put AraSim and type: 
- git clone https://github.com/ara-software/AraSim.git
This will download the github repo

Step 3:
We need to use make and load sourcing
- cd AraSim
- cvmfs
- make
wait and it should compile the code 

Step 4:
We want to do a test run with 100 neutrinos to make sure that it does *actually* run
Try: - ./AraSim SETUP/setup.txt
This errored for me (probably you as well) 
Switch from frequency domain to time domain in the setup.txt
- cd SETUP
- open setup.txt 
- scroll to bottom
- Change SIMULATION_MODE = 1
- save
- cd .. 
- ./AraSim SETUP/setup.txt 
This should run quickly and now you have AraSim setup!

This guide is modified from section (d) of the worksheet inside Module 10 of Phys 6810 Computational Physics (Spring 2023).

NOTE: gprof does not work on macOS. Please use a linux machine (such as OSC)

To use gprof, compile and link the relevant codes with the -pg option:
Take a look at the Makefile make_hello_world and modify both the CFLAGS and LDFLAGS lines to include -pg

Compile and link the script by typing
    make -f make_hello_world

Execute the program
    ./hello_world.x

With the -pg flags, the execution will generate a file called gmon.out that is used by gprof.
The program has to exit normally (e.g. we can't stop with a ctrl-C).
Warning: Any existing gmon.out file will be overwritten.

Run gprof and save the output to a file (e.g., gprof.out) by
    gprof hello_world.x > gprof.out

We should at this point see a text file called gprof.out which contains the profile of hello_world.cpp
    vim gprof.out

These documents contain information on how to run IceCube/ARA coincidence simulations and analysis. All technical information of where codes are stored and how to use them is detailed in the technical note. Other supportive information for physics understanding is in the powerpoint slides. The technical note will direct you to other documents in this elog in the places where you may need supplemental information.

These are instructions I put together as I was first figuring out how to compile PueoSim/NiceMC. This was originally done on machines running CentOS 7, however has since been replicated on the OSC machines (running RedHat 7.9 I think?). I generally try to avoid any `module load` type prerequisites, instead opting to compile any dependencies from source. You _might_ be able to get this to work by `module load`ing e.g. fftw, but try this at your own peril.

#pueoBuilder Installation Tutorial

This tutorial will guide you through the process of building the tools included in pueoBuilder from scratch, including the prerequisites and any environment variables that you will need to set. This sort of thing is always a bit of a nightmare process for me, so hopefully this guide can help you skip some of the frustration that I ran into. I did not have root acces on the system I was building on, so the instructions below are what I had to do to get things working with local installations. If you have root access, then things might be a bit easier. For reference I'm working on CentOS 7, other operating systems might have different problems that arise. 

##Prerequisites
As far as I can tell, the prerequisites that need to be built first are:

-Python 3.9.18 (Apr. 6 2024 edit by Jason Yao, needed for ROOT 6.26-14)
-cmake 3.21.2 (I had problems with 3.11.4)
-gcc 11.1.0 (9.X will not work) (update 4/23/24: If you are trying to compile ROOT 6.30, you might need to downgrade to gcc 10.X, see note about TBB in "Issues I ran into" at the end)
-fftw 3.3.9
-gsl 2.7.1 (for ROOT)
-ROOT 6.24.00
-OneTBB 2021.12.0 (if trying to compile ROOT 6.30)

###CMake
You can download the source files for CMake here: https://cmake.org/download/. Untar the source files with:

    tar -xzvf cmake-3.22.1.tar.gz

Compiling CMake is as easy as following the directions on the website: https://cmake.org/install/, but since we're doing a local build, we'll use the `configure` script instead of the listed `bootstrap` script. As an example, suppose that I downloaded the above tar file to `/scratch/wluszczak/cmake`: 

    mkdir install
    cd cmake-3.22.1
    ./configure --prefix=/scratch/wluszczak/cmake/install
    make
    make install

You should additionally add this directory to your `$PATH` variable:

    export PATH=/scratch/wluszczak/cmake/install/bin:$PATH
    

To check to make sure that you are using the correct version of CMake, run:

    cmake --version

and you should get:

    cmake version 3.22.1

    CMake suite maintained and supported by Kitware (kitware.com/cmake).

### gcc 11.1.0

Download the gcc source from github here: https://github.com/gcc-mirror/gcc/tags. I used the 11.1.0 release, though there is a more recent 11.2.0 release that I have not tried. Once you have downloaded the source files, untar the directory:

    tar -xzvf gcc-releases-gcc-11.1.0.tar.gz

Then install the prerequisites for gcc:
    
    cd gcc-releases-gcc-11.1.0
    contrib/download_prerequisites

One of the guides I looked at also recommended installing flex separately, but I didn't seem to need to do this, and I'm not sure how you would go about it without root priviledges, though I imagine it's similar to the process for all the other packages here (download the source and then build by providing an installation prefix somewhere)

After you have installed the prerequisites, create a build directory:

    cd ../
    mkdir build
    cd build

Then configure GCC for compilation like so:

    ../gcc-releases-gcc-11.1.0/configure -v --prefix=/home/wluszczak/gcc-11.1.0 --enable-checking=release --enable-languages=c,c++,fortran --disable-multilib --program-suffix=-11.1

I don't remember why I installed to my home directory instead of the /scratch/ directories used above. In principle the installation prefix can go wherever you have write access. Once things have configured, compile gcc with:

    make -j $(nproc)
    make install

Where `$(nproc)` is the number of processing threads you want to devote to compilation. More threads will run faster, but be more taxing on your computer. For reference, I used 8 threads and it took ~15 min to finish. 

Once gcc is built, we need to set a few environment variables:

    export PATH=/home/wluszczak/gcc-11.1.0/bin:$PATH
    export LD_LIBRARY_PATH=/home/wluszczak/gcc-11.1.0/lib64:$LD_LIBRARY_PATH

We also need to make sure cmake uses this compiler:

    export CC=/home/wluszczak/gcc-11.1.0/bin/gcc-11.1
    export CXX=/home/wluszczak/gcc-11.1.0/bin/g++-11.1
    export FC=/home/wluszczak/gcc-11.1.0/bin/gfortran-11.1

If your installation prefix in the configure command above was different, substitute that directory in place of `/home/wluszczak/gcc-11.1.0` for all the above export commands. To easily set these variables whenever you want to use gcc-11.1.0, you can stick these commands into a single shell script:

    #load_gcc11.1.sh
    export PATH=/home/wluszczak/gcc-11.1.0/bin:$PATH
    export LD_LIBRARY_PATH=/home/wluszczak/gcc-11.1.0/lib64:$LD_LIBRARY_PATH

    export CC=/home/wluszczak/gcc-11.1.0/bin/gcc-11.1
    export CXX=/home/wluszczak/gcc-11.1.0/bin/g++-11.1
    export FC=/home/wluszczak/gcc-11.1.0/gfortran-11.1

(again substituting your installation prefix in place of mine). You can then set all these environment variables by simply running:
    
    source load_gcc11.1.sh

Once this is done, you can check that gcc-11.1.0 is properly installed by running:

    gcc-11.1 --version

Note that plain old

    gcc --version

might still point to an older version of gcc. This is fine though. 

###FFTW 3.3.9
Grab the source code for the appropriate versino of FFTW from here: http://www.fftw.org/download.html

However, do NOT follow the installation instructions on the webpage. Those instructions might work if you have root privileges, but I personally couldn't seem to to get things to work that way. Instead, we're going to build fftw with cmake. Untar the fftw source files:

    tar -xzvf fftw-3.3.9.tar.gz

Make a build directory and cd into it:
    
    mkdir build
    cd build

Now build using cmake, using the flags shown below. For reference, I downloaded and untarred the source file in `/scratch/wluszczak/fftw/build`, so adjust your install prefix accordingly to point to your own build directory that you created in the previous step.

    cmake -DCMAKE_INSTALL_PREFIX=/scratch/wluszczak/fftw/build/ -DBUILD_SHARED_LIBS=ON -DENABLE_OPENMP=ON -DENABLE_THREADS=ON ../fftw-3.3.9
    make install -j $(nproc)

Now comes the weird part. Remove everything except the `include` and `lib64` directories in your build directory (if you installed to a different `CMAKE_INSTALL_PREFIX`, the include and lib64 directories might be located there instead. The important thing is that you want to remove everything, but leave the `include` and `lib64` directories untouched):

    rm *
    rm -r CMakeFiles

Now rebuild fftw, but with an additional flag:

    cmake -DCMAKE_INSTALL_PREFIX=/scratch/wluszczak/fftw/build/ -DBUILD_SHARED_LIBS=ON -DENABLE_OPENMP=ON -DENABLE_THREADS=ON -DENABLE_FLOAT=ON ../fftw-3.3.9
    make install -j $(nproc)

At the end of the day, your fftw install directory should have the following files:

    include/fftw3.f  
    include/fftw3.f03
    include/fftw3.h  
    include/fftw3l.f03  
    include/fftw3q.f03 
    lib64/libfftw3f.so          
    lib64/libfftw3f_threads.so.3      
    lib64/libfftw3_omp.so.3.6.9  
    lib64/libfftw3_threads.so
    lib64/libfftw3f_omp.so        
    lib64/libfftw3f.so.3        
    lib64/libfftw3f_threads.so.3.6.9  
    lib64/libfftw3.so            
    lib64/libfftw3_threads.so.3
    lib64/libfftw3f_omp.so.3      
    lib64/libfftw3f.so.3.6.9    
    lib64/libfftw3_omp.so             
    lib64/libfftw3.so.3          
    lib64/libfftw3_threads.so.3.6.9
    lib64/libfftw3f_omp.so.3.6.9  
    lib64/libfftw3f_threads.so  
    lib64/libfftw3_omp.so.3           
    lib64/libfftw3.so.3.6.9

Why do we have to do things this way? I don't know, I'm bad at computers. Maybe someone more knowledgeable knows. I found that when I didn't do this step, I'd run into errors that pueoBuilder could not find some subset of the required files (either the ones added by building with `-DENABLE_FLOAT`, or the ones added by building without `-DENABLE_FLOAT`). 

Once fftw has been installed, export your install directory (the one with the include and lib64 folders) to the following environment variable:

    export FFTWDIR=/scratch/wluszczak/fftw/build

Again, substituting your own fftw install prefix that you used above in place of `/scratch/wluszczak/fftw/build`

###gsl 2.7.1
gsl 2.7.1 is needed for the `mathmore` option in ROOT. If you have an outdated version of gsl, ROOT will still compile, but it will skip installing `mathmore` and `root-config --has-mathmore` will return `no`. To fix this, grab the latest source code for gsl from here: https://www.gnu.org/software/gsl/. Untar the files to a directory of your choosing:

    tar -xzvf gsl-latest.tar.gz

For some reason I also installed gsl to my home directory, but in principle you can put it wherever you want. 

    mkdir /home/wluszczak/gsl
    ./configure --prefix=/home/wluszczak/gsl
    make
    make check
    make install

To make sure ROOT can find this installation of gsl, you'll again need to set an environment variable prior to building ROOT:

    export GSL_ROOT_DIR=/home/wluszczak/gsl/
    
I also added this to my $PATH variable, though I don't remember if that was required to get things working or not:

    export PATH=/home/wluszczak/gsl/bin/:$PATH 
    export LD_LIBRARY_PATH=/home/wluszczak/gsl/lib:$LD_LIBRARY_PATH

###Python 3.9.18
Apr. 6, 2024 edit by Jason Yao:
I was able to follow this entire ELOG to install root 6.24.00, but I also was getting warnings/errors that seem to be related to Python,
so I went ahead and installed Python 3.9.18 (and then a newer version of ROOT just for fun).
Note that even though we can `module load python/3.9-2022.05` on OSC, I am 90% sure that this provided Python instance is no good as far as ROOT is concerned.

Head over to https://www.python.org/downloads/release/python-3918/ to check out the source code.
You can run
    wget https://www.python.org/ftp/python/3.9.18/Python-3.9.18.tgz
to download it on OSC; then, run
    tar -xzvf Python-3.9.18.tgz
    cd Python-3.9.18

Next we will compile Python from source. I used this website for this step.
I wanted to install to `${HOME}/usr/Python-3.9.18/install`, so
    ./configure --prefix=${HOME}/usr/Python-3.9.18/install --enable-shared
Note that we must have the flag `--enable-shared` "to ensure that shared libraries are built for Python. By not doing this you are preventing any application which wants to use Python as an embedded environment from working" according to this guy.
(The corresponding error when you try to compile ROOT later on would look like "...can not be used when making a shared object; recompile with -fPIC...")

After configuration,
    make -j8 && make install
(using 8 threads)

After installation, head over to the install directory, and then
    cd bin
You should see `pip3` and `python3.9`. If you run
    ./python3
you should see the Python interactive terminal
    Python 3.9.18 (main, Apr  5 2024, 22:49:51) 
    [GCC 11.1.0] on linux
    Type "help", "copyright", "credits" or "license" for more information.
    >>>
Add the python3 in this folder to your PATH variable. For example, 
    export PATH=${HOME}/usr/Python-3.9.18/install/bin:${PATH}
 

While we are in the python install directory, we might as well also use the `pip3` there to install numpy:
    ./pip3 install numpy
(I am not sure if this is absolutely needed by ROOT, but probably)

Next comes the important bit. You need to add the `lib/` directory inside you python installation to the environment variable $LD_LIBRARY_PATH
    export LD_LIBRARY_PATH=${HOME}/usr/Python-3.9.18/install/lib:$LD_LIBRARY_PATH
according to stackoverflow. Without this step I ran into errors when compiling ROOT.

###ROOT 6.24.00
Download the specific version of ROOT that you need from here: https://root.cern/install/all_releases/

You might need to additionally install some of the dependencies (https://root.cern/install/dependencies/), but it seems like everything I needed was already installed on my system. 

Untar the source you downloaded:
    
    tar -xzvf root_v6.24.00.source.tar.gz

Make some build and install directories:

    mkdir build install
    cd build

Run CMake, but be sure to enable the fortan, mathmore and minuit2 options. For reference, I had downloaded and untarred the source files to `/scratch/wluszczak/root`. Your installation and source paths will be different.

    cmake -DCMAKE_INSTALL_PREFIX=/scratch/wluszczak/root/install/ /scratch/wluszczak/root/root-6.24.00/ -Dfortran=ON -Dminuit2=ON -Dmathmore=ON

Note: if you end up with an error related to compiling XROOTD, then add -Dxrootd=OFF to the original cmake command above.

Then proceed to start the build:

    cmake --build . --target install -j $(nproc)
    

If everything has worked then after the above command finishes running, you should be able to run the following file to finish setting up ROOT:

    source ../install/bin/thisroot.sh

##pueoBuilder
By this point, you should have working installations of CMake 3.21.2, gcc-11.1.0, fftw 3.3.9, and ROOT 6.24.00. Additionally, the following environment variables should have been set:

    export PATH=/scratch/wluszczak/cmake/install/bin:$PATH

    export PATH=/home/wluszczak/gcc-11.1.0/bin:$PATH
    export LD_LIBRARY_PATH=/home/wluszczak/gcc-11.1.0/lib64:$LD_LIBRARY_PATH

    export CC=/home/wluszczak/gcc-11.1.0/bin/gcc-11.1
    export CXX=/home/wluszczak/gcc-11.1.0/bin/g++-11.1
    export FC=/home/wluszczak/gcc-11.1.0/gfortran-11.1

    export FFTWDIR=/scratch/wluszczak/fftw/build

At this point, the hard work is mostly done. Check out pueoBuilder with:

    git clone git@github.com:PUEOCollaboration/pueoBuilder 

set the following environment variables:

    export PUEO_BUILD_DIR=/scratch/wluszczak/PUEO/pueoBuilder
    export PUEO_UTIL_INSTALL_DIR=/scratch/wluszczak/PUEO/pueoBuilder
    export NICEMC_SRC=${PUEO_BUILD_DIR}/components/nicemc
    export NICEMC_BUILD=${PUEO_BUILD_DIR}/build/components/nicemc
    export PUEOSIM_SRC=${PUEO_BUILD_DIR}/components/pueoSim
    export LD_LIBRARY_PATH=${PUEO_UTIL_INSTALL_DIR}/lib:$LD_LIBRARY_PATH

Where $PUEO_BUILD_DIR and $PUEO_UTIL_INSTALL_DIR point to where you cloned pueoBuilder to (in my case, `/scratch/wluszczak/PUEO/pueoBuilder`. Now you should be able to just run:

    ./pueoBuilder.sh

Perform a prayer to the C++ gods while you're waiting for it to compile, and hopefully at the end of the day you'll have a working set of PUEO software. 

##Issues I Ran Into
If you already have an existing installation of ROOT, you may still need to recompile to make sure you're using the same c++ standard that the PUEO software is using. I believe the pre-compiled ROOT binaries available through their website are insufficient, though maybe someone else has been able to get those working. 

If you're running into errors about c++ standard or compiler version even after you have installed gcc-11.1.0, then for some reason your system isn't recognizing your local installation of gcc-11.1.0. Check the path variables ($PATH and $LD_LIBRARY_PATH) to make sure the gcc-11.1.0 `bin` directory is being searched.

If you're running into an error that looks like:
        
    CMake Error: The following variables are used in this project, but they are set to NOTFOUND.
    Please set them or make sure they are set and tested correctly in the CMake files:
    FFTWF_LIB (ADVANCED)

then pueoBuilder can't seem to find your fftw installation (or files that are supposed to be included in that installation), try rebuilding with different flags according to which files it seems to think are missing.

If it seems like pueoBuilder can't seem to find your fftw installation at all (i.e. you're getting some error that looks like `missing: FFTW_LIBRARIES` or `missing: FFTW_INCLUDES`), check the environment variables that are supposed to point to your fftw installation (`$FFTWDIR`) and make sure there are the correct files in the `lib` and `include` subdirectories. 

Update: 6/23/24: The latest version of ROOT (6.30) will fail to compile on OSC unless you manually compile TBB as well. An easy workaround is to simply downgrade to ROOT 6.24, however if you really need ROOT 6.30 you can follow the instructions below to install TBB and compile ROOT:

You will first need to downgrade to GCC 10.X. TBB will not compile with GCC 11. This can be done by following the GCC installation isntructions above, except starting with GCC 10 source code instead of GCC 11.

To install TBB yourself, download the source code (preferably the .tar.gz file) from here: https://github.com/oneapi-src/oneTBB/releases/tag/v2021.12.0. Move the file to the directory where you want to install TBB and untar it with:

    tar -xzvf oneTBB-2021.12.0.tar.gz

Make some build and install directories:

    mkdir build install
    cd build

Then configure cmake:

    cmake -DCMAKE_INSTALL_PREFIX=/path/to/tbb/install

Then compile with:

    cmake --build .

Once this has finished running, you can add the installation to you $PATH and $LD_LIBRARY_PATH variables:

    export PATH=/path/to/tbb/install/bin:$PATH
    export LD_LIBRARY_PATH=/path/to/tbb/install/lib64:$LD_LIBRARY_PATH

You can then proceed as normal, except when compiling root you will need one additional cmake flag (Dbuiltin_tbb=ON):

  cmake -DCMAKE_INSTALL_PREFIX=/scratch/wluszczak/root/install/ /scratch/wluszczak/root/root-6.24.00/ -Dfortran=ON -Dminuit2=ON -Dmathmore=ON -Dbuiltin_tbb=ON

And hopefully this should work. This process is a little bit more involved than just downgrading ROOT, so try to avoid going down this route unless absolutely necessary.

Some programs on OSC require authorized access to use in the form of a license. The license will be automatically read if it is available whenever you open a program on OSC, provided you have access to the license. In order to have access to a license, you need to fill out the attached form and send it to Amy to forward to OSC. Available programs (at least some of) which require a license can be found here: https://www.osc.edu/resources/available_software/software_list . Replace the name of the program at the top of the form with the desired software.

Disclaimer: This might not be the best solution to this problem. I arrived here after a lot of googling and stumbling across this thread with a similar problem for an unrelated project: https://github.com/xtensor-stack/xtensor-fftw/issues/52. If you're someone who actually knows cmake, maybe you have a better solution.

When compiling both pueoBuilder and anitaBuildTools, I have run into a cmake error that looks like:

CMake Error at /apps/cmake/3.17.2/share/cmake-3.17/Modules/FindPackageHandleStandardArgs.cmake:164 (message):
  Could NOT find FFTW (missing: FFTW_LIBRARIES)

(potentially also missing FFTW_INCLUDES). Directing CMake to the pre-existing FFTW installations on OSC does not seem to do anything to resolve this error. From what I can tell, this might be related to how FFTW is built, so to get around this we need to build our own installation of FFTW using cmake instead of the recommended build process. To do this, grab the whatever version of FFTW you need from here: http://www.fftw.org/download.html (for example, I needed 3.3.9). Untar the source file into whatever directory you're working in:

    tar -xzvf fftw-3.3.9.tar.gz

Then make a build directory and cd into it:
    
    mkdir install
    cd install

Now build using cmake, using the flags shown below.

    cmake -DCMAKE_INSTALL_PREFIX=$(path_to_install_loc) -DBUILD_SHARED_LIBS=ON -DENABLE_OPENMP=ON -DENABLE_THREADS=ON ../fftw-3.3.9

For example, I downloaded and untarred the source file in `/scratch/wluszczak/fftw/`, and my install prefix was `/scratch/wluszczak/fftw/install/`. In principle this installation prefix can be anywhere you have write access, but for the sake of organization I usually try to keep everything in one place.

Once you have configured cmake, go ahead and install:

    make install -j $(nproc)

Where $(nproc) is the number of threads you want to use. On OSC I used $(nproc)=4 for compiling the ANITA tools and it finished in a reasonable amount of time.

Once this has finished, cd to your install directory and remove everything except the `include` and `lib64` folders:

    cd $(path_to_install_dir) #You might already be here if you never left
    rm *
    rm -r CMakeFiles

Now we need to rebuild with slightly different flags:

    cmake -DCMAKE_INSTALL_PREFIX=$(path_to_install_loc) -DBUILD_SHARED_LIBS=ON -DENABLE_OPENMP=ON -DENABLE_THREADS=ON -DENABLE_FLOAT=ON ../fftw-3.3.9
    make install -j $(nproc)

At the end of the day, your fftw install directory should have the following files:

    include/fftw3.f  
    include/fftw3.f03
    include/fftw3.h  
    include/fftw3l.f03  
    include/fftw3q.f03 
    lib64/libfftw3f.so          
    lib64/libfftw3f_threads.so.3      
    lib64/libfftw3_omp.so.3.6.9  
    lib64/libfftw3_threads.so
    lib64/libfftw3f_omp.so        
    lib64/libfftw3f.so.3        
    lib64/libfftw3f_threads.so.3.6.9  
    lib64/libfftw3.so            
    lib64/libfftw3_threads.so.3
    lib64/libfftw3f_omp.so.3      
    lib64/libfftw3f.so.3.6.9    
    lib64/libfftw3_omp.so             
    lib64/libfftw3.so.3          
    lib64/libfftw3_threads.so.3.6.9
    lib64/libfftw3f_omp.so.3.6.9  
    lib64/libfftw3f_threads.so  
    lib64/libfftw3_omp.so.3           
    lib64/libfftw3.so.3.6.9

Once fftw has been installed, export your install directory (the one with the include and lib64 folders) to the following environment variable:

    export FFTWDIR=$(path_to_install_loc)

Now you should be able to cd to your anitaBuildTools directory (or pueoBuilder directory) and run their associated build scripts:

    ./buildAnita.sh

or:

    ./pueoBuilder.sh

And hopefully your tools will magically compile (or at least, you'll get a new set of errors that are no longer related to this problem).

If you're running into an error that looks like:
        
    CMake Error: The following variables are used in this project, but they are set to NOTFOUND.
    Please set them or make sure they are set and tested correctly in the CMake files:
    FFTWF_LIB (ADVANCED)

then pueoBuilder/anitaBuildTools can't seem to find your fftw installation (or files that are supposed to be included in that installation), try rebuilding FFTW with different flags according to which files it seems to think are missing.

If it seems like pueoBuilder can't seem to find your FFTW installation at all (i.e. you're getting some error that looks like missing: FFTW_LIBRARIES or missing: FFTW_INCLUDES, check the environment variables that are supposed to point to your local FFTW installation (`$FFTWDIR`) and make sure there are the correct files in the `lib` and `include` subdirectories. 

OSC added ARA's CVMFS repository on Pitzer and Owens. This has been already done in UW's cluster thanks to Ben Hokanson-Fasig and Brian Clark. With CVMFS, all the dependencies are compiled and stored in a single folder (container), meaning that the user can just source the paths to the used environmental variables and not worry about installing them at all. This is very useful, since it usually takes a considerable amount of time to get those dependencies and properly install/debug the ARA software. To use the software, all you have to do is:

Libraries and executables are stored here, in case you want to reference those dependencies as your environmental variables: /cvmfs/ara.opensciencegrid.org/trunk/centos7/

Even if you're not in the ARA collaboration, you can benefit from this through the fact that ROOT6 is installed and compiled in the container. In order to use it you just need to run the same bash command, and ROOT 6 will be available for you to use.

Feel free to email any questions to Brian Clark or myself.

--------

Technical notes: 

The ARA software was compiled with gcc version 4.8.5. In OSC, that compiler can be loaded by doing module load gnu/4.8.5. If you're using any other compiler, you'll get a warning telling you that if you do any compilation against the ARA software, you may need to add the -D_GLIBCXX_USE_CXX11_ABI=0 flag to your Make file. 

Here is a useful link for astroparticle grad students related proposals from NASA:

https://nspires.nasaprs.com/external/solicitations/summary.do?solId=%7BE16CD59F-29DD-06C0-8971-CE1A9C252FD4%7D&path=&method=init

Full email I received regarding this information was:

_______________________________________________________________________________________________________________________________________________________________________________________________________________________________________________

ROSES-19 Amendment adds a new program element to ROSES-2019: Future Investigators in NASA Earth and Space Science and Technology (FINESST), the ROSES Graduate Student Research Program Element, E.6.

Through FINESST, the Science Mission Directorate (SMD) solicits proposals from accredited U.S. universities and other eligible organizations for graduate student-designed and performed research projects that contribute to SMD's science, technology and exploration goals.

A Notice of Intent is not requested for E.6 FINESST. Proposals to FINESST are due by February 4, 2020.

Potential proposers who wish to participate in the optional pre-proposal teleconference December 2, 2019 from 1:00-2:30 p.m. Eastern Time may (no earlier than 30 minutes prior to the start time) call 1-888-324-3185 (U.S.-only Toll Free) or 1-630-395-0272 (U.S. Toll) and use Participant Passcode: 8018549. Restrictions may prevent the use of a toll-free number from a mobile or free-phone or from telephones outside the U.S. For U.S. TTY-equipped callers or other types of relay service no earlier than 30 minutes before the start of the teleconference, call 711 and provide the same conference call number/passcode. Email HQ-FINESST@mail.nasa.gov any teleconference agenda suggestions and questions by November 25, 2019. Afterwards, questions and responses, with identifying information removed, will be posted on the NSPIRES page for FINESST under "other documents".

FINESST awards research grants with a research mentor as the principal investigator and the listed graduate student listed as the "student participant". Unlike the extinct NASA Earth and Space Science Fellowships (NESSF), the Future Investigators (FIs) are not trainees or fellows. Students with existing NESSF awards seeking a third year of funding, may not submit proposals to FINESST. Instead, they may propose to NESSF20R. Subject to a period of performance restriction, some former NESSFs may be eligible to submit proposals to FINESST.

On or about November 1, 2019, this Amendment to the NASA Research Announcement "Research Opportunities in Space and Earth Sciences (ROSES) 2019" (NNH19ZDA001N) will be posted on the NASA research opportunity homepage at http://solicitation.nasaprs.com/ROSES2019 and will appear on the RSS feed at: https://science.nasa.gov/researchers/sara/grant-solicitations/roses-2019/

Questions concerning this program element may be directed to HQ-FINESST@mail.nasa.gov.

If an undergrad has been working with the group on a volunteer basis, they will need that clarified, so that they have not up to then we working for pay without training.  There is a form that they sign saying they have been working as a volunteer.

Below is an email that Pam sent in July 2019 outlining what they need.  Other things to remember:

Undergrads receive emails from the department to remind them about orientation and scheduling prior to first day of hire, and other emails from ASC. They receive more information at orientation.

They will need to show ID.  

If an undergrad (or any hire) does not waive retirement/OPERs within 30 days, they will have to have this deducted from their paycheck. This is another reason that orientation is important.
 
For the person hiring them:
 
1.      Tell your admin (Lisa) asap who, when, etc you want to hire. ASAP is important because (Pam) will process over 75 undergraduates hires in the Fall and over 100 in the summer. Each takes 20 days on average (see below)
2.      On the first day of employment – ask your new hire if they have completed their orientation with the ASC.
 

From: Hood, Pam 
Sent: Tuesday, July 23, 2019 4:49 PM
To: 'physics-all@lists.osu.edu' <physics-all@lists.osu.edu>
Subject: Undergraduate Fall Hires
 
Hello all,
 
Fall is almost upon us and I am working on having positions ready to fill as well as posting on the OSU student job site and Federal Work study job boards. In order to plan my workflow, if you would let me know:
 
1.      Approximately how many undergraduate student assistants you plan to hire and at what rate of pay. Our department’s current average rate of pay is $10.00 - $10.50/hour, however it does depend on the position. Range is $8.55 - $14.00/hour.
 
2.      A brief summary of job duties and responsibilities (i.e. assisting in research for ______)
 
3.      Request for posting OR
 
4.      If you have specific student that has been a non-paid volunteer that you would like to hire, they need to sign a waiver prior to orientation.  Please see attached for volunteer waiver.
 
5.      Please indicate if your start date varies from August 20. And it may vary based on multiply factors i.e. signatures, workflow, etc.
 
6.      All terminations for summer ungraduated student workers OR
 
7.      If you are intending to continue employment but require a reduction in hours worked in order to comply with policy Please see attached policy for min/max hours.
 
 
I have attempted to answer all the FAQs that I noted in the “summer undergraduate hiring wave” in italics, however please let me know if you have questions. As always, it is ASC policy to not start any employee prior to orientation and it will take a minimum of 10 days from the time the contract is signed by both the student and the supervisor– please encourage your students to sign asap or it lengthens the process significantly.
 
I will do my best to facilitate this process for you. Students Buck ID# and email is extremely helpful. If you need the hire to start August 20, please send me the information via email by July 31, 2019. If you respond later or request a hire after July 31, I will estimate the hire date at the time of submittal.
 
Thanks,
 
Pam

PDF of advice by Andy Buckley (U Glasgow) on writing a HEP thesis (and presumably HEP papers too) that was forwarded by John Beacom to the CCAPP mailing list a few months back. 

In order to get PyROOT working for Python 3, you must build ROOT with a flag that specifies Python 3 in the installation.  This method will create a folder titled root-6.16.00 in your current directory, so organize things how you see fit. Then the steps are relatively simple:

wget https://root.cern/download/root_v6.16.00.source.tar.gz
tar -zxf root_v6.16.00.source.tar.gz
cd root-6.16.00
mkdir obj
cd obj
cmake .. -Dminuit2=On -Dpython3=On
make -j8

If you wish to do a different version of ROOT, the steps should be the same:

wget https://root.cern/download/root_v<version>.source.tar.gz
tar -zxf root_v<version>.source.tar.gz
cd root-<version>
mkdir obj
cd obj
cmake .. -Dminuit2=On -Dpython3=On
make -j8

Follow the instructions in the attached "getting_started_with_ara.pdf" file to download AraSim, compile it, generate results, and plot those results.

Attached is a set of slides on Getting Started with QC, a simulation monitoring project.  It has instructions on getting started in using a terminal window, and downloading, compiling and running AraSim, the simulation program for the ARA project.  AraSim has moved from the SVN repository to github, and so now you should be able to retrieve it, compile it using:

git clone https://github.com/ara-software/AraSim.git
cd AraSim
make
./AraSim

It will run without arguments, but the output might be silly. You can follow the instructions for running AraSim that are in the qc_Intro instructions, which will give them not silly results.  Those parts are still correct.

You might get some const expr errors if you are using ROOT 6, such as the ones in the first screen grab below.  As mentioned in the error messages, you need to change from const expr to not.  A few examples are shown in the next screen grab.

If you are here, you likely would also want to know how to install the prerequisites themselves. You might find this entry helpful then: http://radiorm.physics.ohio-state.edu/elog/How-To/4. It is only technically applicable to an older version that is designed for compatibility with ROOT5, but it will give you the idea.

These instructions are also superceded by an updated presentation at http://radiorm.physics.ohio-state.edu/elog/How-To/38

PBS has the option of having arrays of jobs in case you want to easily submit multiple similar jobs. The only difference in them is the array index, which you can use in your PBS script to run each task with a different set of input arguments, or any other operation that requires a unique index.

You need to add the following lines to your submitter file:

#PBS -t array_min-array_max%increment

#PBS -t 1-100%5

valgrind --suppressions=$ROOTSYS/etc/valgrind-root.supp ./myCode

If you use valgrind to identify potential memory leaks in your code, but use a lot of ROOT objects and functions, you'll notice that ROOTs TObjects trigger a lot of "potential memory leak" warnings. This option will suppress many of those. More info at https://root-forum.cern.ch/t/valgrind-and-root/2ss8506

ANITA Binned Analysis: https://github.com/osu-particle-astrophysics/BinnedAnalysis

GENETIS Bicone: https://github.com/mclowdus/BiconeEvolution

GENETIS Dipole: https://github.com/hchasan/XF-Scripts

ANITA Build tool: https://github.com/anitaNeutrino/anitaBuildTool

ANITA Hackathon: https://github.com/anitaNeutrino/hackathon2017

ICEMC: https://github.com/anitaNeutrino/icemc

Brian's Github: https://github.com/clark2668?tab=repositories

Note that you *may* need permissions for some of these. Please email Lauren (ennesser.1@buckeyemail.osu.edu ), Julie (JulieRolla@gmail.com), AND Brian (clark.2668@buckeyemail.osu.edu ) if you have any issues with permssions. Please state which GitHub links you are looking to view. 

Follow the instructions in the attached "getting_started_with_anita.pdf" file to download icemc, compile it, generate results, and plot those results.

To analyze data, we sometimes need to run over many thousands of runs at once. To do this in parallel, we can submit a job for every run we want to do. This will proceed in several steps:

1. We need to prepare an analysis program.
   1. This is demo.cxx.
   2. The program will take an input data file and an output location.
   3. The program will do some analysis on each events, and then write the result of that analysis to an output file labeled by the same number as the input file.
2. We need to prepare a job script for PBS.
   1. This is "run.sh"; this is the set of instructions to be submitted to the cluster.
   2. The instructions say to:
      1. Source a a shell environment
      2. To run the executable
      3. Move the output root file to the output location.
   3. Note that we're telling the program we wrote in step 1 to write to the node-local $TMPDIR, and then moving the result to our final output directory at the end. This is better for cluster performance.
3. We need to make a list of data files to run over
   1. We can do this on OSC by running ls -d -1 /fs/scratch/PAS0654/ara/10pct/RawData/A3/2013/sym_links/event*.root > run_list.txt
   2. This places the full path to the ROOT files in that folder into a list called run_list.txt that we can loop over.
4. Third, we need to script that will submit all of the jobs to the cluster.
   1. This is "submit_jobs.sh".
   2. This loops over all the files in our run_list.txt and submits a run.sh job for each of them.
   3. This is also where we define the $RUNDIR (where the code is to be exeucted) and the $OUTPUTDIR (where the output products are to be stored)

Once you've generated all of these output files, you can run over the output files only to make plots and such.

On ruby, users get charged for the full node, even if you aren't using all 20 cores, so it's a pain if you want to run a bunch of serial jobs. There is, however, a thing called the 'parallel command processor' (pcp) which is provided on ruby, (https://www.osc.edu/resources/available_software/software_list/parallel_command_processor) that makes it very simple.

essentially, you make a text file filled with commands, one command per line, and then you give it to the parallel command processor and it submits each line of your text file as an individual job. the nice thing about this is that you don't have to think about it. you just give it the file and go, and it will use all cores on the full node in the most efficient way possible.

below i provide 2 examples, a very simple one to show you how it works, and a more complicated one. in both files, i make the command file inside of a loop. you don't need to do this-you can make the file in some other way if you choose to. note that you can also do this from within an interactive job. more instructions at the above link.

test.pbs  is just a minimal thing, where you need to submit the same command but with some value that needs to be incremented 1000 times (e.g. 1000 different jobs).

effvol.pbs is more involved, and shows some important steps if your job produces a lot of output, where you use the $TMPDIR or the pbs workdir. (if you don't know what that is, you probably don't need to use it). each command in this file stores an output file to the $TMPDIR directory. this directory is accessed faster than the directories where you store your files, and so your jobs run faster. at the end of the script, all of the output files from all of the run jobs, are copied to my home directory, because $TMPDIR is deleted after each job. also this file shows the sourcing of a particular bash profile for submitted jobs (if you need this. some programs work differently when submitted than jobs run on the login nodes on ruby).

i recommend reading the above link for more information. the pcp is very useful on ruby!