Attached are instructions and scripts for carrying out a parameter space scan with ShelfMC, the simulation package for the ARIANNA detector.

Because some of the plotted outputs looked like colored stripes and did not offer any insight into how effective volume changed with some variables, I made some changes to the simulation and plotting scripts so that different maximum, minimum, and increment values can be chosen for each variable. Now rather than having fixed, hard-coded values for all variables, the parameter space scan and plotting is more flexible for use with variable inputs.

This document will explain how to dowload, configure, and run a parameter space search for ShelfMC on a computing cluster. 
These scripts explore the ShelfMC parameter space by varying ATTEN_UP, REFLECT_RATE, ICETHICK, FIRNDEPTH, and STATION_DEPTH for certain rages. 
The ranges and increments can be found in setup.sh. 

In order to vary STATION_DEPTH, some changes were made to the ShelfMC code. Follow these steps to allow STATION_DEPTH to be an input parameter.
1.cd to ShelfMC directory
2.Do $sed -i -e 's/ATDepth/STATION_DEPTH/g' *.cc
3.Open declaration.hh. Replace line 87 "const double ATDepth = 0.;" with "double STATION_DEPTH;"
4.In functions.cc go to line 1829. This is the ReadInput() method. Add the lines below to the end of this method. 
   GetNextNumber(inputfile, number); // new line for station Depth
   STATION_DEPTH  = (double) atof(number.c_str()); //new line
5.Do $make clean all

#######Script Descriptions########
setup.sh -> This script sets up the necessary directories and setup files for all the runs
scheduler.sh -> This script submits and monitors all jobs. 


#######DOWNLOAD########
1.Download setup.sh and scheduler.sh
2.Move both files into your ShelfMC directory
3.Do $chmod u+x setup.sh and $chmod u+x scheduler.sh

######CONFIGURE#######
1.Open setup.sh
2.On line 4, modify the job name
3.On line 6, modify group name
4.On line 11, specify your ShelfMC directory
5.On lines 15-33, specify the minimum and maximum values you wish to simulate for each variable, as well as what value to increment by
6.On line 36, modify your run name
7.On line 37, specify the NNU per run
8.On line 38, specify the starting seed
9.On line 40, specify the number of processors per node on your cluster
10.On lines 42-79, edit the input.txt parameters that you want to keep constant for every run
11.On line 80, specify the location of the LP_gain_manual.txt
12.On line 143, change walltime depending on total NNU. Remember this wall time will be 20x shorter than a single processor run.
13.On line 144, change job prefix
14.On line 146, change the group name if needed 
15.Save file
16.Open scheduler.sh
17.On line 4, specify your ShelfMC directory
18.On line 5, modify run name. Make sure it is the same runName as you have in setup.sh
19.Ensure that lines 15-33 match with setup.sh
20.On lines 62 and 66, replace cond0092 with your username for the cluster
21.On line 69, you can pick how many nodes you want to use at any given time. It is set to 6 intially. 
22.Save file 

#######RUN#######
1.Do $qsub setup.sh
2.Wait for setup.sh to finish. This script is creating the setup files for all runs. This may take about an hour.
3.When setup.sh is done, there should be a new directory in your home directory. Move this directory to your ShelfMC directory.
4.Do $screen to start a new screen that the scheduler can run on. This is incase you lose connection to the cluster mid run. 
5.Make sure that the minimum, maximum, and increment values for each variable match those in setup.sh
6.Do $./scheduler.sh to start script. This script automatically submits jobs and lets you see the status of the runs. This will run for several hours.
7.The scheduler makes a text file of all jobs called jobList.txt in the ShelfMC dir. Make sure to delete jobList.txt before starting a whole new run.


######RESULT#######
1.When Completed, there will be a great amount of data in the run files, about 460GB (This amount varies depending on how many values looped over for all variables.  
2.The run directory is organized in tree, results for particular runs can be found by cd'ing deeper into the tree.
3.In each run directory, there will be a resulting root file, all the setup files, and a log file for the run.
 

#!/bin/bash
#Jude Rajasekera 3/20/17

tmpShelfmc=$HOME/ShelfMC/git_shelfmc #location of Shelfmc
runName=e18_ParamSpaceScan #name of run




cd $tmpShelfmc #move to home directory




AttenMin=500                      # MINIMUM attenuation length for the simulation
AttenMax=1000                     # MAXIMUM attenuation length
AttenInc=100                      # INCREMENT for attenuation length

RadiusMin=3                       # MINIMUM radius
RadiusMax=31                      # MAXIMUM radius
RadiusInc=7                       # INCREMENT

IceMin=500                        # etc...
IceMax=2900
IceInc=400

FirnMin=60
FirnMax=140
FirnInc=20

StDepthMin=0
StDepthMax=200
StDepthInc=50


if [ ! -f ./jobList.txt ]; then #see if there is an existing job file
    echo "Creating new job List"
    for ((L=$AttenMin;L<=$AttenMax;L+=$AttenInc)) #Attenuation length
    do
	for ((AR=$RadiusMin;AR<=$RadiusMax;AR+=$RadiusInc)) #Station radius (measured from center of radius to antenna
	do
            for ((T=$IceMin;T<=$IceMax;T+=$IceInc)) #Ice thickness
            do
		for ((FT=$FirnMin;FT<=$FirnMax;FT+=$FirnInc)) #Firn thickness
		do
                    for ((SD=$StDepthMin;SD<=$StDepthMax;SD+=$StDepthInc)) #Station depth
                    do
		    echo "cd $runName/Atten_Up$L/AntennaRadius$AR/IceThick$T/FirnThick$FT/StationDepth$SD" >> jobList.txt
                    done
		done
            done
	done
    done
else 
    echo "Picking up from last job"
fi


numbLeft=$(wc -l < ./jobList.txt)
while [ $numbLeft -gt 0 ];
do
    jobs=$(showq | grep "cond0092") #change username here
    echo '__________Current Running Jobs__________'
    echo "$jobs"
    echo ''
    runningJobs=$(showq | grep "cond0092" | wc -l) #change username here
    echo Number of Running Jobs = $runningJobs 
    echo Number of jobs left = $numbLeft
    if [ $runningJobs -le 20 ];then
	line=$(head -n 1 jobList.txt)
	$line
	echo Submit Job && pwd
	qsub run_shelfmc_multithread.sh
	cd $tmpShelfmc
	sed -i 1d jobList.txt
    else
	echo "Full Capacity"
    fi
    sleep 1
    numbLeft=$(wc -l < ./jobList.txt)
done

------------------------------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------------------
***Summary/Instructions for colorplot.sh (followed by same for colorplot_loop.sh)
------------------------------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------------------

colorplot
------------------------------------------------------------------------------------------------------------
  ______         __                        __         __
 /  ___/  _____ |  | _____  __ ___ __ ___ |  | _____ |  |__
|  /     /  _  \|  |/  _  \|  `__/|  `_  \|  |/  _  \|   _/
|  \____|  (_)  |  |  (_)  |  |   |  |_)  |  |  (_)  |  |__
 \______|\_____/|__|\_____/|__|   |  .___/|__|\_____/ \____\
                                  |__|
------------------------------------------------------------------------------------------------------------

Interactive script - to be run after parameter space scan has been performed. Produces a single plot.


Will plot, for a user-selected ShelfMC scan, the effective volume versus any two variables of the user's
choice out of the 5 parameters we vary:
   attenuation length, antenna radius, ice thickness, firn depth, and station depth


Also will allow user to choose what values to hold the other 3 variables constant at

Will add together necessary files, create root script for plotting, and execute root script


------------------------------------------------------------------------------------------------------------


***To use colorplot.sh:

- On line 6 of colorplot.sh, set $shelfmc to your ShelfMC directory. This directory should hold the
  parameter space scan directory, and a directory called "outputs". The "outputs" directory may be empty.

- In the variable lines, on line23, line27, and line30, change 20 to the number of processors your ShelfMC runs were
  split across ($ppn in setup.sh)

- Ensure that the parameter values on lines 8-26 match those for the run you wish to plot



Make sure colorplot.sh has execute permission - if not, type 'chmod u+x colorplot.sh'

Run the script by typing './colorplot.sh'

The following instructions will also be provided in the execution of the script.
- Enter the name of the run as prompted
- Make your selections for the two variables to plot along the X- and Y- axes. Your options are:
  A - Attenuation Length
  R - Antenna Radius
  I - Ice Thickness
  F - Firn Thickness
  S - Station Depth
*****Please only use capital letters in your selection
- Select values at which to hold the remaining parameters constant. Please select only the numbers allowed
  as prompted.
- Your plot will be in the $shelfmc/outputs/[name of scan]_[x variable]_[y variable] directory


If you wish to keep the root file and/or the plotting script, comment out line 488 and/or line 489
accordingly in colorplot.sh



------------------------------------------------------------------------------------------------------------


colorplot_loop
------------------------------------------------------------------------------------------------------------

  ______         __                        __         __             __
 /  ___/  _____ |  | _____  __ ___ __ ___ |  | _____ |  |__         |  | _____   _____  __ ___
|  /     /  _  \|  |/  _  \|  `__/|  `_  \|  |/  _  \|   _/         |  |/  _  \ /  _  \|  `_  \
|  \____|  (_)  |  |  (_)  |  |   |  |_)  |  |  (_)  |  |__         |  |  (_)  |  (_)  |  |_)  |
 \______|\_____/|__|\_____/|__|   |  .___/|__|\_____/ \____\ ______ |__|\_____/ \_____/|  .___/
                                  |__|                                                 |__|
------------------------------------------------------------------------------------------------------------

A more flexible extension of colorplot!
Interactive script to be run after the parameter space scan has been performed. Produces multiple plots.


Like colorplot.sh, colorplot_loop.sh will prompt the user for two variables against which to plot the
effective volume, from the options:
   attenuation length, antenna radius, ice thickness, firn depth, and station depth


Unlike colorplot.sh, colorplot_loop.sh does not prompt for constant values of the remaining three variables,
but instead loops over all combinations of constants to produce many plots with the selected X- and Y- axes


Before using this script, please ensure that you have space for hundreds of thousands of files (some
clusters may have a file limit for each user - for instance, Ruby has a 1 million file limit per user).
The script will delete all files except the .pdf plots, but prior to this deletion there could be a few
hundred thousand new files in your system.


------------------------------------------------------------------------------------------------------------


***To use colorplot_loop.sh:

- On line 6 of colorplot.sh, set $shelfmc to your ShelfMC directory. This directory should hold the
  parameter space scan directory, and a directory called "outputs". The "outputs" directory may be empty.

- Ensure that lines 8-26 match lines 15-33 of setup.sh and scheduler.sh for the desired parameter space scan

- On lines 50, 54, and 57, change 20 to the number of processors your ShelfMC runs were
  split across ($ppn in setup.sh)



Make sure colorplot_loop.sh has execute permission - if not, type 'chmod u+x colorplot_loop.sh'

OPTIONAL: Since colorplot_loop can take several minutes to run, it may be desirable to do $screen prior to
executing the script, in case the connection to the host is interrupted.

Run the script by typing ./colorplot_loop.sh

The following instructions will also be provided in the execution of the script.
- Enter the name of the run as prompted
- Make your selections for the two variables to plot along the X- and Y- axes. Your options are:
  A - Attenuation Length
  R - Antenna Radius
  I - Ice Thickness
  F - Firn Thickness
  S - Station Depth
*****Please only use capital letters in your selection

- Your plots will be in the $shelfmc/outputs/[name of scan]_[x variable]_[y variable] directory


If you wish to keep the collected root files and/or plotting scripts, comment out line 1034
and/or line 1035 accordingly in colorplot_loop.sh




------------------------------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------------------

                                  __ -- __
                              _-`         --_
                            /             ___\
                          /         -```-``   \
                         /        /   /..\     -_ 
                         |        |   \``/       --__      
                         |         \                  - ____
                         |          |_-``--___       o |    --_
                          |           ``--____``--__   |       \
                           \              -`` ``--__`-- \       |
                             \             ``-_     \     \     |   
                               \                `_    --___ \  /
                                |                  \.        |/
                          -_    |                    \
                             -_/                      \
                                                       \
                                                        )
                                                        )
                                                        )
                                                        |
                                                       /
                                                      /
                                                    /
                                                  /
                                               -` 
                                                                

#!/bin/bash
#04/18/2017
# script will collect root files and plot as dictated by user
#Hannah Hasan

shelfmc=$HOME/ShelfMC/git_shelfmc ##set to your ShelfMC directory - should have the parameter space scan directory and a directory called "outputs". outputs may be empty

AttenMin=500
AttenMax=1000
AttenInc=100

RadiusMin=3
RadiusMax=31
RadiusInc=7

IceMin=500
IceMax=2900
IceInc=400

FirnMin=60
FirnMax=140
FirnInc=20

StDepthMin=0
StDepthMax=200
StDepthInc=50

line1='{'
line2='  TFile *f = new TFile("'"Veff_${xvar}_${yvar}.root"'");'   #set later
line3='  TTree *SimParam = (TTree*)f->Get("sim_parameters");'
line4='  TCanvas *c = new TCanvas("c", "V_{eff}", 2000, 1000);'
line5='  TPad* p = new TPad("p","p",0.0,0.0,1.0,1.0);'
line6='  p->Draw();'
line7='  p->SetLeftMargin(0.15);'
line8='  p->SetRightMargin(0.15);'
line9='  p->cd();'
line10='  TH2F *h = new TH2F("''h''", "''V_{eff} vs '"$var1"' vs '"$var2"'"'",$xbins,$xmin,$xmax"', '"$ybins,$ymin,$ymax);"   set later
line11='  double x;'
line12='  double y;'
line13='  double z;'
line14='  double veff=0;'
line15='  double energy;'
line16="  double avgs[$SIZE];"   #set later; SIZE varies depending on which variables we are plotting
line17='  SimParam->SetBranchAddress("'"$xbranch"'", &x );'   #set later
line18='  SimParam->SetBranchAddress("'"$ybranch"'", &y );'   #set later
line19='  SimParam->SetBranchAddress("Veff", &z);'
line20='  int v=0;'
line21="  for(int k=1; k<=$SIZE; k++){"   #set later for SIZE
line22='    veff = 0;'
line23='    for(v; v<(k*20.0); v++){'       #change 20 to number of processors ShelfMC runs were split across
line24='      SimParam->GetEntry(v);'
line25='      veff += z;'
line26='    }'
line27='    avgs[(k-1)] = (veff/20.0);'      #change 20 to number of processors
line28='  }'
line29="  for(int i=0; i<$SIZE; i++){"   #set later for SIZE
line30='    SimParam->GetEntry((i*20));'       #change 20 to number of processors
line31='    h->Fill(x,y,avgs[i]);'
line32='  }'
line33='  gStyle->SetPalette(1,0);'
line34='  h->GetXaxis()->SetTitle("'"$var1"' (m)");'   #set later
line35='  h->GetYaxis()->SetTitle("'"$var2"' (m)");'   #set later
line36='  h->GetZaxis()->SetTitle("V_{eff} (km^{3} str)");'
line37='  h->GetXaxis()->SetTitleOffset(1.2);'
line38='  h->GetYaxis()->SetTitleOffset(1.0);'
line39='  h->GetZaxis()->SetTitleOffset(1.3);'
line40='  gStyle->SetOptStat(0);'
line41='  h->Draw("COLZ");'
line42='  double const1;'
line43='  double const2;'
line44='  double const3;'
line45='  SimParam->SetBranchAddress("'"$const1br"'", &const1);'   #set later
line46='  SimParam->SetBranchAddress("'"$const2br"'", &const2);'   #set later
line47='  SimParam->SetBranchAddress("'"$const3br"'", &const3);'   #set later
line48='  SimParam->SetBranchAddress("exponent", &energy);'
line49='  SimParam->GetEntry(0);'
line50='  char const_1[30];'
line51='  char const_2[30];'
line52='  char const_3[30];'
line53='  char enrgy[30];'
line54='  sprintf(enrgy, "Energy (exponent): %d", energy);'
line55='  sprintf(const_1, "'"$const1br"' %d m", const1);'   
line56='  sprintf(const_2, "'"$const2br"' %d m", const2);'   
line57='  sprintf(const_3, "'"$const3br"' %d m", const3);'    #set later
line58='  TPaveText *t = new TPaveText(0,0,0.07,0.3,"brNDC");'
line59='  t->SetLabel("Simulation Parameters:");'
line60='  t->AddText(enrgy);'
line61='  t->AddText(const_1);'
line62='  t->AddText(const_2);'
line63='  t->AddText(const_3);'
line64='  t->Draw();'
line65='  c->Modified();'
line66='  c->Update();'
line67='  c->SaveAs("'"Veff_${xvar}_${yvar}.pdf"'");'   #set later
line68='}'


echo
echo
echo 'This script will allow you to choose two variables to plot effective volume against.'
echo
echo 'Enter the name of the run you wish to plot data for.'
echo
read -p 'Name of run: ' runName
echo
echo 'Thank you!'
echo
echo 'Please make your selections from the following list:'
echo '(A) - Attenuation Length'
echo '(R) - Antenna Radius'
echo '(I) - Ice Thickness'
echo '(F) - Firn Thickness'
echo '(S) - Station Depth'
echo
echo '** please use capital letters in your selection **'
echo
read -p 'Select the variable you wish to plot along the X-axis: ' xvar
read -p 'Now select the variable you wish to plot along the Y-axis: ' yvar
echo


cd $shelfmc/outputs
mkdir ${runName}_${xvar}_${yvar}

cd $shelfmc/$runName

attenprompt='Select what value at which to hold constant Attenuation Length'
atteninc=' > Choose only values between 500-1000, at increments of 100: '

radiusprompt='Select what value at which to hold constant Antenna Radius'
radiusinc=' > Choose only values between 3-31, at increments of 7: '

iceprompt='Select what value at which to hold constant Ice Thickness'
iceinc=' > Choose only values between 500-2900, at increments of 400: '

firnprompt='Select what value at which to hold constant Firn Thickness'
firninc=' > Choose only values between 60-140, at increments of 20: '

stprompt='Select what value at which to hold constant Station Depth'
stinc=' > Choose only values between 0-200, at increments of 50: '

if ([ "$xvar" = "A" ] && [ "$yvar" = "R" ]) || ([ "$yvar" = "A" ] && [ "$xvar" = "R" ]); then
     echo $iceprompt
     echo $iceinc
     read const1
    const1br='icethick'
     echo $firnprompt
     echo $firninc
     read const2
    const2br='firndepth'
     echo $stprompt
     echo $stinc
     read const3
    const3br='station_depth'
    echo
    echo 'Adding files...'
    echo
    for ((L=$AttenMin;L<=$AttenMax;L+=$AttenInc))
    do
	cd Atten_Up$L
	for ((AR=$RadiusMin;AR<=$RadiusMax;AR+=$RadiusInc))
	do
	    cd AntennaRadius$AR/IceThick$const1/FirnThick$const2/StationDepth$const3
            cp *.root $shelfmc/outputs/${runName}_${xvar}_${yvar}/Result_${L}_${AR}.root
	    cd ../../../../
	done
	cd ..
    done

elif ([ "$xvar" = "A" ] && [ "$yvar" = "I" ]) || ([ "$yvar" = "A" ] && [ "$xvar" = "I" ]); then
     echo $radiusprompt
     echo $radiusinc
     read const1
    const1br='st4_r'
     echo $firnprompt
     echo $firninc
     read const2
    const2br='firndepth'
     echo $stprompt
     echo $stinc
     read const3
    const3br='station_depth'
    echo
    for ((L=$AttenMin;L<=$AttenMax;L+=$AttenInc))
    do
        cd Atten_Up$L/AntennaRadius$const1
        for ((Ice=$IceMin;Ice<=$IceMax;Ice+=$IceInc))
        do
            cd IceThick$Ice/FirnThick$const2/StationDepth$const3
            cp *.root $shelfmc/outputs/${runName}_${xvar}_${yvar}/Result_${L}_${Ice}.root
            cd ../../../
        done
        cd ../../
    done

elif ([ "$xvar" = "A" ] && [ "$yvar" = "F" ]) || ([ "$yvar" = "A" ] && [ "$xvar" = "F" ]); then
     echo $radiusprompt
     echo $radiusinc
     read const1
    const1br='st4_r'
     echo $iceprompt
     echo $iceinc
     read const2
    const2br='icethick'
     echo $stprompt
     echo $stinc
     read const3
    const3br='station_depth'
    echo
    for ((L=$AttenMin;L<=$AttenMax;L+=$AttenInc))
    do
	cd Atten_Up$L/AntennaRadius$const1/IceThick$const2
	for ((FT=$FirnMin;FT<=$FirnMax;FT+=$FirnInc))
	do
	    cd FirnThick$FT/StationDepth$const3
	    cp *.root $shelfmc/outputs/${runName}_${xvar}_${yvar}/Result_${L}_${FT}.root
	    cd ../..
	done
	cd ../../..
    done

elif ([ "$xvar" = "A" ] && [ "$yvar" = "S" ]) || ([ "$yvar" = "A" ] && [ "$xvar" = "S" ]); then
     echo $radiusprompt
     echo $radiusinc
     read const1
    const1br='st4_r'
     echo $iceprompt
     echo $iceinc
     read const2
    const2br='icethick'
     echo $firnprompt
     echo $firninc
     read const3
    const3br='firndepth'
    echo
    for ((L=$AttenMin;L<=$AttenMax;L+=$AttenInc))
    do
	cd Atten_Up$L/AntennaRadius$const1/IceThick$const2/FirnThick$const3
	for ((SD=$StDepthMin;SD<=$StDepthMax;SD+=$StDepthInc))
	do
	    cd StationDepth$SD
	    cp *.root $shelfmc/outputs/${runName}_${xvar}_${yvar}/Result_${L}_${SD}.root
	    cd ../
	done
	cd ../../../..
    done

elif ([ "$xvar" = "R" ] && [ "$yvar" = "I" ]) || ([ "$yvar" = "R" ] && [ "$xvar" = "I" ]); then
     echo $attenprompt
     echo $atteninc
     read const1
    const1br='atten_up'
     echo $firnprompt
     echo $firninc
     read const2
    const2br='firndepth'
     echo $stprompt
     echo $stinc
     read const3
    const3br='station_depth'
    echo
    for ((AR=$RadiusMin;AR<=$RadiusMax;AR+=$RadiusInc))
    do
	cd Atten_Up$const1/AntennaRadius$AR
	for ((Ice=$IceMin;Ice<=$IceMax;Ice+=$IceInc))
	do
	    cd IceThick$Ice/FirnThick$const2/StationDepth$const3
            cp *.root $shelfmc/outputs/${runName}_${xvar}_${yvar}/Result_${AR}_${Ice}.root
	    cd ../../..
	done
	cd ../..
    done

elif ([ "$xvar" = "R" ] && [ "$yvar" = "F" ]) || ([ "$yvar" = "R" ] && [ "$xvar" = "F" ]); then
     echo $attenprompt
     echo $atteninc
     read const1
    const1br='atten_up'
     echo $iceprompt
     echo $iceinc
     read const2
    const2br='icethick'
     echo $stprompt
     echo $stinc
     read const3
    const3br='station_depth'
    echo
    for ((AR=$RadiusMin;AR<=$RadiusMax;AR+=$RadiusInc))
    do
	cd Atten_Up$const1/AntennaRadius$AR/IceThick$const2
	for ((FT=$FirnMin;FT<=$FirnMax;FT+=$FirnInc))
	do
	    cd FirnThick$FT/StationDepth$const3
            cp *.root $shelfmc/outputs/${runName}_${xvar}_${yvar}/Result_${AR}_${FT}.root
	    cd ../..
	done
	cd ../../..
    done

elif ([ "$xvar" = "R" ] && [ "$yvar" = "S" ]) || ([ "$yvar" = "R" ] && [ "$xvar" = "S" ]); then