​Today I spent some time looking at task 1 from the update here: http://radiorm.physics.ohio-state.edu/elog/GENETIS/20. Below are my updates. 

Task-- Fix error at fitness function: 

Currently, the loop stops at the fitness function calculation, and aborts (program fitnessFunction_ARA.cpp). The following are the things I have tried.

1.Run this outside of the loop: Julie and Alex did this 11/19/19. It gave a segmentation fault. UPDATE: HERE'S WHAT I DID TO CHECK THIS TODAY:

(a) Checked using cout: I used some cout commands to print sections looking for errors. It seems we were not properly giving it arguments when running it outside of the loop. Since it's running outside of the loop, these variables are not initialized. Thus, to run you need to type: ./fitnessFunction.exe $NPOP $AraOut_i.txt

• For example: ./fitnessFunction.exe 1 AraOut_1.txt

Status: The output file is verified! Running this way does properly generate our output file named fitnessScore.csv. I've double-checked this file, and it is properly being filled. At the moment it is giving a Veff of 0, thus no valid fitness score; however, this should be fixed automatically once we get real outputs from AraSim without a Veff of 0. 

 2. Check file location: See if fitnessScore.csv is being located in the correct place when the genetic algorithm initializes the next generation. 

(a) Searched bash script for the final location of fitnessScores.cvs: 

In lines 298, 299 of the bash script, it is doing the following 

cp fitnessScores.csv "$WorkingDir"/Run_Outputs/$RunName/0_fitnessScores.csv

mv fitnessScores.csv "$WorkingDir"

This is taking the .csv output generated from fitnessFunction_ARA.cpp, and is moving it to the run name directory and renaming it based on the generation -- ie first gen is 0_fitnessScores.csv-- so that data isn't overwritten. Then, for some reason, it moves the original copy -- ie fitnessScores.csv-- to the location /fs/project/PAS0654/BiconeEvolutionOSC/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop

In line 374 we then run our genetic algorithm again. 

./roulette_algorithm.exe cont $NPOP

This does not seem to take in the input file as an argument.

(b) Checked roulette_algorithm.cpp for the location it accesses fitnessScores.csv:

I checked if it was hard-coded into the roulette_algorithm.cpp script. I found the following lines.

 // Now we need to read the fitness scores:                                                                                                                                                    

        ifstream fitnessScores;

        fitnessScores.open("fitnessScores.csv");

        string fitnessRead[NPOP+2];

        for(int i=0;i<(NPOP+2);i++)

    {

                getline(fitnessScores,fitnessRead[i]);

                if(i>=2)

                {

                        fitness[i-2] = atof(fitnessRead[i].c_str());

                        if (fitness[i-2]<0)

                        {

                                fitness[i-2] = 0; // If the fitness score is less than 0, we set it to 0 to not throw things off.                                                                     

                        }

                }

    }

        fitnessScores.close();

}

This means that our script roulette_algorithm.cpp is grabbing the fitnessScores.csv from its current directory-- which is accurate due to the line (mv fitnessScores.csv "$WorkingDir") referenced above in the bash script. Something we could verify here is to make sure this function is properly working by having it cout elements from the fitnessScores.csv within this function and to run the roulette algorithm separately. We could do this now, however, we don't have real outputs in fitnessScores.csv anyway, since we are getting a veff=0 for AraSim temporarily. 

In the meantime, I've added the following line after section E (the section that runs fitnessFunction_ARA.cpp, and generates/moves/copies fitnessScore.csv):

echo 'Congrats on getting a fitness score!'

This should check to be sure that it runs until after this point. I've also added the following line after section E (section that plots the fitness score versus gen):

echo 'Congrats on getting some nice plots!'

This should check to be sure that it runs until after this point. Note that I have added these lines to both our original bash script, and the one Alex M. is editing to turn every section into a function. I have added these lines to the .sh scripts containing the functions to be sure the function scripts for these tasks ran completely.  Once we have real AraSim outputs, we should be able to create some cout lines in the instructions highlighted in yellow above. 

3. Look at how it's calculating the fitness score.

I have not checked this yet. However, I do not think it is likely this is the issue. I will check later though. 

Attendance for today's session: Cade, Julie, Alex M, Mitchell, Scott. 

Tasks/to-do list:

1. ​Get loop to run up to the fitness function.
   • Right now we have the bash script setup to run AraSim by submitting them as parallel jobs. Today Cade added the "wait" function within the job in order to make sure the loop doesn't continue to the next step before the Arasim data is ready. Cade will give us an update on Thursday letting us know whether or not this completed or gave an error. 
2. Fix error at fitness function. 
   • Currently, the loop stops at the fitness function calculation, and aborts (program fitnessFunction_ARA.cpp). We are unsure of why at the moment. Here are the things we are looking into. 
     • Run this outside of the loop-- ie compile the fitnessFunction_ARA.cpp (g++ -std=c++11 fitnessFunction_ARA.cpp -o fitnessFunction.exe), and run the executable(./fitnessFunction.exe). What happens?
       • Julie and Alex did this. It gave a segmentation fault. We need to look into this further. 
       • We double-checked that this .exe was properly created from compiling, and it was.
     • Double-check our AraSim output by checking:
       • Format
       • File type
     • Is there an error past the fitness function, and it's killing the loop after this runs?
       • Is his output not properly going into the gen alg?
       • Are the output files properly:
         • Filled
         • Formatted
         • Put in the right place
     • Look at how it's calculating the fitness score.

Since it seems like Alex and I may have found a problem at our first "test", this could be the issue. We will investigate it further. If (1) Cade's "wait" function works, and (2) we can fix this issue with the loop stopping at the fitness score calculation, ideally this means that the loop is then running. 

Today we also found that it is better to run on the terminal through an interactive job than a virtual desktop. This is MUCH faster -- ie we don't have to wait nearly as long as we do to get a node via a virtual desktop. This means that everyone with a PC needs to either (1) work out their issues with X11 forwarding crashing (Mitchell succeeded at this today by making sure Xquartz was set up to work via ssh to Pitzer in the settings), or (2) dual boot Linux. It seems that Mac users are not having issues, but that some of the Windows users are having their OS completely crash when XF GUI tries to forward. Users can test whether or not their X11 forwarding is working by doing the following:

1.  Logging onto Pitzer
2. Typing "module load xfdtd"
3. Then typing "xfdtd"

This should successfully open the GUI for XF. If it crashes, users may have to do one of the things I mentioned above. Once we do succeed at having this setup, you can start an interactive job via the command: 

qsub -I -X -l walltime=1:00:00 -l nodes=1:ppn=10

You then have to wait until the node is free. This has taken about 1/3 of the time to get than when requesting a virtual desktop. Note that, however, we may need a higher ppn (maybe something like 40). This is not totally clear yet. 

Attendance for today's session: Cade, Julie, Alex M, Mitchell, Scott. 

Tasks worked on:

• Continue to fix continuity of loop: Cade and Julie worked on fixing the following errors:
  1. In section D of the loop, it currently
     • Moves the output from the python conversion to $AraSimExec directory. 
     • Renames it as a .txt file (note the $i is renaming each in line with $NPOP). 

for i in `seq 1 $NPOP`

do

      mv evol_antenna_model_$i.dat $AraSimExec

done

for i in `seq 1 $NPOP`

do

      cp evol_antenna_model_$i.dat a_$i.txt

done

However, this wasn't working and it was only properly renaming individual 1. To check this,

• We confirmed it is properly iterating the first loop by using the "echo" command. The second loop was not properly iterating, however.
• We were able to fix it by combining these steps. This was confirmed to have worked. 

for i in `seq 1 $NPOP`

do

      mv evol_antenna_model_$i.dat $AraSimExec/a_$i.txt

done

              2. We got this iteration $i to work in the job script -- ie as an input file variable. The loop now successfully submits AraSim as a job with the proper input files. 

• Clean bash script
  • Mitchell and Alex M worked again today on turning every function possible into functions (and putting them in their own .sh scripts that are being called by XF_Loop.sh). 
    • The loop runs the first generation "manually" -- ie each step was spelled out, and nothing is put into a loop. The second+ generations then run as a loop. This is because we need to initialize the first XF run -- which then has no problems in the later iterations.
      • Alex M. and Mitchell are now turning each section A-F into individual functions and put into their own .sh scripts. Thus section A would just be calling another .sh script. In gen 2+ we then loop over these SectionA.sh through SectionF.sh functions. All of these just got completed. He just needs to add the change Julie and Cade made today, and clean some things up. 
        • For example, he has an "if else" statement that tells it whether or not it's the first (XF initializes) generation, or in generation 2+. Cade has a few changes he would like Alex to make to clean up this part of the run. 
    • Idea: The loop used to run the first generation "manually" -- ie each step was spelled out, and nothing was put into a loop. The second+ generations then ran as a loop. Since we were able to get XF to import the xmacro scripts automatically -- as well as save the project with the correct name (matching the expected name initialize in the bash script)-- they've gotten rid of that first "manual" iteration, and are now fully running all generations as a loop. We haven't tested whether or not we can make this change yet, but it is worth looking into now that some changes have been successfully made with XF. 

Quick note from Julie:

I tried to run from the terminal once Cade left. I get the following output/error that we should make sure isn't happening for other users (since most of this is a permissions issue that I thought we fixed. I can look into the other errors and see if anyone else gets them). For whatever reason, Cade is not having the same output errors; he is able to run the same copy just fine. 

Roulette algorithm initialized.

./XF_Loop.sh: line 136: output.xmacro: Permission denied

./XF_Loop.sh: line 137: output.xmacro: Permission denied

./XF_Loop.sh: line 138: output.xmacro: Permission denied

./XF_Loop.sh: line 139: output.xmacro: Permission denied

./XF_Loop.sh: line 140: output.xmacro: Permission denied

./XF_Loop.sh: line 146: simulation_PEC.xmacro: Permission denied

./XF_Loop.sh: line 147: simulation_PEC.xmacro: Permission denied

./XF_Loop.sh: line 149: simulation_PEC.xmacro: Permission denied

./XF_Loop.sh: line 158: simulation_PEC.xmacro: Permission denied

 

Opening XF user interface...

*** Please remember to save the project with the same name as RunName! ***

 

1. Import and run simulation_PEC.xmacro

2. Import and run output.xmacro

3. Close XF

Lmod has detected the following error:  The following module(s) are unknown: "xfdtd"

 

Please check the spelling or version number. Also try "module spider ..."

It is also possible your cache file is out-of-date; it may help to try:

  $ module --ignore-cache load "xfdtd"

Also make sure that all modulefiles written in TCL start with the string #%Module

./XF_Loop.sh: line 172: xfdtd: command not found

./XF_Loop.sh: line 177: cd: /fs/project/PAS0654/BiconeEvolutionOSC/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop/Run_Outputs/Julie/Julie.xf/Simulations/000001/Run0001/: No such file or directory

./XF_Loop.sh: line 178: xfsolver: command not found

./XF_Loop.sh: line 177: cd: /fs/project/PAS0654/BiconeEvolutionOSC/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop/Run_Outputs/Julie/Julie.xf/Simulations/000002/Run0001/: No such file or directory

./XF_Loop.sh: line 178: xfsolver: command not found

./XF_Loop.sh: line 183: xfdtd: command not found

chmod: changing permissions of ‘/fs/project/PAS0654/BiconeEvolutionOSC/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop/AraSimTestJob.sh’: Operation not permitted

chmod: changing permissions of ‘/fs/project/PAS0654/BiconeEvolutionOSC/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop/Antenna_Performance_Metric/AraOut_%num%.txt’: Operation not permitted

chmod: changing permissions of ‘/fs/project/PAS0654/BiconeEvolutionOSC/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop/Loop_Functions/XF_Loop3.sh’: Operation not permitted

chmod: changing permissions of ‘/fs/project/PAS0654/BiconeEvolutionOSC/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop/Loop_Functions/#Part_C.sh#’: Operation not permitted

chmod: changing permissions of ‘/fs/project/PAS0654/BiconeEvolutionOSC/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop/Loop_Functions/#Part_A.sh#’: Operation not permitted

chmod: changing permissions of ‘/fs/project/PAS0654/BiconeEvolutionOSC/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop/scriptEOFiles/AraSimCall.sh.e957233’: Operation not permitted

chmod: changing permissions of ‘/fs/project/PAS0654/BiconeEvolutionOSC/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop/scriptEOFiles/AraSimCall.sh.e957230’: Operation not permitted

chmod: changing permissions of ‘/fs/project/PAS0654/BiconeEvolutionOSC/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop/scriptEOFiles/AraSimCall.sh.e957234’: Operation not permitted

chmod: changing permissions of ‘/fs/project/PAS0654/BiconeEvolutionOSC/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop/scriptEOFiles/AraSimCall.sh.o957224’: Operation not permitted

chmod: changing permissions of ‘/fs/project/PAS0654/BiconeEvolutionOSC/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop/scriptEOFiles/AraSimCall.sh.e957220’: Operation not permitted

chmod: changing permissions of ‘/fs/project/PAS0654/BiconeEvolutionOSC/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop/scriptEOFiles/AraSimCall.sh.o957220’: Operation not permitted

chmod: changing permissions of ‘/fs/project/PAS0654/BiconeEvolutionOSC/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop/scriptEOFiles/AraSimCall.sh.o957234’: Operation not permitted

chmod: changing permissions of ‘/fs/project/PAS0654/BiconeEvolutionOSC/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop/scriptEOFiles/AraSimCall.sh.o957227’: Operation not permitted

chmod: changing permissions of ‘/fs/project/PAS0654/BiconeEvolutionOSC/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop/scriptEOFiles/AraSimCall.sh.o957233’: Operation not permitted

chmod: changing permissions of ‘/fs/project/PAS0654/BiconeEvolutionOSC/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop/scriptEOFiles/AraSimCall.sh.o957218’: Operation not permitted

chmod: changing permissions of ‘/fs/project/PAS0654/BiconeEvolutionOSC/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop/scriptEOFiles/AraSimCall.sh.e957224’: Operation not permitted

chmod: changing permissions of ‘/fs/project/PAS0654/BiconeEvolutionOSC/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop/scriptEOFiles/AraSimCall.sh.e957231’: Operation not permitted

chmod: changing permissions of ‘/fs/project/PAS0654/BiconeEvolutionOSC/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop/scriptEOFiles/AraSimCall.sh.e957226’: Operation not permitted

chmod: changing permissions of ‘/fs/project/PAS0654/BiconeEvolutionOSC/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop/scriptEOFiles/AraSimCall.sh.o957221’: Operation not permitted

chmod: changing permissions of ‘/fs/project/PAS0654/BiconeEvolutionOSC/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop/scriptEOFiles/AraSimCall.sh.o957223’: Operation not permitted

chmod: changing permissions of ‘/fs/project/PAS0654/BiconeEvolutionOSC/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop/scriptEOFiles/AraSimCall.sh.e957217’: Operation not permitted

chmod: changing permissions of ‘/fs/project/PAS0654/BiconeEvolutionOSC/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop/scriptEOFiles/AraSimCall.sh.e957227’: Operation not permitted

chmod: changing permissions of ‘/fs/project/PAS0654/BiconeEvolutionOSC/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop/scriptEOFiles/AraSimCall.sh.e957218’: Operation not permitted

chmod: changing permissions of ‘/fs/project/PAS0654/BiconeEvolutionOSC/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop/scriptEOFiles/AraSimCall.sh.e957223’: Operation not permitted

chmod: changing permissions of ‘/fs/project/PAS0654/BiconeEvolutionOSC/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop/scriptEOFiles/AraSimCall.sh.e957225’: Operation not permitted

chmod: changing permissions of ‘/fs/project/PAS0654/BiconeEvolutionOSC/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop/scriptEOFiles/AraSimCall.sh.o957230’: Operation not permitted

chmod: changing permissions of ‘/fs/project/PAS0654/BiconeEvolutionOSC/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop/scriptEOFiles/AraSimCall.sh.o957225’: Operation not permitted

chmod: changing permissions of ‘/fs/project/PAS0654/BiconeEvolutionOSC/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop/scriptEOFiles/AraSimCall.sh.e957222’: Operation not permitted

chmod: changing permissions of ‘/fs/project/PAS0654/BiconeEvolutionOSC/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop/scriptEOFiles/AraSimCall.sh.o957231’: Operation not permitted

chmod: changing permissions of ‘/fs/project/PAS0654/BiconeEvolutionOSC/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop/scriptEOFiles/AraSimCall.sh.o957217’: Operation not permitted

chmod: changing permissions of ‘/fs/project/PAS0654/BiconeEvolutionOSC/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop/scriptEOFiles/AraSimCall.sh.o957222’: Operation not permitted

chmod: changing permissions of ‘/fs/project/PAS0654/BiconeEvolutionOSC/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop/scriptEOFiles/AraSimCall.sh.o957226’: Operation not permitted

chmod: changing permissions of ‘/fs/project/PAS0654/BiconeEvolutionOSC/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop/scriptEOFiles/AraSimCall.sh.e957221’: Operation not permitted

 

Resuming...

 

chmod: changing permissions of ‘/fs/project/PAS0654/BiconeEvolutionOSC/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop/AraSimTestJob.sh’: Operation not permitted

chmod: changing permissions of ‘/fs/project/PAS0654/BiconeEvolutionOSC/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop/Antenna_Performance_Metric/AraOut_%num%.txt’: Operation not permitted

chmod: changing permissions of ‘/fs/project/PAS0654/BiconeEvolutionOSC/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop/Loop_Functions/XF_Loop3.sh’: Operation not permitted

chmod: changing permissions of ‘/fs/project/PAS0654/BiconeEvolutionOSC/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop/Loop_Functions/#Part_C.sh#’: Operation not permitted

chmod: changing permissions of ‘/fs/project/PAS0654/BiconeEvolutionOSC/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop/Loop_Functions/#Part_A.sh#’: Operation not permitted

chmod: changing permissions of ‘/fs/project/PAS0654/BiconeEvolutionOSC/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop/scriptEOFiles/AraSimCall.sh.e957233’: Operation not permitted

chmod: changing permissions of ‘/fs/project/PAS0654/BiconeEvolutionOSC/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop/scriptEOFiles/AraSimCall.sh.e957230’: Operation not permitted

chmod: changing permissions of ‘/fs/project/PAS0654/BiconeEvolutionOSC/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop/scriptEOFiles/AraSimCall.sh.e957234’: Operation not permitted

chmod: changing permissions of ‘/fs/project/PAS0654/BiconeEvolutionOSC/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop/scriptEOFiles/AraSimCall.sh.o957224’: Operation not permitted

chmod: changing permissions of ‘/fs/project/PAS0654/BiconeEvolutionOSC/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop/scriptEOFiles/AraSimCall.sh.e957220’: Operation not permitted

chmod: changing permissions of ‘/fs/project/PAS0654/BiconeEvolutionOSC/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop/scriptEOFiles/AraSimCall.sh.o957220’: Operation not permitted

chmod: changing permissions of ‘/fs/project/PAS0654/BiconeEvolutionOSC/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop/scriptEOFiles/AraSimCall.sh.o957234’: Operation not permitted

chmod: changing permissions of ‘/fs/project/PAS0654/BiconeEvolutionOSC/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop/scriptEOFiles/AraSimCall.sh.o957227’: Operation not permitted

chmod: changing permissions of ‘/fs/project/PAS0654/BiconeEvolutionOSC/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop/scriptEOFiles/AraSimCall.sh.o957233’: Operation not permitted

chmod: changing permissions of ‘/fs/project/PAS0654/BiconeEvolutionOSC/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop/scriptEOFiles/AraSimCall.sh.o957218’: Operation not permitted

chmod: changing permissions of ‘/fs/project/PAS0654/BiconeEvolutionOSC/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop/scriptEOFiles/AraSimCall.sh.e957224’: Operation not permitted

chmod: changing permissions of ‘/fs/project/PAS0654/BiconeEvolutionOSC/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop/scriptEOFiles/AraSimCall.sh.e957231’: Operation not permitted

chmod: changing permissions of ‘/fs/project/PAS0654/BiconeEvolutionOSC/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop/scriptEOFiles/AraSimCall.sh.e957226’: Operation not permitted

chmod: changing permissions of ‘/fs/project/PAS0654/BiconeEvolutionOSC/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop/scriptEOFiles/AraSimCall.sh.o957221’: Operation not permitted

chmod: changing permissions of ‘/fs/project/PAS0654/BiconeEvolutionOSC/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop/scriptEOFiles/AraSimCall.sh.o957223’: Operation not permitted

chmod: changing permissions of ‘/fs/project/PAS0654/BiconeEvolutionOSC/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop/scriptEOFiles/AraSimCall.sh.e957217’: Operation not permitted

chmod: changing permissions of ‘/fs/project/PAS0654/BiconeEvolutionOSC/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop/scriptEOFiles/AraSimCall.sh.e957227’: Operation not permitted

chmod: changing permissions of ‘/fs/project/PAS0654/BiconeEvolutionOSC/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop/scriptEOFiles/AraSimCall.sh.e957218’: Operation not permitted

chmod: changing permissions of ‘/fs/project/PAS0654/BiconeEvolutionOSC/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop/scriptEOFiles/AraSimCall.sh.e957223’: Operation not permitted

chmod: changing permissions of ‘/fs/project/PAS0654/BiconeEvolutionOSC/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop/scriptEOFiles/AraSimCall.sh.e957225’: Operation not permitted

chmod: changing permissions of ‘/fs/project/PAS0654/BiconeEvolutionOSC/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop/scriptEOFiles/AraSimCall.sh.o957230’: Operation not permitted

chmod: changing permissions of ‘/fs/project/PAS0654/BiconeEvolutionOSC/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop/scriptEOFiles/AraSimCall.sh.o957225’: Operation not permitted

chmod: changing permissions of ‘/fs/project/PAS0654/BiconeEvolutionOSC/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop/scriptEOFiles/AraSimCall.sh.e957222’: Operation not permitted

chmod: changing permissions of ‘/fs/project/PAS0654/BiconeEvolutionOSC/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop/scriptEOFiles/AraSimCall.sh.o957231’: Operation not permitted

chmod: changing permissions of ‘/fs/project/PAS0654/BiconeEvolutionOSC/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop/scriptEOFiles/AraSimCall.sh.o957217’: Operation not permitted

chmod: changing permissions of ‘/fs/project/PAS0654/BiconeEvolutionOSC/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop/scriptEOFiles/AraSimCall.sh.o957222’: Operation not permitted

chmod: changing permissions of ‘/fs/project/PAS0654/BiconeEvolutionOSC/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop/scriptEOFiles/AraSimCall.sh.o957226’: Operation not permitted

chmod: changing permissions of ‘/fs/project/PAS0654/BiconeEvolutionOSC/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop/scriptEOFiles/AraSimCall.sh.e957221’: Operation not permitted

Resuming...

NOTE:  You may want to change the following in your job for Owens

 

    nodes=1:ppn=1 defaulting to mem=4571MB

 

Please review the Owens documentation:

    https://www.osc.edu/supercomputing/computing/owens

 

Feel free to contact oschelp@osc.edu if you have any questions.

 

8485416.owens-batch.ten.osc.edu

rm: cannot remove ‘outputs/*.root’: No such file or directory

NOTE:  You may want to change the following in your job for Owens

 

    nodes=1:ppn=1 defaulting to mem=4571MB

 

Please review the Owens documentation:

    https://www.osc.edu/supercomputing/computing/owens

 

Feel free to contact oschelp@osc.edu if you have any questions.

 

8485417.owens-batch.ten.osc.edu

rm: cannot remove ‘outputs/*.root’: No such file or directory

chmod: changing permissions of ‘/fs/project/PAS0654/BiconeEvolutionOSC/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop/AraSimTestJob.sh’: Operation not permitted

chmod: changing permissions of ‘/fs/project/PAS0654/BiconeEvolutionOSC/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop/Antenna_Performance_Metric/AraOut_%num%.txt’: Operation not permitted

chmod: changing permissions of ‘/fs/project/PAS0654/BiconeEvolutionOSC/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop/Loop_Functions/XF_Loop3.sh’: Operation not permitted

chmod: changing permissions of ‘/fs/project/PAS0654/BiconeEvolutionOSC/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop/Loop_Functions/#Part_C.sh#’: Operation not permitted

chmod: changing permissions of ‘/fs/project/PAS0654/BiconeEvolutionOSC/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop/Loop_Functions/#Part_A.sh#’: Operation not permitted

chmod: changing permissions of ‘/fs/project/PAS0654/BiconeEvolutionOSC/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop/scriptEOFiles/AraSimCall.sh.e957233’: Operation not permitted

chmod: changing permissions of ‘/fs/project/PAS0654/BiconeEvolutionOSC/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop/scriptEOFiles/AraSimCall.sh.e957230’: Operation not permitted

chmod: changing permissions of ‘/fs/project/PAS0654/BiconeEvolutionOSC/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop/scriptEOFiles/AraSimCall.sh.e957234’: Operation not permitted

chmod: changing permissions of ‘/fs/project/PAS0654/BiconeEvolutionOSC/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop/scriptEOFiles/AraSimCall.sh.o957224’: Operation not permitted

chmod: changing permissions of ‘/fs/project/PAS0654/BiconeEvolutionOSC/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop/scriptEOFiles/AraSimCall.sh.e957220’: Operation not permitted

chmod: changing permissions of ‘/fs/project/PAS0654/BiconeEvolutionOSC/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop/scriptEOFiles/AraSimCall.sh.o957220’: Operation not permitted

chmod: changing permissions of ‘/fs/project/PAS0654/BiconeEvolutionOSC/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop/scriptEOFiles/AraSimCall.sh.o957234’: Operation not permitted

chmod: changing permissions of ‘/fs/project/PAS0654/BiconeEvolutionOSC/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop/scriptEOFiles/AraSimCall.sh.o957227’: Operation not permitted

chmod: changing permissions of ‘/fs/project/PAS0654/BiconeEvolutionOSC/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop/scriptEOFiles/AraSimCall.sh.o957233’: Operation not permitted

chmod: changing permissions of ‘/fs/project/PAS0654/BiconeEvolutionOSC/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop/scriptEOFiles/AraSimCall.sh.o957218’: Operation not permitted

chmod: changing permissions of ‘/fs/project/PAS0654/BiconeEvolutionOSC/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop/scriptEOFiles/AraSimCall.sh.e957224’: Operation not permitted

chmod: changing permissions of ‘/fs/project/PAS0654/BiconeEvolutionOSC/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop/scriptEOFiles/AraSimCall.sh.e957231’: Operation not permitted

chmod: changing permissions of ‘/fs/project/PAS0654/BiconeEvolutionOSC/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop/scriptEOFiles/AraSimCall.sh.e957226’: Operation not permitted

chmod: changing permissions of ‘/fs/project/PAS0654/BiconeEvolutionOSC/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop/scriptEOFiles/AraSimCall.sh.o957221’: Operation not permitted

chmod: changing permissions of ‘/fs/project/PAS0654/BiconeEvolutionOSC/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop/scriptEOFiles/AraSimCall.sh.o957223’: Operation not permitted

chmod: changing permissions of ‘/fs/project/PAS0654/BiconeEvolutionOSC/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop/scriptEOFiles/AraSimCall.sh.e957217’: Operation not permitted

chmod: changing permissions of ‘/fs/project/PAS0654/BiconeEvolutionOSC/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop/scriptEOFiles/AraSimCall.sh.e957227’: Operation not permitted

chmod: changing permissions of ‘/fs/project/PAS0654/BiconeEvolutionOSC/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop/scriptEOFiles/AraSimCall.sh.e957218’: Operation not permitted

chmod: changing permissions of ‘/fs/project/PAS0654/BiconeEvolutionOSC/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop/scriptEOFiles/AraSimCall.sh.e957223’: Operation not permitted

chmod: changing permissions of ‘/fs/project/PAS0654/BiconeEvolutionOSC/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop/scriptEOFiles/AraSimCall.sh.e957225’: Operation not permitted

chmod: changing permissions of ‘/fs/project/PAS0654/BiconeEvolutionOSC/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop/scriptEOFiles/AraSimCall.sh.o957230’: Operation not permitted

chmod: changing permissions of ‘/fs/project/PAS0654/BiconeEvolutionOSC/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop/scriptEOFiles/AraSimCall.sh.o957225’: Operation not permitted

chmod: changing permissions of ‘/fs/project/PAS0654/BiconeEvolutionOSC/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop/scriptEOFiles/AraSimCall.sh.e957222’: Operation not permitted

chmod: changing permissions of ‘/fs/project/PAS0654/BiconeEvolutionOSC/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop/scriptEOFiles/AraSimCall.sh.o957231’: Operation not permitted

chmod: changing permissions of ‘/fs/project/PAS0654/BiconeEvolutionOSC/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop/scriptEOFiles/AraSimCall.sh.o957217’: Operation not permitted

chmod: changing permissions of ‘/fs/project/PAS0654/BiconeEvolutionOSC/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop/scriptEOFiles/AraSimCall.sh.o957222’: Operation not permitted

chmod: changing permissions of ‘/fs/project/PAS0654/BiconeEvolutionOSC/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop/scriptEOFiles/AraSimCall.sh.o957226’: Operation not permitted

chmod: changing permissions of ‘/fs/project/PAS0654/BiconeEvolutionOSC/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop/scriptEOFiles/AraSimCall.sh.e957221’: Operation not permitted

mv: cannot stat ‘*.root’: No such file or directory

Fitness function initialized.

0 

Data successfully read. Data successfully written.

Attendance for today's session: Cade, Julie, Alex M, Alex P, Mitchell, Scott. 

Monday was a holiday, and we will be replaceing that meeting later this week. Today we worked on the following tasks:

Tasks:

• Check continuity of loop
  • Alex M will try another run tonight, and has already requested a virtual desktop. 
• Install Arasim to /fs/projects/PAS0654/BiconeEvolutionOCS to be edited and messed with
  • Cade is testing whether or not this is running within the loop. He had gotten a few errors, and it is not completing. It's submitting the job with the new bash script, but it is not actually running.
    • Found an error. The bash script was not grabbing the correct AraSim directory. More updates to come. 
• Clean bash script
  • Mitchell and Alex M are putting all current functions hard-coded in into their own .sh scrips that are now being called in the main bash script.
• Update Manual
  • Cade and I are updating the manual with new changes. We have officially retired the old version of the manual that was used for runs with Nutau. We have saved that version and uploaded it to the elog here: http://radiorm.physics.ohio-state.edu/elog/GENETIS/17

Right now Elliot and Leo have decided to focus on working with Keith. They will not be contributing to this project for the remainder of the semester. Students have been posting individual updates here: https://www.dropbox.com/home/GP_Antennas/Updates

I've also attached the meeting minutes from the call last week, in order to keep track of updates. 

We are off again with a new team. The new team consists of Alex Machtay, Alex Patton, Cade Sbrocco, Eliot Ferstl, Evelyn Shank, Mitchell Halley, Scott Janse. A spreadsheet of tasks and deadlines can be found here: https://docs.google.com/spreadsheets/d/1CkcdbediFbripdlwPw6UmhAHdAl5IdkL2efaseJkTTM/edit#gid=0

Updates:

We are now running on OSC. We have acquired a version of XF on OSC and have now transferred the loop to OSC. All of the xmacros for XF should be functioning, and we have managed to make the bash script auto import the xmacro scripts to XF when it boots. The loop exists on /fs/projects/PAS0654/BiconeEvolutionOCS. This is a shared copy so that everyone can run off of the same installation. This is important so that we can make sure everyone is pushing edits regularly, and so that everyone is always running with the most up-to-date edits. 

The loop had not yet been tested for it's current status. Alex M. is testing it and we are waiting to hear back. 

Tasks:

• Check continuity of loop
  • Update: Alex M is testing this
• Install Arasim to /fs/projects/PAS0654/BiconeEvolutionOCS to be edited and messed with
  • Update: this has been completed as of 11/5/19
• Test Alex M's code for editing Arasim's Setup.txt variables
• Clean bash script
  • Update: Mitchell is working on this.
• Make version of bash script which writes .job scripts for Arasim
  • This must write a different .job for each individual so it parallelizes Arasim running for each individual
  • The number of .jobs written must depend on $NPOP
    • ie this bash script should be writing .jobs from scratch each time as (1) the number of them we need depends on $NPOP, (2) the names of the input and output files depends on $NPOP. 
      • It should also delete .jobs at the end -- ie when all generations have completed so that we don't have a random number of .jobs floating around. For example, say we run with 10 individuals one time, and it writes 10 .jobs ($NPOP=10), and then the next run we only have $NPOP=5, we will have 5 extra .jobs floating around not being used unless we clean them all at the end line of the run in the bash script. 
  • Update: Work on this has begun by Alex M, Cade, and Evelyn (Alex P has also been messing with Arasim).
• Make XF be submitted as a job
  • Cade has ideas for this.

Students have been posting individual updates here: https://www.dropbox.com/home/GP_Antennas/Updates

The following are links to the 2018 and 2018 APS slides.

2018 APS talk:

https://www.dropbox.com/home/GP_Antennas/APS%20Talk?preview=April+APS+Talk_final.pdf

2019 APS talk:

https://absuploads.aps.org/presentation.cfm?pid=15183

We ran valgrind (a profiler) on AraSim to find out where it was spending most of its time, we ran it with callgrind to give an output file of the time and opened that file with qcashegrind to display it in a legible format. This let us know that the function Param_RE_Tterm_approx was taking up about 75% of the time because it was being called hundreds of millions of times. Now more specifically the standard math functions pow() and exp() were being called in this function every time and took up about 44% and 20% respectively. They were called about the same number of times as Param_RE_Tterm_approx but they are only called in the else block in an if-else statement.

JULY 31st Update:

I used the C++ library future and set up iterations of a loop to run asynchronously as they do not rely on each other's calculations. This launches a seperate thread whenever I call one. The struggle with this is that launching a thread and getting its results takes time as well, so there had to be enough calulations in a thread in order for it to make up any time. The purpose of the function GetVm_FarField_Tarray is to create arrays of doubles called Earray and Tarray and those are the only values that get altered in this function. Tarray is created with simple calculations so I left that out of threads but making Earray calls the other function that takes up so much time.

I edited the files signal.cc and signal.hh to work with 4 threads and ran 10,000 neutrinos with 4 cores. The cpu time was 1:48:41 but the real walltime it used only took 00:54:40. The time these tests for 10,000 neutrinos with base AraSim normally take anywhere from 1:35:00 to 1:45:00 or in that range. It should still work with less threads available it would just run slower and probably at the cpu time, which might be slightly slower than base AraSim but as soon as more threads are available it runs much faster as we see here. I still need to check for the accuracy of these edits, the number passed is in a reasonable range, but I will need to investigate further by giving the exact same neutrinos to both programs and making sure their outputs are identical.

I also have some further ideas to speed it up even more. Currently every iteration of the loop calls a thread but what would be faster would be to either

A: Set it up so it can call more threads at a time and thus go faster, which would be an easy edit but only work if more cores are given

B: Set it up so the current threads run more than one iteration at a time, I've started experimenting with this but having trouble making sure that everything involving pointers stays safe, but the benefit of this is that the threads would get called and joined significantly less and would cause a great speed increase without adding more threads.

All these developments are new after me spending a while trying many different ways to speed it up or run parts in parallel like this, so even if this isn't accurate or valid I believe I would be able to make it so with more time and work.

Attached I have a zip that contains the two files I edited as well as an output file from my first test. I will continue to test and improve this but am unable to work on this again until August 19th so I wanted to share my current status before I am unavailable.

Right now the following people are working on this project:

Julie Rolla, Cade Sbrocco, Scott Janse, Alex Patton, Eliot Ferstl, Abassi Brown, Evelyn Shank, Sophie Dunlap

Today we met for our first working meeting. Scott, Alex, and Eliot were asked to speed up and optimize AraSim. Sophie and Evelyn are working on editing the .csv file with the beam patterns coming out of XF. Currently it's written in a different format than AraSim takes in. Abassi is helping Evelyn and Sophie by giving them a sample program helping reorganize the cells in a .csv file. Since there's confusion on how to optimize AraSim, we've started dissecting the software so we can understand how it works. The following are questions on the setup.txt document. 

Setup.txt

What is "timestep"? Could we increase the timestep? Is this timestep big or small compared to 2 GHz?

Answer: Timestep is the unit of discrete time the computer operates at. The timestep by default is set to 2 GHz, or 5 * 10^-10 seconds. We might be able to make up some cheap time by lowering that but would have to check if it makes the data less accurate.

What is "TRIG_WINDOW"? Is this a "coincidence"? Meaning if this signal is picked up within a certain time between stations it's considered the same event?

What is "V_MIMIC_MODE"? Comments in Setup.CC says "// default : 0 - write out all chs where global triggered bin is center of the window"

Answer: Not sure exactly what it does but we can't change it from default as the other values are for the test bed mode only.

Random Question from Setup.txt, but relevant for other things. What is "testbed mode"? Answer: The "testbed" is the original test station only 40m under. This mode is for if we do detector=3 in the first part of the setup.txt file; thus, we won't use this. 

What's the difference between "DETECTOR" and "DETECTOR_SIMULATION"? We found "DETECTOR_STATION" in Settings.h. The comments say "// for DETECTOR=4, indicates the single station to be simulated // 0 = testbed, 1 = A1, 2 = A2, 3 = A3"

In this file, there's an ANTENNA_MODE variable. You can look at what it does in Settings.h. In this file, it says that you can decide whether to use a bicone antenna response or "a different response for the top vpol antennas". How does this function work? What does it mean by "top vpol antennas"? Someone should look up the ARA antennas, and look into this function. 

Our goal is to meet and work as a group twice a week. Collaboration in work should help us progress faster. 

We have 2 physical XF flash drive keys in existence. These are non-replaceable and cannot be lost! Until further notice, the keys are held by:

(1) Cade Sbrocco (sbrocco.6@buckeyemail.osu.edu )

and

(2) Abasi Brown (brown.7146@buckeyemail.osu.edu)

                                                        General update

The following were the fixes we made to make AraSim run:

(1) As of March, the loop breaks at AraSim. We have found that our program XFintoARA.py does not properly format the output in the form that AraSim is looking for (as an input). Jorge gave us a sample AraSim input file to help us edit XFintoARA.py so that it formats the file correctly. Note that we found white space is also must match in order for the format to be correct. 

Now that we have fixed this, AraSim now runs. Attached is the following sample Jorge gave us of the proper AraSim inputs. If needed, it's available to refer back to. 

(2) After correcting the AraSim input file formats (outputs of XFintoARA.py), we ran it. It failed to finish running after hours of run time. We found that we were using default input parameters for AraSim -- which wasn't triggering in a timely fashion. Once we edited the parameters, it ran successfully. Note that this was not run within the loop. Confirmation of the full loop running has not been completed. 

The parameters of interest need to be edited in setup.txt. To see which file in AraSim I am referring to, please see the .txt file attached. Note that we will be editing the bash script to allow the user to change these input parameters there. The following red lines are what need to be set:

EXPONENT=21
NNU=100 // number of neutrino events
NNU_PASSED=10 // number of neutrino events that are allowed to pass the trigger
ONLY_PASSED_EVENTS=1 // 0 (default): AraSim throws NNU events whether or not they pass; 1: AraSim throws events until the number of events that pass the trigger is equal to NNU_PASSED (WARNING: may cause long run times if reasonable values are not chosen)
NOISE_WAVEFORM_GENERATE_MODE=0 // generate new noise waveforms for each events
NOISE_EVENTS=16 // number of pure noise waveforms
TRIG_ANALYSIS_MODE=0 // 0 = signal + noise, 1 = signal only, 2 = noise only
DETECTOR=1 // ARA stations 1 to 7
NOFZ=1
core_x=10000
core_y=10000

EXPONENT gives the energy of the neutrinos. Note that higher energies will have more pass, as the energies don't dissipate as rapidly. For now, we are running at this energy. Eventally, we will want to use the full Kotera spectrum. Otherwsie, we would be finding the performance of an antenna at only one energy. Just to check the loop, we are going to use EXPONENT=21 to get events to trigger faster. 

ONLY_PASSED_EVENTS decides which of the two options to use to determine when AraSim is finished running: (1) Throws 'NNU' number of events at antennas and stops regardless of how many pass (in this case ONLY_PASSED_EVENTS=0) , or (2) throws as many events at the antenna as needed until 'NNU_PASSED' number triggers (in this case ONLY_PASSED_EVENTS=1). Ie if this number is 10, we throw as many neutrinos at the antenna as necessary until 10 events trigger. Note that this can result in a longer run time. Additionally, a higher EXPONENT will make them trigger faster. 

TRIG_ANALYSIS_MODE is a setting that determines what is being thrown. Ie- 0=signal + noise (which is what we want, and is the 'real' case), 1=signal only (would make it run faster), 2=noise only (not of interest for us). 

                                                                To-Do List

(1) Commit version of XFintoARA.py since recent formatting change. 

(2) Figure out how to add in Kotera spectrum (note that we wont want to do this until the WHOLE loop is functional)

(3) Edit the bash script to recreat the setup.txt making the variables in red above an editable option in the top of the bash script. Ie our variables in red above will be at the top of the bash script. Later in the bash script we will need to have it create this .txt file with the user inputs for those variables. We can do this by either (1) creating a skeleton, or (2) creating another .sh script that is run in our XFLoop.sh -- which writes the first handfull of lines based on the user inputs for our variables in red above, and then pastes in the rest of what is unchanged in setup.txt. 

(4) Run the whole loop: update- we tried this today and keep getting a Veff-0 for our first antenna along with the error 

Warning in <TFile::Init>: no StreamerInfo found in /datapool/software/anita/lib/G__RootFftwWrapper_rdict.pcm therefore preventing schema evolution when reading this file.
Error in <TStreamerInfo::Build>: AraEventCalibrator, discarding: unsigned short* fAtriPeds, no [dimension]
others run as expected

(5) Work on the paper: Amy added a skeleton of sections to add for the GA. Date of completion should be in-line with APS. 

(6) APS talk slides. 

(7) Edit the manual and add: 

• The changes that will occur when task (3) is done.
• A sample of how AraSim inputs -- or XFintoARA.py outputs -- should look. 
• This stuff: http://radiorm.physics.ohio-state.edu/elog/GENETIS/8
• Max's AraSim plotting software (when it's done)
• Evelyn and Sophie's plotting software info (when it's done). 

(8) Finish Max's AraSim plotting software.

(9) Finish Evenlyn and Sophie's plotting softare. 

Tue Mar 5 17:29:02 2019
 

The things we worked on this week is in bold. Below the bold items is a description of exactly what we took care of. 
 

1. Paper:
   • Julie added Amy's edits.
   • Julie let Kai know he is welcome to edit, too. 
   • Julie will also keep edits updated as they come in!

     
      

2. Get loop working-- specifically XF:
   • Cade has now officially gotten reasonable outputs for his XF scripts. See the attached images for the XF outputs and associated CAD drawing. It seems to look like a reasonable gain pattern. 
     
      
   • 
     
      
   • 
     
      
   • 
      
   • The loop now works until AraSim. We are now getting the following AraSim errors:
     • XF_Loop.sh: line 244: ./AraSim: No such file or directory
     • XF_Loop.sh: line 244: ./AraSim: No such file or directory
   • Julie is going to help Cade make sure that this directory is correct and that it has been compiled! More info to come.
     
     
      
3. Test GA (Suren): 
   •  I've been working on creating a gaussian fitness score with which I can test the performance of our genetic algorithm. The idea is as follows: We run the GA and get a generation of individuals. We provide their fitness score by simply taking the values {radius, length, angle} as the {x, y, z} coordinate in a 3D space. We can then use a 3D gaussian to output a value at that location, and use that number as our fitness score. The beauty of this method is that we can then plot the progress of our genetic algorithm each generation as a dot in this 3D space, and we can see it slowly snake to the center. Additionally, we can use more complex fitness scores, like one with a local maxima and a global maxima to see if it get's stuck in the local maxima. Below, I have an image I generated on python of a 2D section contour plot of a 3D gaussian in length-angle space. The dark purple is a lower value and the yellow is a higher value of the gaussian. The lines and dots hand-drawn on top are an idea of what it would look for like to start the GA at the red dot and see it's progress each generation until it reaches the center, shown by the white dots. Running this for our current roulette algorithm would make a very nice GA proof of concept plot for the paper, I believe!
   • 

     
      

4. Gain pattern plots -- Make them 3D (Evelyn) & put on GitHub
   • We need to figure out how to make a group Github. We have a lot of Githubs made by people and we need them all accessible from a group account. Right now the old dipole Github and the CalPoly stuff are linked to a group account. We need to find out how to get the Bicone account held by Max, and Evelyn's plotting code all on the group account: https://github.com/gp-antennas.
   • Update: Evelyn and Max have contacted Brian to get their accounts linked. It looks like Amy, Brian, and Stephanie (Cal Poly) are the owners. 
     
     
      
5. Insert LR plot & Gain plot to loop (?)
   1. Edit bash script
   2. Update Github

       

6. AraSim plots (Julie and Max) -- see #2 and #3 here: http://radiorm.physics.ohio-state.edu/elog/GENETIS/7 for details. 
   • Max and Julie have gotten sample data on Owens.
     • located at /users/PCON0003/cond0068/ARA/AraSim/branches/AraSimLite/outputs
   • Max and Julie have gotten sample programs from Jorge.
     • https://github.com/toej93/AraSim_noise_calib/blob/master/readTree.cc
     • https://github.com/toej93/AraSim_noise_calib/blob/master/M.readTree
   • Julie installed root, and had Max source Julie's thisroot.sh files to run root on Owens.
   • Max installed Arasim to his user. 
   • Next step: run plotting software given by Jorge.
     
     
      
7. Update manual with things from Suren's last post here: http://radiorm.physics.ohio-state.edu/elog/GENETIS/8

Here are the following things we need to work on:
 

1. Paper (Julie and Suren)
2. Get loop working-- specifically XF (Cade)
3. Test GA (Suren)
4. Gain pattern plots -- Make them 3D (Evelyn) & put on GitHub
5. Insert LR plot & Gain plot to loop (?)
   1. Edit bash script
   2. Update Github
6. AraSim plots (Julie and Max) -- see #2 and #3 here: http://radiorm.physics.ohio-state.edu/elog/GENETIS/7 for details. 
7. Update manual with things from Suren's last post (?)

We are redesigning the way we simulate antennas in our loop. To do this, we changed our simulationPEC macro skeletons and our output skeleton. 

To make this easier, we changed the way we name the files, from i.uan where i is the simulation number, to i_j.uan where i is the antenna and j is the frequency.

We also rewrote XFintoARA.py so it's clean. Note that before there were comments from a previous author claiming it was VERY questionably written. This has now been cleaned up. We also changed the bash script relevant to the above changes.

This change needs to be added to the manual, and until we update this update, we have not done it.

We are getting to the stage where we would like to verify each step. It has been requested by Amy that we verify them visually. Here are the plots we need to make. 

1. Add plotting for L & R parameters (note that we already have similar software to this from the dipole check. You can find this in the GitHub from the dipole test here: https://github.com/hchasan/XF-Scripts). This should be between A, and B in the loop drawing on the manual. (Done)
2. Plot the distribution of theta and phi of incoming signal. This should be after D in the loop drawing on the manual. 
3. Plot the distribution in cartesian coordinates of the neutrino interaction position. Maybe we can also plot the distance between the interaction point and the antenna. This should be after D in the loop drawing on the manual. 

Finally, as a tasks not to do with plotting, we need to make sure we don't lose any data as we go. We want to make sure that we keep tagged or labelled which individuals survive, and save the data flagged for those as we go. 

Here are a list of things we wish XF would do. We are keeping this list in hopes to add it to a proposal with them. 

1. Save pictures of CAD drawings of simulations. 
2. Close/quit XF GUI. App.quit doesn't actually work for us. 
3. Suppress the GUI and run solely through terminal. 

As mentioned in the manual, using ssh to login to Nutau causes delays; the XFdtd GUI cannot be surpress and must be forwarded via X11 forwarding, and this is extremely slow. We are currently looking into using a newly suggested option, XRDP. Information will be added as this process continues. 

Atttendance: Julie, Max, Cade, Suren, Evelyn, Sophie

Today's tasks:

Max works on AraSim section (needs to do more research and reading)

Julie, Suren, Sophie, Evelyn work on plotting software to plot gain patterns of indiviudals

Cade preps impuse XF code to be implemented

To-Do list:

I. Divide up sections on paper, and work on them. See http://radiorm.physics.ohio-state.edu/elog/GENETIS/1 for section assignments. 

II. Implement impuse code. 

• This will require us to change XFintoARA.py so that it takes less output files. 
• Is there anything else different with the output files?

III. Finish Evelyn and Sophie's plotting software.

• This means also puting it onto Nutau, and adding in lines to the bash script (both to move and save the images, and to run the code!)

Here are things we can do if we are waiting for Remcom:

I. Change sides of bicone to be sinusoid instead of lines.

II. Take known bicone parameters and put it through AraSim to check if it's working properly.

• This also means we should make plots with it to triple check. 

III. Test GA and make it maximize other paramaters to see how it does. 

Slack: https://gpantennas.slack.com/messages 

elog:http://radiorm.physics.ohio-state.edu/elog/GENETIS/

Github link: https://github.com/mclowdus/BiconeEvolution

DropBox link: https://www.dropbox.com/home/GP_Antennas

Old GitHub for the evolution of the dipole: https://github.com/hchasan/XF_Scripts

Hi all, 

The working copy of the user manual can be found here: https://www.overleaf.com/6955763886pjtkckprxsyp

It still need to be tested by someone new to see if edits need to be made. I have uploaded the copy that is finalized as of this day. Note if you click on the overleaf version that may be more up-to-date. This copy is added as a PDF so we can keep track of versions.