Slides shown at experts building meeting Aug. 2nd, 2023.

Here are some reading materials about HPol antennas:

This paper discusses some initial HPol designs that were tested in a prototype station of ARA.  We went with the slotted cylinder design.  

Here you can see what the RNO-G experiment in Greenland has considered.

We can add other sources here as we find them.

Input File Format for pueoSim (Also ICEMC)

Frequency Range: From 200 MHz to 1500 MHz incrementing by 10 MHz steps

There are 8 different files that are required for pueoSim. They are:

vv_0: Max Gain at each Frequency, Vertical Polarization

hh_0: Max Gain at each Frequency, Horizontal Polarization

vh_0: Max Gain at each Frequency, Vertical to Horizontal Polarization

hv_0: Max Gain at each Frequency, Horizontal to Vertical Polarization

vv_el: Gain at Theta Angles [5, 10, 20, 30, 45, 90] for each Frequency, Vertical Polarization

vv_az: Gain at Phi Angles [5, 10, 20, 30, 45, 90] for each Frequency, Vertical Polarization

hh_el: Gain at Theta Angles [5, 10, 20, 30, 45, 90] for each Frequency, Horizontal Polarization

hh_az: Gain at Phi Angles [5, 10, 20, 30, 45, 90] for each Frequency, Horizontal Polarization

Currently XF doesn't output all the information we need to create all these files. The only files able to be made from current XF outputs are vv_0, vv_el, and vv_az. Once we have the correct XF Outputs, it shouldn't be too much of a hassle to fix the current translation code.

Format for each of the files are the same. One column is Frequency (Hz) and the other is Gain. Attached are example files vv_0 and vv_el to visually see the format, though the frequency range is the current ARA range and not the final PUEO range as I am using ARA data to make these files. The files are named vv_0 (no .txt or .csv or any extension) and vv_el

Link to Google Doc with this information: https://docs.google.com/document/d/1iRUF6hIEyQfMK0LL21caRuHPgXYP30ZdkSkFv_-Y8R0/edit

Here are some preliminary results from testing the effect of error on growth in the GA. For this test, we start with a simulated error of 0.5 because our true fitness score is bounded between 0.0 and 1.0. From here, we simulate doubling the number of neutrinos by reducing the error by root(2), then root(4), and observed the growth on the fitness scores plots. We did these three tests with both 50 and 100 individuals (plots starting with 30 use 50 and starting with 60 use 100). Looking at the plots, we can see that the only plot that shows growth is 100 individuals with halved error (corresponding to 4 times the number of neutrinos). In fact, it took going to root(25) (corresponding to 25 times the number of neutrinos) to observe comparable growth with 50 individuals. This leads us to believe we likely need to increase both neutrinos and individuals to see meaningful growth in our GA. Further tests are required to see how consistent these results are. Another test, though unrealistic, we ran 1000 individuals with the root(25), this test was optimized in around 10 generations, this could be seen as a maximum performance standard.

William Luszczak  1:30 PM

This is the directory with the current PUEO antenna data files: https://github.com/PUEOCollaboration/pueoSim/tree/main/data/antennas. Each type of antenna will need several files:

• vv_0 and hh_0 describe the on axis v and h pol gain as a function of frequency. Gain is listed as a multiplicative factor (not in dB, if I'm remembering correctly)
• hv_0 and vh_0 describe the on axis cross polarization gain
• vv_az and hh_az describe the off-axis gain via multiplicative factors of the boresight gain. These are currently listed as a function of frequency for reference angles of 0,10,20,30,40,50,60,70,80, and 90 degrees, though we can always adjust the reference angles if needed.
• vv_el and hh_el are similar to above, but for gain as a function of elevation angle instead of azimuthal angle.

So for example, the total vpol gain in a particular frequency bin for a signal incident at azimuthal angle az and elevation angle el would be vv_0(f)*vv_az(f, az)*vv_el(f, el)

New

1:31

We can also potentially adjust this if it would be more convenient for the GENETIS people to output antenna information in a different format (or at different reference angles or whatever). The important thing is that the boresight h, v, and cross pol gains are all defined, as well as the off-axis response (as a function of azimuth and elevation)

Recording of training on XFdtd by Alex Machtay on July 12, 2023.

https://osu.zoom.us/rec/share/FVKiTcQrRh8NLNYJADwugy0H2VjDo8ZlBsH16sJAGv2JINNwxxzyNfo5ueqVgOTO._XErF57Kz8KOM3a5

This link will expire in 120 days so note that we need to post a permanent recording.

Here's the GitHub repository with examples of Xmacros: https://github.com/Machtay/XFdtd_Scripts 

The reviewer for the paper we recently submitted mentioned that our step sizes at which we are measuring the gain in XFdtd may be too large. They pointed out that there appear to be "lobes" at 400 MHz. I ran the antenna we presented in the paper through XFdtd using a step size of 5 degrees (what we've been doing) and a step size of 1 degree (the reveiwer's recommendation). Attached are antenna responses for these two different step sizes at 200 MHz and 400 MHz. Qualitatively, there are noticeable differences in the "jaggedness" of the 400 MHz plot and how extreme the minima and maxima appear, but the basic shape is preserved. We can try to do a more quantitative analysis (ex: run these through AraSim), but doing so may be more time intensive than necessary considering that AraSim may require 5 degree steps and that adding these images to an appendix may be sufficient anyway.

We have 2 physical XF flash drive keys in existence. These are non-replaceable and cannot be lost! Until further notice, the keys are held by:

(1) Cade Sbrocco (sbrocco.6@buckeyemail.osu.edu )

and

(2) Abasi Brown (brown.7146@buckeyemail.osu.edu)

Beam patterns with ARA bicone contained in a cylinder of air surrounded by a shell of ice.

 

I've been building in XF and have ran into some issues but currently have the following completed:

- Struts are placed, though they are not evenly spaced and the same. I have contacted XF to try and figure out how to get rotation/patterns to do what we need

- Circuit components and circuit board are in the xf file, though the components are not on the circuit board (I'm not totally sure how to put them on, that's the next step). The circuit components should be of spec, I looked up the part numbers from the previous ELOG post and grabbed the values from the manufacturer.

Here are the things that have not been completed/need answered:

- How to evenly space the struts connecting the shape (as stated previously, I've contacted XF to figure out how to do this easily. Im just bad at modeling in XF)

- Add circuit components to circuit board

- Add voltage and ground connections to the circuit board

- On the circuit diagram, figure out J1 and J2 connections. Meaning, which one connects to the signal from the cable and which connects to the other cone that is used as ground?

To see current antenna build go to: " /users/PAS1977/jacobweiler/GENETIS/XFzone/straightened_antenna_for_Jack.xf "

Attached are pictures in case there are issues pulling up model in XF. :)

Most of our students 

(1)Ubuntu doesn't come with X11 forwarding on its own. So you can install Xlaunch

xeyes -display :0

Attendance: Julie, Alex M., Alex P., Cade, Scott.

What we have done today:
(1) Fixed Concerns regarding copying the output .uan files properly so they are not overwritten. (see #8 in this elog entry: http://radiorm.physics.ohio-state.edu/elog/GENETIS/22 for details on the error). This was corrected in part E as:

for i in `seq 1 $NPOP`

do

    for freq in `seq 1 60`

    do

    #Remove if plotting software doesnt need                                                                                                                                                             

    #cp data/$i.uan ${i}uan.csv                                                                                                                                                                          

        cp Antenna_Performance_Metric/${i}_${freq}.uan "$WorkingDir"/Run_Outputs/$RunName/0_${i}_${freq).uan

    done

done

And in the "looping part" -- ie part G we have corrected it to say the following. Note that every time we run an iteration/generation in the loop, it writes the name of the output files the exact same (individual_frequencystep.uan). It will overwrite these files in the next generation unless we move them and rename them like so. We don't want this, because we will likely want to plot the gain patterns of the individuals we like (gain pattern data is in the .uan files). 

 for i in `seq 1 $NPOP`

        do

            for freq in `seq 1 60`

            do

    # Gens data used to create a .csv file for the uan file for gain plotting                                                                                                                            

    # cp Antenna_Performance_Metric/$i.uan ${i}uan.csv                                                                                                                                                   

                cp Antenna_Performance_Metric/${i}_${freq}.uan "$WorkingDir"/Run_Outputs/$RunName/${gen}_${i}_${freq}.uan

            done

        done

(2) Added scale factor "A" (actually called "Scale Factor" in our bash script and fitnessFunction_ARA.cpp), and the generationDNA.csv as an input file (this file is where the radii of all individuals in the current generation live).

• We have added this as a variable in the fitnessFunction_ARA.cpp. It now runs as a flag pass in when it runs in the bash script. In section E:

./fitnessFunction.exe $NPOP $ScaleFactor $AntennaRadii/generationDNA.csv $InputFiles #Here's where we add the flags for the generation

and in the looping part (part G):

     ./fitnessFunction.exe $NPOP $ScaleFactor $AntennaRadii/generationDNA.csv $InputFiles #Here's where we add the flags for the generation

and at the top where variables are declared:

RunName='Julie_12_2_2'           ## Replace when needed                                                                                                                                                  

TotalGens=6                     ## number of generations (after initial) to run through                                                                                                                  

NPOP=4                          ## number of individuals per generation; please keep this value below 99                                                                                                 

FREQ=60                         ## frequencies being iterated over in XF (Currectly only affects the output.xmacro loop)                                                                                 

NNT=10000                          ##Number of Neutrinos Thrown in AraSim                                                                                                                                

ScaleFactor=1.0                   ##ScaleFactor used when punishing fitness scores of antennae larger than holes used in fitnessFunctoin_ARA.cpp 

Note that every time the roulette algorithm runs, it's written into generationDNA.csv. We copy those right before rerunning the roulette alg in the next generation -- ie cp generationDNA.csv Run_Outputs/$RunName/${gen}_generationDNA.csv -- so that each generations parameters don't get overwritten. However, the current generation's parameters live in generationDNA.csv until the roulette alg runs in the next generation, so we can read in that file to the fitnessFunction_ARA.cpp file to determine the constraint (ie R_antenna) 

(4) Implemented Alex P.'s version of his optimized AraSim. This is now fully functional in the loop. 

NOTE: WE HAVE MADE A MISTAKE! WE MADE THE CONSTRAINT THE R-VALUE WE READ IN FROM THE GENERATIONDNA.CSV FILE. THIS IS NOT THE RADIUS WE THINK IT IS! WE NEED TO DO SOME TRIG ON IT TO FIX IT!!

Attendance: Julie, Alex P., Cade, Alex M., Mitchell, Evelyn (Go team!)

I. Adding Constraints on the in-ice hole size.

Today we worked on adding constraints on the drilled hole size. Our plan is to do so by penalizing the fitness scores of indiviuals with a radius larger than that of the hole in the ice. Ie in fitnessFunction_ara.cpp we would have:

if R_antenna>=R _hole

fitnessScore =fitnessScore x (e^{-[A(R_antenna-R_hole)/R_hole)]^2}) 

else

fitnessScore= veff (or whatever way we decide to calculate the fitness score in the future, if this ever changed)

Where "A" is a constant we can adjust to tweak how heavy the penalty is. Other things to do in order to make this work:

(1) Make constant "A" a variable in the bash script. 

• Do this the same way $NPOP is used in the roulette algorithm cpp program! :)

(2)Read in the roulette algorithm output file for your radius of the antenna. 

• Note that this isn't a super obvious thing. Right now our roulette alg outputs are saved as $(generation)_filename.csv. So, for the first gen -- ie gen 0-- it will be 0_filename.csv. For the second gen -- ie gen 1-- it will be 1_filename.csv...etc. This means we need to make sure the fitness score calculator has a way of knowing what generation we are on, in order to be sure it is grabbing the correct radius for each individual (not that of an old generation). 
   

We will be working on this more in our meeting tomorrow. As for now, the game plan has been to make comments on what we should be adding in the correct sections of fitnessFunction_ara.cpp (and will be doing the same in the bash script shortly, too --ie XF_Loop.sh). 

II. Adding Ara Values to Plot. 

We haven't started this yet, but it is something we need to do. When we plot Veff vs Generation, Theta vs Generation, and R vs Generation we need to add in the value of these for the current ARA bicone as a baseline for our evolutions.

Note: All of the contents over the last month need to be added to the manual! DO NOT FORGET!

To Do List:

We want to keep testing the loop. Right now, we are using a cpu running at half scale. This gives estimated times for each antenna of up to 2 hours. GPUs are suddenly hard to come by, so if we are relegated to using CPUs we'll probably need to scale down considerably.

Evelyn, Ryan, and Mitchell are probably going to be ready for the python project soon. If someone is around on Friday to help them they should be able to finish the bash project.

One idea is to run all the inital xmacros in the interactive job and then submit a GPU job that runs all the xfsolver command. With the hope that not having the GPU in the interactive job will make it easier to get the interactive job and to get the GPU for a shorter time because it would only be needed for a smaller segment. This would especially help with running larger amounts of individuals.

AraSim EField project Error:
 

 *** Break *** segmentation violation

===========================================================
There was a crash.
This is the entire stack trace of all threads:
===========================================================
#0  0x00002b845aeb445c in waitpid () from /lib64/libc.so.6
#1  0x00002b845ae31f52 in do_system () from /lib64/libc.so.6
#2  0x00002b84562bebf4 in TUnixSystem::StackTrace() () from /cvmfs/ara.opensciencegrid.org/trunk/centos7/root_build/lib/libCore.so
#3  0x00002b84562c13ea in TUnixSystem::DispatchSignals(ESignals) () from /cvmfs/ara.opensciencegrid.org/trunk/centos7/root_build/lib/libCore.so
#4  <signal handler called>
#5  0x000000000044fd70 in void std::vector<double, std::allocator<double> >::emplace_back<double>(double&&) ()
#6  0x00000000004c3c0e in Report::Connect_Interaction_Detector(Event*, Detector*, RaySolver*, Signal*, IceModel*, Settings*, Trigger*, int) ()
#7  0x0000000000436eb3 in main ()
===========================================================

The lines below might hint at the cause of the crash.
You may get help by asking at the ROOT forum http://root.cern.ch/forum
Only if you are really convinced it is a bug in ROOT then please submit a
report at http://root.cern.ch/bugs Please post the ENTIRE stack trace
from above as an attachment in addition to anything else
that might help us fixing this issue.
===========================================================
#5  0x000000000044fd70 in void std::vector<double, std::allocator<double> >::emplace_back<double>(double&&) ()
#6  0x00000000004c3c0e in Report::Connect_Interaction_Detector(Event*, Detector*, RaySolver*, Signal*, IceModel*, Settings*, Trigger*, int) ()
#7  0x0000000000436eb3 in main ()
===========================================================

The following items from our to-do list have been successfully completed. Below this list, you can find the remaining urgent items for our proposal due date of 1/28 (items must be completed with plenty of time to spare). 

Completed items: 

*Note* please check the highlighted item. This one still must be verified. 

1. Made software plot Theta VS Gen
2. Fixed current L and R plots
   1. These now have correct axis labels, a legend, and connect data for each individual between generations.
3. Produced Veff for ARA's actual bicone parameters. 
   1. Made XF_Loop.sh submit a job for the ARA gain AraSim input file in BiconeEvolutionOSC/AraSim/Ara_Bicone6in_outputs.txt
   2. STILL MUST: Make sure that the "wait function" has been properly edited to consider the submission of this extra job in the first iteration of the loop. (Not sure if $NPOP+1 instead of $NPOP fixed it correctly) ----> CADE SHOULD LOOK AT THIS :)
4. Added plotting software for V_eff VS Generation (includes ARA's actual bicone V_eff as generated in #3).
5. Fixed the ARA hole constraint in fitnessFunction_ARA.cpp -- ie fixed the calculation to consider the correct radius "r". See http://radiorm.physics.ohio-state.edu/elog/GENETIS/28 for details. 
6. Successfully made energy a variable in setup.txt. 
   1. used the same "sed" format as before in the bash script (XF_Loop.sh).
7. Pushed to Github. 

Items to still do:

1. Save CAD models in XF: Due 1/15                                                   ------>(Cade-- second most urgent one)
2. Stop plots from popping up during run: Due 1/15                            ------>(Alex P.)
3. Add "save state" idea to loop: Due 1/15                                         ------> (Cade and maybe Alex P can help -- This is the most urgent one)
4. Figure out why fitnessFunction_ARA.cpp isn't plotting: Due 1/15     ------>  (Julie, and Alex M)
5. Run the loop and evolve for results: Due 1/24                               ------>(Alex M, Mitchell, Cade)
6. Write proposal: Final Due 1/28                                                   ------> (Amy, Kai, Julie)

Note that the people running the loop (Alex M, Mitchell, Cade) should be producing plots and saving them. Please save all plots in this dropbox folder: https://www.dropbox.com/home/GP_Antennas/NSF%20Proposal%20Plots%20(1_28_2020)

When submitting plots, please make a new folder in there and label it "Date_of_run, Energy, Generations, Individuals". For example, my folder within this dropbox link would be: "1_10, 10^19, 10, 5". I have made sample folders in this dropbox link if needed.

You can export the plots from OSC to your machine (using CP command) or can type "display NameOfPlot.png" then screenshot and crop your plot to upload to dropbox.

Attendance for today's session: Cade, Julie, Alex M, Mitchell, Scott. 

Tasks/to-do list:

1. ​Get loop to run up to the fitness function.
   • Right now we have the bash script setup to run AraSim by submitting them as parallel jobs. Today Cade added the "wait" function within the job in order to make sure the loop doesn't continue to the next step before the Arasim data is ready. Cade will give us an update on Thursday letting us know whether or not this completed or gave an error. 
2. Fix error at fitness function. 
   • Currently, the loop stops at the fitness function calculation, and aborts (program fitnessFunction_ARA.cpp). We are unsure of why at the moment. Here are the things we are looking into. 
     • Run this outside of the loop-- ie compile the fitnessFunction_ARA.cpp (g++ -std=c++11 fitnessFunction_ARA.cpp -o fitnessFunction.exe), and run the executable(./fitnessFunction.exe). What happens?
       • Julie and Alex did this. It gave a segmentation fault. We need to look into this further. 
       • We double-checked that this .exe was properly created from compiling, and it was.
     • Double-check our AraSim output by checking:
       • Format
       • File type
     • Is there an error past the fitness function, and it's killing the loop after this runs?
       • Is his output not properly going into the gen alg?
       • Are the output files properly:
         • Filled
         • Formatted
         • Put in the right place
     • Look at how it's calculating the fitness score.

Since it seems like Alex and I may have found a problem at our first "test", this could be the issue. We will investigate it further. If (1) Cade's "wait" function works, and (2) we can fix this issue with the loop stopping at the fitness score calculation, ideally this means that the loop is then running. 

Today we also found that it is better to run on the terminal through an interactive job than a virtual desktop. This is MUCH faster -- ie we don't have to wait nearly as long as we do to get a node via a virtual desktop. This means that everyone with a PC needs to either (1) work out their issues with X11 forwarding crashing (Mitchell succeeded at this today by making sure Xquartz was set up to work via ssh to Pitzer in the settings), or (2) dual boot Linux. It seems that Mac users are not having issues, but that some of the Windows users are having their OS completely crash when XF GUI tries to forward. Users can test whether or not their X11 forwarding is working by doing the following:

1.  Logging onto Pitzer
2. Typing "module load xfdtd"
3. Then typing "xfdtd"

This should successfully open the GUI for XF. If it crashes, users may have to do one of the things I mentioned above. Once we do succeed at having this setup, you can start an interactive job via the command: 

qsub -I -X -l walltime=1:00:00 -l nodes=1:ppn=10

You then have to wait until the node is free. This has taken about 1/3 of the time to get than when requesting a virtual desktop. Note that, however, we may need a higher ppn (maybe something like 40). This is not totally clear yet. 

Attendance for today's session: Cade, Julie, Alex M, Mitchell, Scott. 

Tasks worked on:

• Continue to fix continuity of loop: Cade and Julie worked on fixing the following errors:
  1. In section D of the loop, it currently
     • Moves the output from the python conversion to $AraSimExec directory. 
     • Renames it as a .txt file (note the $i is renaming each in line with $NPOP). 

for i in `seq 1 $NPOP`

do

      mv evol_antenna_model_$i.dat $AraSimExec

done

for i in `seq 1 $NPOP`

do

      cp evol_antenna_model_$i.dat a_$i.txt

done

However, this wasn't working and it was only properly renaming individual 1. To check this,

• We confirmed it is properly iterating the first loop by using the "echo" command. The second loop was not properly iterating, however.
• We were able to fix it by combining these steps. This was confirmed to have worked. 

for i in `seq 1 $NPOP`

do

      mv evol_antenna_model_$i.dat $AraSimExec/a_$i.txt

done

              2. We got this iteration $i to work in the job script -- ie as an input file variable. The loop now successfully submits AraSim as a job with the proper input files. 

• Clean bash script
  • Mitchell and Alex M worked again today on turning every function possible into functions (and putting them in their own .sh scripts that are being called by XF_Loop.sh). 
    • The loop runs the first generation "manually" -- ie each step was spelled out, and nothing is put into a loop. The second+ generations then run as a loop. This is because we need to initialize the first XF run -- which then has no problems in the later iterations.
      • Alex M. and Mitchell are now turning each section A-F into individual functions and put into their own .sh scripts. Thus section A would just be calling another .sh script. In gen 2+ we then loop over these SectionA.sh through SectionF.sh functions. All of these just got completed. He just needs to add the change Julie and Cade made today, and clean some things up. 
        • For example, he has an "if else" statement that tells it whether or not it's the first (XF initializes) generation, or in generation 2+. Cade has a few changes he would like Alex to make to clean up this part of the run. 
    • Idea: The loop used to run the first generation "manually" -- ie each step was spelled out, and nothing was put into a loop. The second+ generations then ran as a loop. Since we were able to get XF to import the xmacro scripts automatically -- as well as save the project with the correct name (matching the expected name initialize in the bash script)-- they've gotten rid of that first "manual" iteration, and are now fully running all generations as a loop. We haven't tested whether or not we can make this change yet, but it is worth looking into now that some changes have been successfully made with XF. 

Quick note from Julie:

I tried to run from the terminal once Cade left. I get the following output/error that we should make sure isn't happening for other users (since most of this is a permissions issue that I thought we fixed. I can look into the other errors and see if anyone else gets them). For whatever reason, Cade is not having the same output errors; he is able to run the same copy just fine. 

Roulette algorithm initialized.

./XF_Loop.sh: line 136: output.xmacro: Permission denied

./XF_Loop.sh: line 137: output.xmacro: Permission denied

./XF_Loop.sh: line 138: output.xmacro: Permission denied

./XF_Loop.sh: line 139: output.xmacro: Permission denied

./XF_Loop.sh: line 140: output.xmacro: Permission denied

./XF_Loop.sh: line 146: simulation_PEC.xmacro: Permission denied

./XF_Loop.sh: line 147: simulation_PEC.xmacro: Permission denied

./XF_Loop.sh: line 149: simulation_PEC.xmacro: Permission denied

./XF_Loop.sh: line 158: simulation_PEC.xmacro: Permission denied

 

Opening XF user interface...

*** Please remember to save the project with the same name as RunName! ***

 

1. Import and run simulation_PEC.xmacro

2. Import and run output.xmacro

3. Close XF

Lmod has detected the following error:  The following module(s) are unknown: "xfdtd"

 

Please check the spelling or version number. Also try "module spider ..."

It is also possible your cache file is out-of-date; it may help to try:

  $ module --ignore-cache load "xfdtd"

Also make sure that all modulefiles written in TCL start with the string #%Module

./XF_Loop.sh: line 172: xfdtd: command not found

./XF_Loop.sh: line 177: cd: /fs/project/PAS0654/BiconeEvolutionOSC/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop/Run_Outputs/Julie/Julie.xf/Simulations/000001/Run0001/: No such file or directory

./XF_Loop.sh: line 178: xfsolver: command not found

./XF_Loop.sh: line 177: cd: /fs/project/PAS0654/BiconeEvolutionOSC/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop/Run_Outputs/Julie/Julie.xf/Simulations/000002/Run0001/: No such file or directory

./XF_Loop.sh: line 178: xfsolver: command not found

./XF_Loop.sh: line 183: xfdtd: command not found

chmod: changing permissions of ‘/fs/project/PAS0654/BiconeEvolutionOSC/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop/AraSimTestJob.sh’: Operation not permitted

chmod: changing permissions of ‘/fs/project/PAS0654/BiconeEvolutionOSC/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop/Antenna_Performance_Metric/AraOut_%num%.txt’: Operation not permitted

chmod: changing permissions of ‘/fs/project/PAS0654/BiconeEvolutionOSC/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop/Loop_Functions/XF_Loop3.sh’: Operation not permitted

chmod: changing permissions of ‘/fs/project/PAS0654/BiconeEvolutionOSC/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop/Loop_Functions/#Part_C.sh#’: Operation not permitted

chmod: changing permissions of ‘/fs/project/PAS0654/BiconeEvolutionOSC/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop/Loop_Functions/#Part_A.sh#’: Operation not permitted

chmod: changing permissions of ‘/fs/project/PAS0654/BiconeEvolutionOSC/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop/scriptEOFiles/AraSimCall.sh.e957233’: Operation not permitted

chmod: changing permissions of ‘/fs/project/PAS0654/BiconeEvolutionOSC/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop/scriptEOFiles/AraSimCall.sh.e957230’: Operation not permitted

chmod: changing permissions of ‘/fs/project/PAS0654/BiconeEvolutionOSC/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop/scriptEOFiles/AraSimCall.sh.e957234’: Operation not permitted

chmod: changing permissions of ‘/fs/project/PAS0654/BiconeEvolutionOSC/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop/scriptEOFiles/AraSimCall.sh.o957224’: Operation not permitted

chmod: changing permissions of ‘/fs/project/PAS0654/BiconeEvolutionOSC/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop/scriptEOFiles/AraSimCall.sh.e957220’: Operation not permitted

chmod: changing permissions of ‘/fs/project/PAS0654/BiconeEvolutionOSC/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop/scriptEOFiles/AraSimCall.sh.o957220’: Operation not permitted

chmod: changing permissions of ‘/fs/project/PAS0654/BiconeEvolutionOSC/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop/scriptEOFiles/AraSimCall.sh.o957234’: Operation not permitted

chmod: changing permissions of ‘/fs/project/PAS0654/BiconeEvolutionOSC/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop/scriptEOFiles/AraSimCall.sh.o957227’: Operation not permitted

chmod: changing permissions of ‘/fs/project/PAS0654/BiconeEvolutionOSC/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop/scriptEOFiles/AraSimCall.sh.o957233’: Operation not permitted

chmod: changing permissions of ‘/fs/project/PAS0654/BiconeEvolutionOSC/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop/scriptEOFiles/AraSimCall.sh.o957218’: Operation not permitted

chmod: changing permissions of ‘/fs/project/PAS0654/BiconeEvolutionOSC/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop/scriptEOFiles/AraSimCall.sh.e957224’: Operation not permitted

chmod: changing permissions of ‘/fs/project/PAS0654/BiconeEvolutionOSC/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop/scriptEOFiles/AraSimCall.sh.e957231’: Operation not permitted

chmod: changing permissions of ‘/fs/project/PAS0654/BiconeEvolutionOSC/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop/scriptEOFiles/AraSimCall.sh.e957226’: Operation not permitted

chmod: changing permissions of ‘/fs/project/PAS0654/BiconeEvolutionOSC/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop/scriptEOFiles/AraSimCall.sh.o957221’: Operation not permitted

chmod: changing permissions of ‘/fs/project/PAS0654/BiconeEvolutionOSC/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop/scriptEOFiles/AraSimCall.sh.o957223’: Operation not permitted

chmod: changing permissions of ‘/fs/project/PAS0654/BiconeEvolutionOSC/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop/scriptEOFiles/AraSimCall.sh.e957217’: Operation not permitted

chmod: changing permissions of ‘/fs/project/PAS0654/BiconeEvolutionOSC/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop/scriptEOFiles/AraSimCall.sh.e957227’: Operation not permitted

chmod: changing permissions of ‘/fs/project/PAS0654/BiconeEvolutionOSC/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop/scriptEOFiles/AraSimCall.sh.e957218’: Operation not permitted

chmod: changing permissions of ‘/fs/project/PAS0654/BiconeEvolutionOSC/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop/scriptEOFiles/AraSimCall.sh.e957223’: Operation not permitted

chmod: changing permissions of ‘/fs/project/PAS0654/BiconeEvolutionOSC/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop/scriptEOFiles/AraSimCall.sh.e957225’: Operation not permitted

chmod: changing permissions of ‘/fs/project/PAS0654/BiconeEvolutionOSC/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop/scriptEOFiles/AraSimCall.sh.o957230’: Operation not permitted

chmod: changing permissions of ‘/fs/project/PAS0654/BiconeEvolutionOSC/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop/scriptEOFiles/AraSimCall.sh.o957225’: Operation not permitted

chmod: changing permissions of ‘/fs/project/PAS0654/BiconeEvolutionOSC/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop/scriptEOFiles/AraSimCall.sh.e957222’: Operation not permitted

chmod: changing permissions of ‘/fs/project/PAS0654/BiconeEvolutionOSC/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop/scriptEOFiles/AraSimCall.sh.o957231’: Operation not permitted

chmod: changing permissions of ‘/fs/project/PAS0654/BiconeEvolutionOSC/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop/scriptEOFiles/AraSimCall.sh.o957217’: Operation not permitted

chmod: changing permissions of ‘/fs/project/PAS0654/BiconeEvolutionOSC/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop/scriptEOFiles/AraSimCall.sh.o957222’: Operation not permitted

chmod: changing permissions of ‘/fs/project/PAS0654/BiconeEvolutionOSC/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop/scriptEOFiles/AraSimCall.sh.o957226’: Operation not permitted

chmod: changing permissions of ‘/fs/project/PAS0654/BiconeEvolutionOSC/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop/scriptEOFiles/AraSimCall.sh.e957221’: Operation not permitted

 

Resuming...

 

chmod: changing permissions of ‘/fs/project/PAS0654/BiconeEvolutionOSC/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop/AraSimTestJob.sh’: Operation not permitted

chmod: changing permissions of ‘/fs/project/PAS0654/BiconeEvolutionOSC/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop/Antenna_Performance_Metric/AraOut_%num%.txt’: Operation not permitted

chmod: changing permissions of ‘/fs/project/PAS0654/BiconeEvolutionOSC/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop/Loop_Functions/XF_Loop3.sh’: Operation not permitted

chmod: changing permissions of ‘/fs/project/PAS0654/BiconeEvolutionOSC/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop/Loop_Functions/#Part_C.sh#’: Operation not permitted

chmod: changing permissions of ‘/fs/project/PAS0654/BiconeEvolutionOSC/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop/Loop_Functions/#Part_A.sh#’: Operation not permitted

chmod: changing permissions of ‘/fs/project/PAS0654/BiconeEvolutionOSC/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop/scriptEOFiles/AraSimCall.sh.e957233’: Operation not permitted

chmod: changing permissions of ‘/fs/project/PAS0654/BiconeEvolutionOSC/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop/scriptEOFiles/AraSimCall.sh.e957230’: Operation not permitted

chmod: changing permissions of ‘/fs/project/PAS0654/BiconeEvolutionOSC/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop/scriptEOFiles/AraSimCall.sh.e957234’: Operation not permitted

chmod: changing permissions of ‘/fs/project/PAS0654/BiconeEvolutionOSC/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop/scriptEOFiles/AraSimCall.sh.o957224’: Operation not permitted

chmod: changing permissions of ‘/fs/project/PAS0654/BiconeEvolutionOSC/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop/scriptEOFiles/AraSimCall.sh.e957220’: Operation not permitted

chmod: changing permissions of ‘/fs/project/PAS0654/BiconeEvolutionOSC/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop/scriptEOFiles/AraSimCall.sh.o957220’: Operation not permitted

chmod: changing permissions of ‘/fs/project/PAS0654/BiconeEvolutionOSC/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop/scriptEOFiles/AraSimCall.sh.o957234’: Operation not permitted

chmod: changing permissions of ‘/fs/project/PAS0654/BiconeEvolutionOSC/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop/scriptEOFiles/AraSimCall.sh.o957227’: Operation not permitted

chmod: changing permissions of ‘/fs/project/PAS0654/BiconeEvolutionOSC/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop/scriptEOFiles/AraSimCall.sh.o957233’: Operation not permitted

chmod: changing permissions of ‘/fs/project/PAS0654/BiconeEvolutionOSC/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop/scriptEOFiles/AraSimCall.sh.o957218’: Operation not permitted

chmod: changing permissions of ‘/fs/project/PAS0654/BiconeEvolutionOSC/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop/scriptEOFiles/AraSimCall.sh.e957224’: Operation not permitted

chmod: changing permissions of ‘/fs/project/PAS0654/BiconeEvolutionOSC/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop/scriptEOFiles/AraSimCall.sh.e957231’: Operation not permitted

chmod: changing permissions of ‘/fs/project/PAS0654/BiconeEvolutionOSC/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop/scriptEOFiles/AraSimCall.sh.e957226’: Operation not permitted

chmod: changing permissions of ‘/fs/project/PAS0654/BiconeEvolutionOSC/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop/scriptEOFiles/AraSimCall.sh.o957221’: Operation not permitted

chmod: changing permissions of ‘/fs/project/PAS0654/BiconeEvolutionOSC/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop/scriptEOFiles/AraSimCall.sh.o957223’: Operation not permitted

chmod: changing permissions of ‘/fs/project/PAS0654/BiconeEvolutionOSC/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop/scriptEOFiles/AraSimCall.sh.e957217’: Operation not permitted

chmod: changing permissions of ‘/fs/project/PAS0654/BiconeEvolutionOSC/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop/scriptEOFiles/AraSimCall.sh.e957227’: Operation not permitted

chmod: changing permissions of ‘/fs/project/PAS0654/BiconeEvolutionOSC/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop/scriptEOFiles/AraSimCall.sh.e957218’: Operation not permitted

chmod: changing permissions of ‘/fs/project/PAS0654/BiconeEvolutionOSC/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop/scriptEOFiles/AraSimCall.sh.e957223’: Operation not permitted

chmod: changing permissions of ‘/fs/project/PAS0654/BiconeEvolutionOSC/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop/scriptEOFiles/AraSimCall.sh.e957225’: Operation not permitted

chmod: changing permissions of ‘/fs/project/PAS0654/BiconeEvolutionOSC/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop/scriptEOFiles/AraSimCall.sh.o957230’: Operation not permitted

chmod: changing permissions of ‘/fs/project/PAS0654/BiconeEvolutionOSC/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop/scriptEOFiles/AraSimCall.sh.o957225’: Operation not permitted

chmod: changing permissions of ‘/fs/project/PAS0654/BiconeEvolutionOSC/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop/scriptEOFiles/AraSimCall.sh.e957222’: Operation not permitted

chmod: changing permissions of ‘/fs/project/PAS0654/BiconeEvolutionOSC/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop/scriptEOFiles/AraSimCall.sh.o957231’: Operation not permitted

chmod: changing permissions of ‘/fs/project/PAS0654/BiconeEvolutionOSC/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop/scriptEOFiles/AraSimCall.sh.o957217’: Operation not permitted

chmod: changing permissions of ‘/fs/project/PAS0654/BiconeEvolutionOSC/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop/scriptEOFiles/AraSimCall.sh.o957222’: Operation not permitted

chmod: changing permissions of ‘/fs/project/PAS0654/BiconeEvolutionOSC/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop/scriptEOFiles/AraSimCall.sh.o957226’: Operation not permitted

chmod: changing permissions of ‘/fs/project/PAS0654/BiconeEvolutionOSC/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop/scriptEOFiles/AraSimCall.sh.e957221’: Operation not permitted

Resuming...

NOTE:  You may want to change the following in your job for Owens

 

    nodes=1:ppn=1 defaulting to mem=4571MB

 

Please review the Owens documentation:

    https://www.osc.edu/supercomputing/computing/owens

 

Feel free to contact oschelp@osc.edu if you have any questions.

 

8485416.owens-batch.ten.osc.edu

rm: cannot remove ‘outputs/*.root’: No such file or directory

NOTE:  You may want to change the following in your job for Owens

 

    nodes=1:ppn=1 defaulting to mem=4571MB

 

Please review the Owens documentation:

    https://www.osc.edu/supercomputing/computing/owens

 

Feel free to contact oschelp@osc.edu if you have any questions.

 

8485417.owens-batch.ten.osc.edu

rm: cannot remove ‘outputs/*.root’: No such file or directory

chmod: changing permissions of ‘/fs/project/PAS0654/BiconeEvolutionOSC/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop/AraSimTestJob.sh’: Operation not permitted

chmod: changing permissions of ‘/fs/project/PAS0654/BiconeEvolutionOSC/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop/Antenna_Performance_Metric/AraOut_%num%.txt’: Operation not permitted

chmod: changing permissions of ‘/fs/project/PAS0654/BiconeEvolutionOSC/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop/Loop_Functions/XF_Loop3.sh’: Operation not permitted

chmod: changing permissions of ‘/fs/project/PAS0654/BiconeEvolutionOSC/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop/Loop_Functions/#Part_C.sh#’: Operation not permitted

chmod: changing permissions of ‘/fs/project/PAS0654/BiconeEvolutionOSC/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop/Loop_Functions/#Part_A.sh#’: Operation not permitted

chmod: changing permissions of ‘/fs/project/PAS0654/BiconeEvolutionOSC/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop/scriptEOFiles/AraSimCall.sh.e957233’: Operation not permitted

chmod: changing permissions of ‘/fs/project/PAS0654/BiconeEvolutionOSC/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop/scriptEOFiles/AraSimCall.sh.e957230’: Operation not permitted

chmod: changing permissions of ‘/fs/project/PAS0654/BiconeEvolutionOSC/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop/scriptEOFiles/AraSimCall.sh.e957234’: Operation not permitted

chmod: changing permissions of ‘/fs/project/PAS0654/BiconeEvolutionOSC/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop/scriptEOFiles/AraSimCall.sh.o957224’: Operation not permitted

chmod: changing permissions of ‘/fs/project/PAS0654/BiconeEvolutionOSC/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop/scriptEOFiles/AraSimCall.sh.e957220’: Operation not permitted

chmod: changing permissions of ‘/fs/project/PAS0654/BiconeEvolutionOSC/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop/scriptEOFiles/AraSimCall.sh.o957220’: Operation not permitted

chmod: changing permissions of ‘/fs/project/PAS0654/BiconeEvolutionOSC/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop/scriptEOFiles/AraSimCall.sh.o957234’: Operation not permitted

chmod: changing permissions of ‘/fs/project/PAS0654/BiconeEvolutionOSC/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop/scriptEOFiles/AraSimCall.sh.o957227’: Operation not permitted

chmod: changing permissions of ‘/fs/project/PAS0654/BiconeEvolutionOSC/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop/scriptEOFiles/AraSimCall.sh.o957233’: Operation not permitted

chmod: changing permissions of ‘/fs/project/PAS0654/BiconeEvolutionOSC/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop/scriptEOFiles/AraSimCall.sh.o957218’: Operation not permitted

chmod: changing permissions of ‘/fs/project/PAS0654/BiconeEvolutionOSC/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop/scriptEOFiles/AraSimCall.sh.e957224’: Operation not permitted

chmod: changing permissions of ‘/fs/project/PAS0654/BiconeEvolutionOSC/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop/scriptEOFiles/AraSimCall.sh.e957231’: Operation not permitted

chmod: changing permissions of ‘/fs/project/PAS0654/BiconeEvolutionOSC/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop/scriptEOFiles/AraSimCall.sh.e957226’: Operation not permitted

chmod: changing permissions of ‘/fs/project/PAS0654/BiconeEvolutionOSC/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop/scriptEOFiles/AraSimCall.sh.o957221’: Operation not permitted

chmod: changing permissions of ‘/fs/project/PAS0654/BiconeEvolutionOSC/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop/scriptEOFiles/AraSimCall.sh.o957223’: Operation not permitted

chmod: changing permissions of ‘/fs/project/PAS0654/BiconeEvolutionOSC/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop/scriptEOFiles/AraSimCall.sh.e957217’: Operation not permitted

chmod: changing permissions of ‘/fs/project/PAS0654/BiconeEvolutionOSC/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop/scriptEOFiles/AraSimCall.sh.e957227’: Operation not permitted

chmod: changing permissions of ‘/fs/project/PAS0654/BiconeEvolutionOSC/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop/scriptEOFiles/AraSimCall.sh.e957218’: Operation not permitted

chmod: changing permissions of ‘/fs/project/PAS0654/BiconeEvolutionOSC/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop/scriptEOFiles/AraSimCall.sh.e957223’: Operation not permitted

chmod: changing permissions of ‘/fs/project/PAS0654/BiconeEvolutionOSC/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop/scriptEOFiles/AraSimCall.sh.e957225’: Operation not permitted

chmod: changing permissions of ‘/fs/project/PAS0654/BiconeEvolutionOSC/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop/scriptEOFiles/AraSimCall.sh.o957230’: Operation not permitted

chmod: changing permissions of ‘/fs/project/PAS0654/BiconeEvolutionOSC/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop/scriptEOFiles/AraSimCall.sh.o957225’: Operation not permitted

chmod: changing permissions of ‘/fs/project/PAS0654/BiconeEvolutionOSC/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop/scriptEOFiles/AraSimCall.sh.e957222’: Operation not permitted

chmod: changing permissions of ‘/fs/project/PAS0654/BiconeEvolutionOSC/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop/scriptEOFiles/AraSimCall.sh.o957231’: Operation not permitted

chmod: changing permissions of ‘/fs/project/PAS0654/BiconeEvolutionOSC/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop/scriptEOFiles/AraSimCall.sh.o957217’: Operation not permitted

chmod: changing permissions of ‘/fs/project/PAS0654/BiconeEvolutionOSC/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop/scriptEOFiles/AraSimCall.sh.o957222’: Operation not permitted

chmod: changing permissions of ‘/fs/project/PAS0654/BiconeEvolutionOSC/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop/scriptEOFiles/AraSimCall.sh.o957226’: Operation not permitted

chmod: changing permissions of ‘/fs/project/PAS0654/BiconeEvolutionOSC/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop/scriptEOFiles/AraSimCall.sh.e957221’: Operation not permitted

mv: cannot stat ‘*.root’: No such file or directory

Fitness function initialized.

0 

Data successfully read. Data successfully written.

Attendance for today's session: Cade, Julie, Alex M, Alex P, Mitchell, Scott. 

Monday was a holiday, and we will be replaceing that meeting later this week. Today we worked on the following tasks:

Tasks:

• Check continuity of loop
  • Alex M will try another run tonight, and has already requested a virtual desktop. 
• Install Arasim to /fs/projects/PAS0654/BiconeEvolutionOCS to be edited and messed with
  • Cade is testing whether or not this is running within the loop. He had gotten a few errors, and it is not completing. It's submitting the job with the new bash script, but it is not actually running.
    • Found an error. The bash script was not grabbing the correct AraSim directory. More updates to come. 
• Clean bash script
  • Mitchell and Alex M are putting all current functions hard-coded in into their own .sh scrips that are now being called in the main bash script.
• Update Manual
  • Cade and I are updating the manual with new changes. We have officially retired the old version of the manual that was used for runs with Nutau. We have saved that version and uploaded it to the elog here: http://radiorm.physics.ohio-state.edu/elog/GENETIS/17

Right now Elliot and Leo have decided to focus on working with Keith. They will not be contributing to this project for the remainder of the semester. Students have been posting individual updates here: https://www.dropbox.com/home/GP_Antennas/Updates

I've also attached the meeting minutes from the call last week, in order to keep track of updates. 

We are redesigning the way we simulate antennas in our loop. To do this, we changed our simulationPEC macro skeletons and our output skeleton. 

To make this easier, we changed the way we name the files, from i.uan where i is the simulation number, to i_j.uan where i is the antenna and j is the frequency.

We also rewrote XFintoARA.py so it's clean. Note that before there were comments from a previous author claiming it was VERY questionably written. This has now been cleaned up. We also changed the bash script relevant to the above changes.

This change needs to be added to the manual, and until we update this update, we have not done it.

Updated To-Do List:

1. Add ARA constants in plotting code: ie add Veff, theta, R for ARA. This will sit as a constant line for us to compare each evolving parameter to. For example, in the plot Veff vs Generation, we will see a constant line for ARA's Veff to see how it is evolving in time. 

TASK 1: MAKE Veff PLOT. Note: One of our variables in this run is Energy. Unfortunately, we care about how Veff changes with Energy for ARA_actualBicone. This means that before plotting "Veff versus Generation" we need to generate Veff for ARA_actualBicone at the energy we are assigning the run. 

STEP 1: Make plotting code to plot "Veff vs Generation". 

• You can follow the way we make LRPlot.py.
  • However, in this case, we will need to take the files /fs/project/PAS0654/BiconeEvolutionOSC/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop/Run_Ouputs/$RunName/AraOut${gen}_${i}.txt -- ie the AraSim outputs for each individual in ALL generations. ie Gen 0 - ${gen}  -- INCLUDING the one we created in Gen 1 for ARA_actualBicone -- and plot Veff from those. 
  • We also want to plot error bars with Veff. The error bars for each are labeled "and Veff (water eq)" at the bottom of these AraSim output files we are already reading in for Veff. 
    • Note, to know how to read in Veff, you can look at how we are reading that *exact* line from the AraSim output files in on the fitnessFunction_ARA.cpp code in the Antenna_Performance_Metric directory, since we are doing it there. 
    • Note that since we are plotting "Veff vs Generation", we will either have to read in all the files AraOut${gen}_${i}.txt (up to how may ${gen} we are in) every time -- including the ARA_actualBicone one-- or we can append them all to one .txt file. It's your choice. 
  • Make sure you label the units for each axis every time, and that you have a key! If not, Amy will hate the plot!

STEP2: Add V_eff for ARA_actualBicone as a constat in the newly made "Veff vs Gen" plotting code. Remember, as I mentioned above, we must first put the ARA_actualBicone through AraSim to see Veff at the same energy as the rest of the individuals. Here's what to do:

•  Edit the Bash script to submit a job that runs the gain patterns of ARA_actualBicone through AraSim in the initial gen (gen 0)! 
  • There's a current file that exits for ARA_actualBicone that has the gain patterns of the current antenna, in the correct format for AraSim to run. In the bash script (XF_Loop.sh) we need to create ONE more AraSim job submission for this antenna (as a comparison to what we see with our genetically evolved ones) just in gen 0 (the first generation).
    • This current AraSim input file for ARA_actualBicone exists in BiconeEvolutionOSC/AraSim/Ara_Bicone6in_output.txt
      • This needs to be the input for our .job script. 
• Add lines to your newly made "Veff vs Gen" plotting code that reads in the Veff output for this ARA_actualBicone AraSim output file (whatever you've chosen to name it when you wrote it into the job script. 

 STEP 3: Move the ARA_actualBicone output file into the $Run_Name directory so it is not overwritten ever. 

• We will want to make sure this is not overwritten after the run ends, and we start a new one. 
• We will also be using it for plotting "Veff vs Generation". 

STEP 4: Correct the "wait" command for the AraSim job submissions. 

• The "wait" command, on the first iteration, will have to wait for $NPOP+1 now-- not $NPOP.

TASK 2: MAKE "Theta vs Gen" Plot.  

STEP 1: Make plotting code to plot "Theta vs Generation". 

• You can follow the way we make LRPlot.py; however, MAKE SURE YOU LABEL AXIS WITH UNITS, AND HAVE A KEY.
• This will be exactly the same as this other code, except you will need to:
  1. Read in theta from your generation DNA files
  2. Add the ARA_actualBicone constant to the code as a reference point

TASK 3: Fix "L vs Gen" and "R vs Gen" plot.  

STEP 1: Correct both plots to have labeled the units for each axis every time, and a key. This is plotted in the "LRPlot.py" code. 

STEP 2: Fix our 'R' values. I do believe we are plotting the incorrect "R". Here is what I think is happening:

• We need to double-check and see what ARA defines "R" as on their current antennas. 
  • Check with Amy?
  • What does Brian Clark's thesis say?
  • Can you find a paper that shows a picture of how they define it?
• If we are wrong, this means we need to change the "R vs Gen" part in the "LRPlot.py" code so that we:
  1. Read in R, L, and Theta for each generation.
     • See how this is already done for reading in R, and repeat it for L and theta. 
  2. Do trig to calculate the same "R" as ARA_actualBicone defines. See picture below!
  3. Store that newly calculated "R" as an array, and plot THAT.
  4. Don't forget to add the ARA_actualBicone Radius as a constant line. 

TASK 4: See PDF for other items!

We are getting to the stage where we would like to verify each step. It has been requested by Amy that we verify them visually. Here are the plots we need to make. 

1. Add plotting for L & R parameters (note that we already have similar software to this from the dipole check. You can find this in the GitHub from the dipole test here: https://github.com/hchasan/XF-Scripts). This should be between A, and B in the loop drawing on the manual. (Done)
2. Plot the distribution of theta and phi of incoming signal. This should be after D in the loop drawing on the manual. 
3. Plot the distribution in cartesian coordinates of the neutrino interaction position. Maybe we can also plot the distance between the interaction point and the antenna. This should be after D in the loop drawing on the manual. 

Finally, as a tasks not to do with plotting, we need to make sure we don't lose any data as we go. We want to make sure that we keep tagged or labelled which individuals survive, and save the data flagged for those as we go. 

Update to previous ELOG post for Effective Volumes and Errors for AraSim simulations for antennas from Paper_Run and Curved_Sides.

AraSim version compiled ~11/2021 and the (old) GENETIS version

Here are the column definitions:

• The first column indicates the antenna being described. For example, XF_model_21_20 was the 20th individual in the 21st generation for the run discussed in the paper. Curved_side_1 was the best performing individual from the curved run done in February 2022. Curved_Side_1_Quad_Zero indicates the curved individual with its quadratic term set to 0. Curved_Side_1_Straightened indicates the curved individual with the quadratic term set to zero and the linear term modified so as to keep the outer radius the same as it originally was (that is, for Curved_Side_1).
• The next three columns are the measurements for that individual.
  • The first one idicates the mean effective volume of the runs done for that individual. There were ~300 runs, each of 10,000 neutrinos.
  • The second column indicates the standard deviation of the mean (the previous column). This was measured by taking the standard deviation of the effective volumes (the line that says test Veff(ice) in the AraOut file) from all 300 runs (that is, Sigma(mean - x_i)/Sqrt(300)).
    • example of line:
      • Veff(ice) : 5.25337e+09 m3sr, 5.25337 km3sr
  • The final column indicates the average of the errors on the effective volume measurements from the AraOut files (the line that Veff(water eq.)) divided by Sqrt(300) for each of the 300 jobs (that is, Sigma(x_i/Sqrt(300))/300).
    • example of line:
      • And Veff(water eq.) error plus : 0.552722 and error minus : 0.552722
         

Note: Previously, the best curved individual (according to the run data) had been excluded (instead, we had the 2nd-6th best individuals). This has been corrected, with the order corrected. 

Also find attached a file called full_resimulate_DNA.csv . It contains the DNA for the top five curved individuals, the DNA for those same individuals with their quadratic term set to 0, and the DNA for those individuals with the quadratic term set to zero and the linear term modified to maintain the same outer radius as in the original run. 

Finally, the three images show the original best curved bicone (Curved_Side_1), the best curved bicone when its quadratic gene was set to 0 (Curved_Side_1_Quad_Zero), and the best curved bicone when its quadratic gene was set to 0 *and* its linear gene was changed to keep the same outer radius (Curved_Side_1_Straightened).

We have chosen XF_Model_23_08 to be the one actually built for testing. This was decided based on the data presented here. XF_Model_23_08 was the highest scoring individual in the paper run. Because the paper data was generated using the old version of AraSim, it was decided that using those results would make the most sense for which antenna to build. This would also mean that we can compare directly to the best individual in the paper. This individual was chosen over the curved sided inviduals because it has a higher fitness score, both when using the old version of AraSim and the new version.

Here is a list of things we agreed to do before starting a new PUEO run.  Feel free to add notes here as things are done and we will revisit the list next week.

Change range of the height variable to 0.75-1.75m

Start with all bad individuals so that we can be sure and see growth - all at 0.75m?

Make sure we are using the ratios in the GA that we think we are

If we are using reproduction, are we combining results when individuals are resimulated to reduce errors?  I made a program to accomplish this and created a pull request on the PUEO Github (Dylan)

Investigate whether there are obvious places to speed up PUEOSim

Consolidate the GA to one branch

Understand what is happening with cross-pol, make sure we're not gettting Veff=0 when using XF-generated cross-pol response

Edit: Have the xmacros use the realized gain instead of the idealized gain, which allows for the design to build in matching

Types of runs I need (May need some other people to help me run these):

1. Optimization runs

   1. Non-error

      1. Search over combinations of selection methods and genetic operators

         1. Fix Sigma at 10%.

      2. Do this for 10 runs for each 

      3. Re-run the best run for 100 tests to see if the results agree with 10 test results

      4. This will be the main talking point

      5. Save the plot for the best combination (proxy and metric)

   2. Error

      1. Re-run the best run for 100 tests with 3 different error amounts (0.1, 0.2, 0.3)

      2. We will compare these results to the non-error run to talk about how errors may affect consistency/speed. 

      3. Save an example plot of an error test for each error (both metric and proxy score)

   3. Population

      1. Re-run the best run for 100 tests with 3 different population sizes (100, 500, 1000)

      2. We will compare these results to the non-error run to talk about how population size affects consistency/speed. 

      3. Save an example plot of a population test for each size (just proxy score)

2. Demonstration runs (single runs, fitness score, no error population 100)

   1. Crossover only

   2. Mutation only

   3. Reproduction only

   4. Injection only

Copy this slide deck to use when presenting on run updates!

https://docs.google.com/presentation/d/1Tk-B1QbFTP_5pQZovfn0_wbTswZTmOrJURpi374xJWo/edit?usp=sharing

​Today I spent some time looking at task 1 from the update here: http://radiorm.physics.ohio-state.edu/elog/GENETIS/20. Below are my updates. 

Task-- Fix error at fitness function: 

Currently, the loop stops at the fitness function calculation, and aborts (program fitnessFunction_ARA.cpp). The following are the things I have tried.

1.Run this outside of the loop: Julie and Alex did this 11/19/19. It gave a segmentation fault. UPDATE: HERE'S WHAT I DID TO CHECK THIS TODAY:

(a) Checked using cout: I used some cout commands to print sections looking for errors. It seems we were not properly giving it arguments when running it outside of the loop. Since it's running outside of the loop, these variables are not initialized. Thus, to run you need to type: ./fitnessFunction.exe $NPOP $AraOut_i.txt

• For example: ./fitnessFunction.exe 1 AraOut_1.txt

Status: The output file is verified! Running this way does properly generate our output file named fitnessScore.csv. I've double-checked this file, and it is properly being filled. At the moment it is giving a Veff of 0, thus no valid fitness score; however, this should be fixed automatically once we get real outputs from AraSim without a Veff of 0. 

 2. Check file location: See if fitnessScore.csv is being located in the correct place when the genetic algorithm initializes the next generation. 

(a) Searched bash script for the final location of fitnessScores.cvs: 

In lines 298, 299 of the bash script, it is doing the following 

cp fitnessScores.csv "$WorkingDir"/Run_Outputs/$RunName/0_fitnessScores.csv

mv fitnessScores.csv "$WorkingDir"

This is taking the .csv output generated from fitnessFunction_ARA.cpp, and is moving it to the run name directory and renaming it based on the generation -- ie first gen is 0_fitnessScores.csv-- so that data isn't overwritten. Then, for some reason, it moves the original copy -- ie fitnessScores.csv-- to the location /fs/project/PAS0654/BiconeEvolutionOSC/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop

In line 374 we then run our genetic algorithm again. 

./roulette_algorithm.exe cont $NPOP

This does not seem to take in the input file as an argument.

(b) Checked roulette_algorithm.cpp for the location it accesses fitnessScores.csv:

I checked if it was hard-coded into the roulette_algorithm.cpp script. I found the following lines.

 // Now we need to read the fitness scores:                                                                                                                                                    

        ifstream fitnessScores;

        fitnessScores.open("fitnessScores.csv");

        string fitnessRead[NPOP+2];

        for(int i=0;i<(NPOP+2);i++)

    {

                getline(fitnessScores,fitnessRead[i]);

                if(i>=2)

                {

                        fitness[i-2] = atof(fitnessRead[i].c_str());

                        if (fitness[i-2]<0)

                        {

                                fitness[i-2] = 0; // If the fitness score is less than 0, we set it to 0 to not throw things off.                                                                     

                        }

                }

    }

        fitnessScores.close();

}

This means that our script roulette_algorithm.cpp is grabbing the fitnessScores.csv from its current directory-- which is accurate due to the line (mv fitnessScores.csv "$WorkingDir") referenced above in the bash script. Something we could verify here is to make sure this function is properly working by having it cout elements from the fitnessScores.csv within this function and to run the roulette algorithm separately. We could do this now, however, we don't have real outputs in fitnessScores.csv anyway, since we are getting a veff=0 for AraSim temporarily. 

In the meantime, I've added the following line after section E (the section that runs fitnessFunction_ARA.cpp, and generates/moves/copies fitnessScore.csv):

echo 'Congrats on getting a fitness score!'

This should check to be sure that it runs until after this point. I've also added the following line after section E (section that plots the fitness score versus gen):

echo 'Congrats on getting some nice plots!'

This should check to be sure that it runs until after this point. Note that I have added these lines to both our original bash script, and the one Alex M. is editing to turn every section into a function. I have added these lines to the .sh scripts containing the functions to be sure the function scripts for these tasks ran completely.  Once we have real AraSim outputs, we should be able to create some cout lines in the instructions highlighted in yellow above. 

3. Look at how it's calculating the fitness score.

I have not checked this yet. However, I do not think it is likely this is the issue. I will check later though. 

Right now the following people are working on this project:

Julie Rolla, Cade Sbrocco, Scott Janse, Alex Patton, Eliot Ferstl, Abassi Brown, Evelyn Shank, Sophie Dunlap

Today we met for our first working meeting. Scott, Alex, and Eliot were asked to speed up and optimize AraSim. Sophie and Evelyn are working on editing the .csv file with the beam patterns coming out of XF. Currently it's written in a different format than AraSim takes in. Abassi is helping Evelyn and Sophie by giving them a sample program helping reorganize the cells in a .csv file. Since there's confusion on how to optimize AraSim, we've started dissecting the software so we can understand how it works. The following are questions on the setup.txt document. 

Setup.txt

What is "timestep"? Could we increase the timestep? Is this timestep big or small compared to 2 GHz?

Answer: Timestep is the unit of discrete time the computer operates at. The timestep by default is set to 2 GHz, or 5 * 10^-10 seconds. We might be able to make up some cheap time by lowering that but would have to check if it makes the data less accurate.

What is "TRIG_WINDOW"? Is this a "coincidence"? Meaning if this signal is picked up within a certain time between stations it's considered the same event?

What is "V_MIMIC_MODE"? Comments in Setup.CC says "// default : 0 - write out all chs where global triggered bin is center of the window"

Answer: Not sure exactly what it does but we can't change it from default as the other values are for the test bed mode only.

Random Question from Setup.txt, but relevant for other things. What is "testbed mode"? Answer: The "testbed" is the original test station only 40m under. This mode is for if we do detector=3 in the first part of the setup.txt file; thus, we won't use this. 

What's the difference between "DETECTOR" and "DETECTOR_SIMULATION"? We found "DETECTOR_STATION" in Settings.h. The comments say "// for DETECTOR=4, indicates the single station to be simulated // 0 = testbed, 1 = A1, 2 = A2, 3 = A3"

In this file, there's an ANTENNA_MODE variable. You can look at what it does in Settings.h. In this file, it says that you can decide whether to use a bicone antenna response or "a different response for the top vpol antennas". How does this function work? What does it mean by "top vpol antennas"? Someone should look up the ARA antennas, and look into this function. 

Our goal is to meet and work as a group twice a week. Collaboration in work should help us progress faster. 

After discretizing and straightening the sides of the curved antenna that performed the best (Generation 13, individual 84) and running the antenna through AraSim, the following results and physics plots were generated. 

We ran valgrind (a profiler) on AraSim to find out where it was spending most of its time, we ran it with callgrind to give an output file of the time and opened that file with qcashegrind to display it in a legible format. This let us know that the function Param_RE_Tterm_approx was taking up about 75% of the time because it was being called hundreds of millions of times. Now more specifically the standard math functions pow() and exp() were being called in this function every time and took up about 44% and 20% respectively. They were called about the same number of times as Param_RE_Tterm_approx but they are only called in the else block in an if-else statement.

JULY 31st Update:

I used the C++ library future and set up iterations of a loop to run asynchronously as they do not rely on each other's calculations. This launches a seperate thread whenever I call one. The struggle with this is that launching a thread and getting its results takes time as well, so there had to be enough calulations in a thread in order for it to make up any time. The purpose of the function GetVm_FarField_Tarray is to create arrays of doubles called Earray and Tarray and those are the only values that get altered in this function. Tarray is created with simple calculations so I left that out of threads but making Earray calls the other function that takes up so much time.

I edited the files signal.cc and signal.hh to work with 4 threads and ran 10,000 neutrinos with 4 cores. The cpu time was 1:48:41 but the real walltime it used only took 00:54:40. The time these tests for 10,000 neutrinos with base AraSim normally take anywhere from 1:35:00 to 1:45:00 or in that range. It should still work with less threads available it would just run slower and probably at the cpu time, which might be slightly slower than base AraSim but as soon as more threads are available it runs much faster as we see here. I still need to check for the accuracy of these edits, the number passed is in a reasonable range, but I will need to investigate further by giving the exact same neutrinos to both programs and making sure their outputs are identical.

I also have some further ideas to speed it up even more. Currently every iteration of the loop calls a thread but what would be faster would be to either

A: Set it up so it can call more threads at a time and thus go faster, which would be an easy edit but only work if more cores are given

B: Set it up so the current threads run more than one iteration at a time, I've started experimenting with this but having trouble making sure that everything involving pointers stays safe, but the benefit of this is that the threads would get called and joined significantly less and would cause a great speed increase without adding more threads.

All these developments are new after me spending a while trying many different ways to speed it up or run parts in parallel like this, so even if this isn't accurate or valid I believe I would be able to make it so with more time and work.

Attached I have a zip that contains the two files I edited as well as an output file from my first test. I will continue to test and improve this but am unable to work on this again until August 19th so I wanted to share my current status before I am unavailable.

Because of issues we were having with XF when running at the command line, we've shifted to running through a Pitzer VDI (see here: https://ondemand.osc.edu/pun/sys/dashboard/batch_connect/sys/bc_desktop/vdi/session_contexts/new).

The VDI will give you a mirror of a desktop running on OSC's side, so things like XF will run much faster. You can open a terminal by clicking the black box at the bottom of the screen and then run the loop like normal (./Asym_XF_Loop.sh). This has the added benefit of making it possible to exit out of your browser, close your laptop, or turn off your computer without interrupting the run, since the process is running on OSC's side rather than piping to your computer.

For PAEA Algorithm:

Part I: Complete as soon as the run starts

Run details: Please answer all of the questions below!

• Run Type
  • Answer here whether or not it's AREA (Gain pattern evolution) or PAEA (Bicone evolution)
• Run Date
  • Add answer here
• Run Name
  • Add answer here
• Parameters evolved
  • Add answer here
• Why are we doing this run?
  • Add answer here about what we are testing
• What is different about this run from the last?
  • Add answer here with info on what we are doing differently from the last run
  • Example: We are testing a new operator, we are changing the percentages of each selection method used, etc.
• Symmetric, asymmetric, linear, nonlinear (what order):
  • Add answer here (say N/A if this is an AREA run)
• Number of individuals (NPOP):
  • Add answer here
• Number of neutrinos thrown in AraSim (NNT):
  • Add answer here
• Operatiors used (% of each):
  • Add answer here
• Selection methods used (% of each):
  • Add answer here
• Are we using the database?
  • Add answer here. This can be found in the main bash script within the variables section. (Answer N/A for AREA run)

Please upload the text file with all run details before closing this!

Part II: Complete as soon as the run ends

Results: Once this run completes, please upload the plot(s) to this post as an attachment, as well as a general explanation of the results. 

• Summary and comments on results
  • Add thoughts on what we should do next here!
  • Example: "It seems like we are seeing minimal evolution. We should take a step back and try to see why we don't see improvement. Once we trouble shoot, we can start a new run to investigate."
  • Example 2: "Run looks good! Our best individual is in generation #12. Attached are the CAD drawings of that individual."

**Upload all plots, run parameter text files (file that has run settings saved), CAD models of best indivduals, etc**

For AREA Algorithm:

Part I: Complete as soon as the run starts

Run details: Please answer all of the questions below!

• Run Type
  • Answer here whether or not it's AREA (Gain pattern evolution) or PAEA (Bicone evolution)
• Run Date
  • Add answer here
• Run Name
  • Add answer here
• Parameters evolved
  • Add answer here
• Why are we doing this run?
  • Add answer here about what we are testing
• What is different about this run from the last?
  • Add answer here with info on what we are doing differently from the last run
  • Example: We are testing a new operator, we are changing the percentages of each selection method used, etc.
• Single frequency run or run with broadband frequency dependence:
  • Add answer here
• Number of individuals (NPOP):
  • Add answer here
• Number of neutrinos thrown in AraSim (NNT):
  • Add answer here
• Operatiors used (% of each):
  • Add answer here
• Selection methods used (% of each):
  • Add answer here
• Any other things to note?
  • Add answer here

Please upload the text file with all run details before closing this!

Part II: Complete as soon as the run ends

Results: Once this run completes, please upload the plot(s) to this post as an attachment, as well as a general explanation of the results. 

• Summary and comments on results
  • Add thoughts on what we should do next here!
  • Example: "It seems like we are seeing minimal evolution. We should take a step back and try to see why we don't see improvement. Once we trouble shoot, we can start a new run to investigate."
  • Example 2: "Run looks good! Our best individual is in generation #12. Attached are the CAD drawings of that individual."

**Upload all plots, run parameter text files (file that has run settings saved), gain patterns of the two best and two worst individuals, etc**

This run was started on 10/23/2020. The purpose was to attempt to demonstrate evolution by beginning from 50 identical individuals in the initial generation (which had previously produced a low score).

This run was the first time we removed the penalty on fitness scores for antennas which exceeded the borehole radius. It was also the first time we increased the number of neutrinos thrown up to 300,000. 50 fully asymmetric individuals were evolved over 25 generations.

There was a 50/50 split for roulette/tournament selection and 75%/10%/15% for crossover/reproduction/immigration. While the evolution was somewhat flat, we do believe we demonstrated evolution because the average score rose as the run evolved. 

(Note that the final generation was interrupted but the plot was still made, hence the drop to 0 for all scores on the plot)

This was a symmetric run that began on 9/11/2020. This run used fewer neutrinos to determine how significant the number would be on the performance of the evolution. 30,000 neutrinos were used per individual.

10 individuals were evolved over 35 generations, using 100% roulette selection and the old mutation algorithm, where genes had a 40% chance of mutating. Mutations changed the gene by reselecting the value from a guassian with a mean and sigma equalt to the mean and sigma of that gene's value in the generation.

This run was conducted beginning on August 31, 2020.

Both the angle and length were asymmetric for the two cones. It was run with 10 individuals over 42 generations using 150,000 neutrinos. A penalty was still implemented on bicones exceeding the borehole width.

As this was before the substantial rewrite of the genetic algorithm, 100% of selection was done using roulette. Genes had a 40% chance to "mutate," which was done by selecting a new value from a guassian, with a mean and sigma equal to the mean and standard deviation of that gene's value in that generation.

This run followed the first symmetric run. It had an asymmetric length, but the opening angle and inner radius were kept symmetric. 

10 individuals were evolved over 17 generations using 100,000 neutrinos. All selection was done through roulette, and all individuals were formed through crossover and the old mutation method (where individual genes had a 40% to be mutated, and mutated genes were chosen from a gaussian based on the mean and standard deviation of that gene's value in the generation).

This is considered to be the first run of "good" data. Prior to this run, we ran with few neutrinos and had errors which needed to be resolved in AraSim and with the way we handled XFdtd's simulation results being passed to AraSim. Those were resolved in the summer before this run.

This was a symmetric run of 10 individuals over 15 generations using 100,000 neutrinos per individual. This preceded the substantial modifications made by Ryan to the GA. All individuals were formed through crossover, but there was a probability of 40% for genes to be mutated. In this case, mutation was a complete change to the gene, though it was based on the average value of that gene in the generation. There was no reproduction or mutation. All individuals were selected through roulette.

Green individuals indicate that there was no penalty applied, while red individuals had a penalty factor multiplied by their effective volumes. The penalty was of the form exp(-(R-7.5)^2), where R is the outer radius, in cm, of the bicone.

This is the run that is discussed in the GENETIS paper submitted to Physical Review D in December of 2021. It was conducted beginning on March 19, 2021. The preprint can be found here. It ran for 31 generations with 50 individuals per generation. Each individual was run through AraSim for 300,000 neutrinos.

The ratio of generative operators was: 72% crossover, 22% immigration, 6% reproduction. 

The ratio of selection operators was: 80% roulette, 20% tournament, 0% rank.

The highest scoring individual had a fitness score of 5.24. However, when it was rerun with many more neutrinos (3*10^7), it had a score of 4.90, an 11% improvement over the ARA bicone.

This run was conducted before the introduction of the automatic run_details.txt generator. Instead, there is a file called Run_Notes.txt (attached) which details some of the errors that were encountered. These errors were remedied during runs by removing the offending AraOut file (which had too high of a fitness score due to a weight being calculated to be greater than 1, which shouldn't be possible) and have since been prevented by modifying the data file AraSim uses to calculate the weights.

We began a new run of the curved sided GA. Previously, we conducted a run just like this in July of 2022 (titled 2022_07_15_Latest_Greatest). In that run, we made substantial updates to the loop, including using the latest version of AraSim. However, we discovered after the run (which had a very flat fitness score growth curve) that we had not modified AraSim to take in gainfiles appropriately (so it was just evaluating the ARA bicone repeatedly). We've fixed this issue and a few more (see the github repository for the issues that exist and have been resolved) and are redoing this run. The run details will look much the same as that one and are visible in the attached file. Here is the location of the directory where the data for this run will be stored: /fs/ess/PAS1960/BiconeEvolutionOSC/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop/Run_Outputs/2022_12_29

For a list of changes made in this run compared to previous ones, see this ELOG post on the previous run: http://radiorm.physics.ohio-state.edu/elog/GENETIS/186

We have started a new run. Here is the directory: /fs/ess/PAS1960/BiconeEvolutionOSC/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop/Run_Outputs/2022_07_15_Latest_Greatest

This run is a substantial overhaul. It is a curved run using the latest parameters from Ryan's optimization loop. Here is a list of important changes:

1. The newest version of AraSim has been implemented.
2. The minimum length has been decreased to 10 cm per side.
3. The GA is recording the parents and operators used to generate individuals.
4. The number of individuals has been increased to 100 per generation.
5. The number of neutrinos per individual has been increased to 600k.

This run may take longer than previous runs due to the increased number of individuals. There may need to be modifications made to resolve or work around AraSim errors that delay the loop (due to the auto-resubmit function). Find the details of the run attached.

This run was a short run (stopped due to resource limits) that was conducted to test the minimum length constraint that had been applied in the past. Previously, the minimum length was set to be 37.5 cm for each cone; in this run, it was lowered to 10 cm. It ran for 60 generations with 50 individuals per generation, each evaluated with 300,000 neutrinos. To compare to the paper run, this run was performed using the asymmetric algorithm, meaning each individual had 6 genes (length, inner radius, and opening angle for each cone). The initial generation began with identical individuals, with genes matching the genes of the best performing individual mentioned in the GENETIS paper.

The ratio of generative operators is as follows: 72% crossover, 22% immigration, 6% reproduction. Mutation was also used, with 1% of genes produced by crossover mutated by adding a number chosen from a Gaussian centered at 0 with a width of 5% of the gene's value.

The ratio of selection operators was: 20% roulette, 80% tournament, 0% rank, 0% elite.

Attached are the run details, violin plot, and "rainbow" plot.

This run was started on February 2, 2022. It is a full run of 50 generations with 50 individuals per generation using the quadratic version of the loop. This means that each individual is defined by 8 genes (length, inner radius, linear coefficient, and quadratic coefficient for each cone). Attached is the txt file run_details.txt that is automatically generated when the loop is run. Each individual was run for 300,000 neutrinos.

This run used the usual ratio of generative operators: 72% crossover, 22% immigration, and 6% reproduction. It also used the proper mutation function: 1% of genes created through crossover were mutated by adding a number chosen from a Gaussian distribution centered at 0 with a width of 5% of the gene's value (MUTATION WAS NOT USED BUT WAS AVAILABLE).

This run was the first full run in which the rank selection operator was used. The ratio of selection operators was: 0% roulette, 10% tournament, 90% rank, 0% elite. 

This run used the script fitness_check.py to average the scores of identical individuals that appeared across multiple generations. This gives us a more accurate measure of those individuals' scores. It may also explain why this run appears so flat (on the violin plot): fluctuations in the scores should die down as repeated individuals have more accurate scores, and newly generated individuals that perform highly regress to their mean (actual) score when they are reproduced/recreated through crossover. This has led us to question our GA parameters, as Audrey and Autumn demonstrated that roughly 1/4 individuals in the run are repeat individuals.

Here are the changes that have been made in this (and the preceding) run relative to what we have been doing before.

1. The newest version of AraSim has been implemented.
   1. This was done because AraSim has gone through significant changes since 2019 that make it more accurate and makes it run faster.
2. The minimum length has been decreased to 10 cm per side.
   1. This was done because we previously tested shorter lengths for a response to the reviewer when we submitting the paper to Phys Rev D. Previously, we thought that going below 37.5 cm would lead to unreliable results, but we are reassured that that is not the case now.
3. The GA is recording the parents and operators used to generate individuals.
   1. This should allow us to look back and make a "history" of the evolution by seeing where genes come from in each generation.
4. The number of individuals has been increased to 100 per generation.
   1. Ryan demonstrated that a higher number of individuals does lead to a substantial improvement in the speed at which the evolution converges. 
      1. We previously thought that it did not make a big difference. However, that was done by measuring the average maximum fitness score. If we measure how many generations it takes for the evolution to reach a benchmark score, we find that more individuals helps significantly.
5. The number of neutrinos per individual has been increased to 600k.
   1. This was done to make the fitness scores more precise. At 300k neutrinos, we expect an error of around 0.2. This will cut things down closer to 0.1, which may help us decrease the number of outlier measurements without increasing run time too much because of the AraSim speed up.

This run may take longer than previous runs due to the increased number of individuals. There may need to be modifications made to resolve or work around AraSim errors that delay the loop (due to the auto-resubmit function). Find the details of the run attached.

The following is a run summary of an AREA test using 100 individuals per generation, without the linear frequency dependence on the gain, with the frequency fixed (i.e. the same gain pattern is produced for all frequencies). This run was a preliminary test of the AREA algorithm with a larger NPop (100 individuals per generation) after seeing promising results in the previous test that only used 20 individuals per generation. The percentages of selection methods/operators used attempted to mimic the optimal percentages used for the PAEA Bicone Loop, but because the algorithms do not set these in the same manner, these are highly unlikely to be the best percentages to use. The results show a very quick plateau in fitness score with a loss in diversity of solutions after a low number of generations, potentially meaning that a poor breakdown of selection methods/operators was chosen. This result seems to underscore the need for a test loop-style optimization for AREA to determine the best breakdown of selection methods and operators to use for AREA.

Run details:

• Run Type
  • AREA
• Run Date
  • January 18, 2023
• Run Name
  • 20230118breynoldsrun_noLinearDependAREATest_38RCO_2RM_58TCO_2TM_150000NNU_2Seeds
• Why are we doing this run?
  • This test increased the number of individuals to 100 and attempted to mimic the percentages of selection methods and operators found to be optimal for the PAEA loop.
• What is different about this run from the last?
  • The previous test showed slow/minimal evolution after ~12 generations, and it was speculated that this was due to a small NPop of 20 individuals. This test uses an increased NPop of 100 individuals.
• Symmetric, asymmetric, linear, nonlinear (what order):
  • N/A
• Number of individuals (NPOP):
  • 100
• Number of neutrinos thrown in AraSim (NNT):
  • 300,000 Total (150,000 NNU x 2 Seeds)
• Operatiors used (% of each):
  • 96% Cross-Over, 4% Mutation
• Selection methods used (% of each):
  • 40% Roullette, 60% Tournament
• Are we using the database?
  • N/A

Results:

• Summary and comments on results
  • The fitness score quickly plaeaued and diversity of solutions was lost, potentially due to a unsuccessful choice of selection method and opterator percentages.
  • An important future step is to work with Ryan to create a test loop optimization for the AREA algorithm, as it is written differently in how it is passed selection method and operator percentages and requires its own optimization to find the optimal ratios of these. (Work on creating a test loop optimization for AREA is currently underway).

EDIT 4/10/23
Re-uploading gain pattern associated with most fit individual from this run

The following is a run summary of the AREA test without the linear frequency dependence on the gain, with the frequency fixed (i.e. the same gain pattern is produced for all frequencies). This run was a preliminary test of the AREA algorithm after fixing long-standing errors in the algorithm and finding that the initial constaints implemented did not work with the frequency dependence on gain. Results from this run did show a "proof-of-concept" that the AREA algorithm seems to evolve single frequencies in a promising way, however a low number of individuals may have stalled the evolution.

Run details:

• Run Type
  • AREA
• Run Date
  • December 26, 2022
• Run Name
  • 20221226breynoldsrun_noLinearDependAREATest_8RCO_8RM_2TCO_2TM_30000NNU_10Seeds
• Why are we doing this run?
  • We tested the AREA algorithm set to evolve at a single frequency in order to see if it would evolve a gain pattern at that frequency within the required constraints.
• What is different about this run from the last?
  • This run only evolves gain patterns at a single frequency, allowing us to investigate the evolved gain patterns and check that the required constraints on the power conservation are met, without yet fixing the constraints function to work with the linear frequency dependence relationship. This is also the longest test run since the AREA algorithm was fixed and gotten into working order.
• Symmetric, asymmetric, linear, nonlinear (what order):
  • N/A
• Number of individuals (NPOP):
  • 20
• Number of neutrinos thrown in AraSim (NNT):
  • 300,000 Total (30,000 NNU x 10 Seeds)
• Operatiors used (% of each):
  • 50% Cross-Over, 50% Mutation
• Selection methods used (% of each):
  • 80% Roullette, 20% Tournament
• Are we using the database?
  • N/A

Results:

• Summary and comments on results
  • The evolution appeared to be successful in that the most fit gain pattern found (attached) seems to make sense with our understanding of the underlying physics. Additionally, after inspecting the outputs, conservation of power was confirmed to be properly constrained. However, in the attached violin plot, the fitness score plateaued quickly and never evolved further, with the best individual coming in generation 12. This was speculated to be due to the small population used in this test (only 20 total individuals per generation).

The reviewer asked us to run the loop again but for just symmetric antenna designs. We have completed that run and have results ready. We still need to resimulate the top five individuals, which I'll update this post with later. I attached the violin plot (though it should be fixed to not let the legend overlap the data). See this previous entry for the details of the run: http://radiorm.physics.ohio-state.edu/elog/GENETIS/198

Requesting an interactive job:

qsub -I -X -l walltime=1:00:00 -l nodes=1:ppn=40

(ppn=24 makes it quicker FYI)

Note that walltime is the length of time you'll need it. You will likely need it for 4+ hours if you are running the loop... Probably longer. Note that before you enter this command into your command line, you must be logged into OSC.

For submitting a job with GPUs (which we will be doing to speed up XF now as of 1/24/20), use this command:

qsub -I -X -l walltime=1:00:00 -l nodes=1:ppn=40:gpus=1:default      OUTDATED

srun -A PAS0654 -t 1:00:00 -N 1 -n 40      CURRENT

I fixed a bug in the loop, so we started another rank test run. Run details in the attached file.

The bug was searching for the generationDNA.csv file in the wrong place, meaning that it wasn't able to copy it to the run directory. That meant we didn't have a record of the generation data in the usual format. I don't think that this explains the flatness, since the the generationDNA.csv file was still created every generation correctly, so the GA knew where it was. But this test now corrects that problem and tests the usage of the rank selection method.

The Physics of Results Plots have been added to the Pueo Loop. The current version of the plotter is built for pueoSim v1.0 and located in ${WorkingDir}/Antenna_Performance_Metric (Hasn't been pulled into the loop directory yet).

The pueoSim v1.0 IceFinal files were missing information on the RF direction and information needed to see an amplitude spectrum. I asked Will, and he said that the new version of pueoSim (v1.1.0) outputs the needed information.

I created an updated version of the plotter, and have a pull request here: https://github.com/osu-particle-astrophysics/GENETIS_PUEO/pull/40

However, before this can be implemented in the loop, we need a way to get errors from pueoSim v1.1.0.

Currently, there is a version of rootAnalysis.py here that can analyze the new root files and output fitness scores and errors, but instead of taking 20 minutes for a generation, it now takes 20 minutes per individual to run.

This is because the update splits the IceFinal file into IceFinal_allTree, IceFinal_passTree0, and IceFinal_passTree1. IceFinal_allTree and IceFinal_passTree1 are of comparable size to the previous IceFinal file, but the passTree0 final is

about 10-20 times larger, causing a slowdown in calculations. If we calculate without this file, it takes about 1 minute per individual, still a bit slower than before.

Ideas to solve this issue:

Don't use the passTree0 files (I have asked Will what they're for, and if we need them. Hopefully he responds after the Holiday.)

Instead of running through a Python for loop, call a C++ script that creates a CSV file the Python program can easily load in.

Submit a batch job to run the analysis in parallel before combining the outputs in the correct format. (We should maybe do this anyways?)

Continue using pueoSim v1.0 (I think we can still retain the Theta graphs for the incoming neutrinos, but there won't be any RF graphs or the possibility of amplitude spectrum plots) 

Attached is a preview of plots the v1.1 plotter is capable of making.

Add weights.

Here are a list of things we wish XF would do. We are keeping this list in hopes to add it to a proposal with them. 

1. Save pictures of CAD drawings of simulations. 
2. Close/quit XF GUI. App.quit doesn't actually work for us. 
3. Suppress the GUI and run solely through terminal. 

We have the PUEO loop in working order, though there are a few errors in accuracy and efficiency. We started a preliminary run with few individuals and neutrrinos as a large scale test that may still give us some useful data. 

For this run, we are only evolving the inner side length (10cm to 25 cm) and height (10 cm to 50 cm). The walls are slanted at 45 degrees, so the outer length is completely defined by those two. The ridges are kept at 6 cm wide and 4 cm from the walls at the base. The curvature is set to 0,1 and the ridges are set to have the same height as the walls. 

At the moment, we are using 24 individuals and 300,000 neutrinos. I'll update this ELOG post with more details from the run as they emerge.

https://docs.google.com/spreadsheets/d/1vvcmjByKfcns0-tbAjtePB8ZVGsXAKxXCfbMc99weMI/edit?usp=sharing Here is the spreadsheet link for the population test. 

Here are plots I made for the meeting on 9/3/21. These plots represent a comparison of the gain and realized gain for the 23rd generation of the run being discussed in the upcoming paper. Here is a list of the plots

• Gain vs realized gain
  • Polar plots of the best individual from generation 23 
• S11/VSWR plots
  • Shows the S11/VSWR over the bandwidth for the best individual

This ELOG post contains plots I made this week for comparing the antennas as they were evolved in the run being discussed in the upcoming paper with those same antennas when using realized gain instead of gain. These plots are preliminary, in that they should be edited before being placed in a paper (for example, VSWR is not in dBi).

Plots:

• Gain vs Realized gain
  • Polar plot showing the gain and realized gain of the best individual from the run in the paper
• VSWR/S11
  • The VSWR of the best individual in the run over the frequency bandwidth
• Gain differences
  • The difference between the gain and the realized gain for the best individual over the frequency bandwidth

Attached is a pdf of physics plots generated for the 9_50, 13_84, 18_89, 19_96, and 29_87 crazy sides individuals along with another containing information on the amount of triggered events and effective volumes for each individual. The crazy sides individuals were simulated for 3 million events using the same Arasim version as is currently in the loop.

Asym Straight Sides (from the paper - Generation 23, Individual 8)

(inner radius, length, opening angle in radians)

2.08711,89.924,0.0161734
0.30175,45.3616,0.0910478

individual found in 

/fs/project/PAS0654/BiconeEvolutionOSC/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop/Run_Outputs/AraSim_Polarity_Fix_2021_03_19/AraSim_Polarity_Fix_2021_03_19.xf/Simulations/001108/Run0001/

Below is an example of our Parents.csv file written by the GA. This file tracks the parents of the individuals of the current generation. The columns and their contained information are as follows:

Current Gen: 

The numbered individual of the current generation.

Parent 1:

The number of the first parent of this individual as read from the previous generation.

Parent 2:

The number of the first parent of this individual as read from the previous generation.

Operator:

The genetic operator that created the individual in that row. 

Currently, the PUEO loop runs pueoSim with 1 pueoSim process per job submitted.

Each of these jobs has 1 node and 8 cores, however, pueoSim only needs a single core to run.

Here is some data I collected by running the same seed of pueoSim with different numbers of cores in the job:

So, we should be able to run multiple processes of pueoSim without losing much performance.

Tests with more cores:

A test comparing the parallelization with the 40000 neutrinos per process we simulate in the loop:

So, these tests suggest we can simulate neutrinos ~20 times faster than currently done in the loop by utilizing more cores per job.

This method of multiple processes per job should be applicable to how we run other simulation software like AraSim as well.

(Note: both of these jobs had comparable queue times of 27 and 31 seconds respectively.

Also, I have tested and seen that the outputted root files are identical for the same seed regardless of if ran with multiple processes at the same time or with processes in serial)

(data found in /users/PAS1960/dylanwells1629/buildingPueoSim/testingouts/times.txt)

How to run multiple processes of pueoSim:

Previous Call:

./pueoBuilder/build/components/pueoSim/simulatePueo -i pueo.conf -o ${PSIMDIR}/outputs/${gen}_outputs/$SLURM_ARRAY_TASK_ID -r $run_num -n $NNT -e $Exp > $TMPDIR/out.txt

Multiple Processes Test Call:

for ((i=0; i<$threads; i++))

do

    ./simulatePueo -i pueo.conf -o $TMPDIR -r $i -n $NNU -e 19 > pueo.out &

done

wait

Instead of just calling peuoSim once per job, we can use the & symbol and a loop to run $threads number of pueoSim processes in the background and then wait for them to finish.

Standard Loop Architecture:

Complete an evolutionary step FOR EACH antenna before continuing on with the next step.

Steps:

1. Run the Genetic Algorithm for the entire population.

2. Run the XF radio simulation for the entire population.

3. Run the neutrino simulation software for the entire population.

4. Run root analysis and plots for the entire population.

However, due to constraints on the amount XF keys we have, we can only run 4-5 XF simulations at a time. 

So, when the first antennas finish their XF simulations, their outputs will simply sit there until EVERY other antenna finished their XF simulations.

New Proposed Architecture:

Complete necessary evolutionary steps as a population, but string together those that don't rely on data from other antennas.

Steps:

1. Run the Genetic Algorithm for the entire population

2. Run the XF radio simulation for each antenna

      - When an XF simulation finishes for an antenna, submit the pueoSim / root analysis jobs for that antenna

3. Run the plots for the entire population

This will allow us to complete most of our pueoSim computation while the XF portion of the loop is still running, cutting down the time between the final XF job finishing and the final pueoSim job finishing.

Test run of this new architecture with 100 antennas, 7,840,000 neutrinos per antenna.

After the final XF job finished, the pueoSim simulations and analysis of outputted root files were completed in 954 seconds,

Comparison Between the Two Architectures (both using job optimization from Elog 232)

Notes on Chart:

Source of data located in /users/PAS1960/dylanwells1629/buildingPueoSim/testingouts/times.txt and /fs/ess/PAS1960/HornEvolutionTestingOSC/GENETIS_PUEO/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop/Run_Outputs/2023_07_24_test5/time.txt respectively)

Time from standard uses the time one of the 250 jobs running in parallel took in my testing of parallelizing processes inside of pueoSim jobs: 250 jobs * (40 * 40000 neutrinos per job) / 100 individuals = 4,000,000 neutrinos per individual)

The New Proposed time includes time spent analyzing the outputted root files to find fitness scores and errors, which would have taken around 100 seconds * population size for its number of neutrinos and files per individual (/fs/ess/PAS1960/HornEvolutionTestingOSC/GENETIS_PUEO/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop/Run_Outputs/2023_07_23_test5/time.txt for data on this number)

So, this new architecture can provide more improvements for the amount and speed of neutrino simulation in the loop on top of the methods discussed in Elog 232.

This architecture could also be applied to see improvements in the Bicone and Hpol loops which are both affected by the limited number of XF keys.

Additional Notes:

1. For this new architecture test, each antenna uses 49 jobs for neutrino simulation instead of 2.5 previously. (49 pueoSim + 1 XF for 50 jobs per antenna, 5,000 jobs per generation)

2. The time for each antenna to submit, queue, and finish its neutrino simulation jobs must be less than the length of the XF job, or the extra time will accumulate for each antenna, losing much of the time benefits. (As long as it is less, the time spent on just pueoSim should be invariant under an increase in population)

3. The number of core hours spent on pueoSim jobs will be roughly the same for the same number of neutrinos as each job is shorter (except for a small contribution of the job overhead taking a higher percentage of total time for shorter jobs)

4. Initially I had thought that maybe queuing pueoSim jobs while running the XF jobs could slow down the queue for the XF jobs. So, I made the loop wait to submit the batch pueoSim jobs until we had space for all of them to be active with the ~250 max jobs per user.

Additionally, while observing my many tests, I didn't notice any correlation between the number of CPU peuoSim jobs in the queue and the number of GPU XF jobs out of the queue.

5. Branch I'm developing this on is here

6. The total time for the test generation was 15.4 hours, which is slightly longer than the ~14 hours from the 2023_05_08 run. However, this test also used double the population size, larger values for the range of antenna heights (on average about 3x taller), and 20x more neutrinos simulated per antenna. So, the actual speed is better than it first looks.

We began a new run for a symmetric bicone antenna. The purpose of this is to satisfy the most recent comments from the reviewer on the paper. We are using the same GA as for the paper with the same parameters. I will update this post soon with more details about this run. There is a run_details file attached, but please wait for me to update this post with more specifics as the version of the GA being used is less connected to the details presented in that file.

The data can be found here: /fs/ess/PAS1960/BiconeEvolutionOSC/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop/Run_Outputs/2023_02_20_Symmetric_Run .

Plots we want to have for PUEO:

FScorePlot2D

Fitness_Scores_RG

VariablePlot (PUEO equivalent of LRT plot)

Veff_Plot (currently the veff is equivalent to the fitness score for PUEO)

Veffectives_RG

Rainbow_Plot 

Violin Plot

avg_freq

gain_vs_freq

polar_plotter

physics of results

Next steps:

Get the Error values out of the pueoSim root output (talking to Will)

Use output uan files from the new two-run XF simulation to test the frequency plots. (/fs/ess/PAS1960/HornEvolutionOSC/GENETIS_PUEO/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop/Run_Outputs/2023_04_21_Test_6)

Implement the physics of results plots into the loop (Talk to Bryan / Dennis)

Get the average Veff/Fitness score from the default gain pattern in pueoSim to compare our results to.

Here are my recommended starting genes for the initial generation of the upcoming run (confirmed that these will produce a working antenna in XF). We want to use the lowest values we can because we expect that a larger (namely, taller) antenna will perform better:

S, H, x0, y0, yf, zf, beta
7.50000,50.0000,1.0000,1.00000,1.00000,50.0000,0.2

• S: Half the side length of the bottom of the antenna
• H: height of the antenna
• x0: distance of the bottom of the ridge from origin
• y0: width of ridge at the bottom
• yf: width of ridge at top
• zf: final height of the ridge (cannot exceed H)
• beta: a curvature parameter for the ridges

Slack: https://gpantennas.slack.com/messages 

elog:http://radiorm.physics.ohio-state.edu/elog/GENETIS/

Github link: https://github.com/mclowdus/BiconeEvolution

DropBox link: https://www.dropbox.com/home/GP_Antennas

Old GitHub for the evolution of the dipole: https://github.com/hchasan/XF_Scripts

This is the plot for the run type that had the best performance in the most recent optimization run that was completed. The most optimized runs from these collections are decided by which run types achieve a chi-squared value of 0.1 in the fewest generations (correlating to about a 0.9 fitness score). However, the genetic algorithm does not use Chi-squared directly, instead trying to maximize the fitness score (which uses a chi-squared in the denominator) that simulates the error we see in arasim. This fitness score is usually maximized at 1.0 but the simulated error can push the score slightly beyond this. This test was conducted with a simulated error of 0.1. This optimization holds the selection methods steady at 2 tournament 2 roulette and 6 rank selected individuals per 10 selection events. The changing parameter values are in the amounts of Reproduction, Crossover, Immigration, and mutation (rate and width) that the GA performs each generation. Each generation uses 100 individuals. In the graph below, the dotted lines are the minimum Chi^2 values at each generation, solid lines are the maximum fitness score. Each color represents a different run using the same parameters and dotted and solid lines sharing the same color represent the same run. The prevailing trend in the tests leading up to this test has been that the most optimized runs have low amounts of reproduction, high amounts of crossover, and they have been pushing the boundaries of the mutation rate and gaussian width of those mutations and this run follows that trend. The ranges of values we tested over were 0-12R, 88-100C, 15-25M_Rate, and 3-7 sigma, up from the previous test. This run took an average of 16.8 +/- 11.9 generations to reach the 0.1 chi-squared benchmark, the next closest run took 20.3 +/- 10.9 generations to reach the same point, while the most optimized run from an earlier test came in as the 10th best run taking 21.9 +/- 14.4 generations. The graph shows the behavior of this by showing large improvements in early generations that tend to quickly plateau around the maximum value before oscillating around the peak it reaches. Oscillations could be caused by either small mutations around our answer and displacement from the simulated error. The Chi-squared values show a much smoother behavior with smaller oscillations at the end of the run, which helps demonstrate the impact of the error on the behavior.

All data and plots discussed in this post are taken from this spreadsheet:

https://docs.google.com/spreadsheets/d/1BbpD81mugWQf10ozGDLI60takAn3tlvrq8ksT10yV5I/edit?usp=sharing

Run Details:

This optimization run was done to do a course search for the optimal set of parameters to run the GA with. This run used a fixed 100-individual population with a simulated error of 0.25 over 50 generations, to best replicate the current PUEO-loop environment. The selection methods have also been held fixed in this run. This run was tested using both the ARA and PUEO design to see if there were similarities in the best run type. This run encompassed 210 different run combinations that searched the following parameter space:

• 0-16 Reproduction, step sizes of 4
• 72-96 Crossover, step sizes of 4
• 4-16 Mutation, step sizes of 4
• 5-15 Sigma, step sizes of 5

Results:

Each run combination was run 10 times and we tracked the average number of generations it took for the distance metric to reach 0.05, which corresponds to a 0.95 true fitness score. The best runs for each design were as followed:

• ARA
  • Parameters
    • 12 Reproduction
    • 72 Crossover
    • 4 Mutation
    • 5 Sigma
  • Average gens to benchmark
    • 28.7 generations
  • Standard Deviation
    • 17.2 generations
• PUEO
  • Parameters
    • 0 Reproduction
    • 96 Crossover
    • 4 Mutation
    • 15 Sigma
  • Average gens to benchmark
    • 23.0 generations
  • Standard Deviation
    • 13.0 generations

We can see that these run types do not appear to have any strong correlations. In fact, looking at our complete data set, there does not appear to be a strong trend around any of the best runs. This issue became more prominent when we ran these best two run types for 100 tests rather than 10. When we did this, they returned averages of 44.4 generations for PUEO and 46.9 generations for ARA. For comparison, the average number of generations across all run types was 43.1 generations for PUEO and 42.8 generations for ARA. This seems to suggest that all of these run types are inherently inconsistent and regress to the mean given enough tests. Therefore, I do not believe we can draw any conclusions from this run.

Moving Forward:

Our ongoing hypothesis is that the size of the error could be causing inconsistency in how quickly our population can grow over time. Therefore, our next step is to repeat this experiment, but with zero error included, and see if we achieve results that show consistent behavior. If we can find more consistency in runs with no error, then we can more deeply explore the effects that error can have on GA growth. If there continues to be a lack of consistency, then we can look to other factors, such as population size, and try to find the root of the inconsistency.

Here are some scholarships available to physics majors I've been lucky enough to receive throughout undergrad that allowed me to work on this project unpaid without needing supplemental income from a separate job or loans. 

1. OSU Arts and Sciences Merit Scholarship Pooled Application 

• 500 word personal statement
• 1 letter of recommendation from a faculty member (Amy wrote mine)
• Varying amounts awarded (I got the $3,300 David and Velva Zarley Scholarship)
• Can be used for any education related expensed (tution, rent, food, ...)

2. James L. Smith Scholarship for Physics Majors

• 3 short essays (~200 words)
• Probably varying amounts (I got $4,000)
• Can be used for any education related expensed (tution, rent, food, ...)

3. Physics Class Awards 

• No application, chosen by department (just get good grades, honors might help. I didn't really talk to any of my professors or go to office hours, so milage might vary there)
• Freshman: $50, Sophomore: $250, Junior: $500, Senior: $?
• Can be used for any education related expensed (tution, rent, food, ...)

4. OSU Scholarship Universe

• A couple ~500 word essay
• Very few awarded, I've applied twice: $1,000 the first time and completely ghosted the second
• Can be used for any education related expensed (tution, rent, food, ...)

5. Licking County Foundation (If anyone is from licking county, I HIGHLY RECOMMEND APPLYNG TO THIS. I think there are similar ones for other counties) 

• ~500 word essay
• Varying awards (I've applied 3 times and gotten $2,375, $5,000, and $7,500) 
• Can be used for any education related expensed (tution, rent, food, ...)

Here is a link to a google drive with all my winning scholarship essays:

https://drive.google.com/drive/folders/1qJKpBf4by9wlReU5l_XjxjVsDIfXR4Jc

Attached is the license agreement that each person should sign to be able to use XF on OSC.  You can sign it, send it to Amy, and she will return it to OSC with her signature on it.

For each new student joining the group, we ask them to onboard by completing each of the following items in the order given:

(1) Go to osc.edu and request an account

• https://www.osc.edu/supercomputing/portals/client_portal/self_signup_for_accounts
• Fill out the info and for the Project Code put PAS1960 (and PAS0654 if you're able to do both).

(2) Complete the following tutorials and reading. Note that the order of these does not matter, however, I would suggest "how to use Git" comes last, as it is the lease important for you to start. Finally, I advise you work on a tutorial concurrently with reading the pages assigned below from my Dissertation. 

• OSC tutorials
  • Because this project uses a lot of computation power, we run on the Ohio Super Computing Center's clusters. Please utilize these tutorials to learn more. 
• Read Julie's Dissertation pages 1-9, 12-20 (starting at section 1.5.2), 28-37 (starting at section 2.3)
  • Feel free to read chapter 3, as it has the history of this project and goes into details of our most up-to-date endeavors. This is optional and not mandatory. 
• Complete the introduction to Bash project
• Complete the introduction to Python Project
• How to use Git (optional)
  • Git is a resource used for version control when writing software. See more for an introduction here: https://www.simplilearn.com/tutorials/git-tutorial/what-is-git
  • BEFORE reviewing this power point, I highly advise you take the code academy class https://www.codecademy.com/learn/learn-git. We use Git and GitHub to keep records and versions of the changes we make, so reverting back to other versions is doable. 
• Make sure to get an XFdtd license on OSC
  • See instructions here 

(3) Finally, review the full manual and complete the bullets below. The full manual is Appendix A of Julie's Dissertation.

• Appendix A of my dissertation is our manual and acts as the holy grail of our research. Anything you need can be found here (from where our info exists on the web, to how to run the code). 
• Once you have read this through entirely, we advise you log on to OSC and explore the code for PAEA in PAS1960. Once you have done so, complete this homework by making a copy of this google doc and filling it out on your own. 

Once these are completed, the students are ready to start helping. This is all of the coding information and basic background information on evolutionary algorithms students must know to be proficient in assisting. 

Other useful materials we have may include:

1. Our old GENETIS manual 
   1. This is somewhat out of date; however, it does have information on the general working on the loop. 
2. Julie's candidacy paper 
3. Complete the writing a genetic algorithm in C++ project
   1. This is still a work in progress! I'll complete it shortly!
4. Read chapter 3 of this book (more if you have time) by Eiben and Smith, "Introduction to Evolutionary Computation" for a quick introduction. The book should be available electronically from a university library.  Let us know if you can't find it there.
5. Get to know XFdtd
   1. XFdtd manual is also attached; however, the link above (in 5) is more useful for coding a .xmacro.
       

Important notes:

MAC USERS: Using a GUI when connecting to OSC via SSH from your terminal.

You will need to set up X11 forwarding. Otherwise, when you go to plot via SSH, they will not display (they don't forward from OSC to your machine). If you are unable to run GUI's on OSC through your computer's terminal, follow these instructions:

1. In your own system's terminal, type: sudo vi /etc/ssh/sshd_config
2. Search for the following line: X11Forwarding no
3. Change it to: X11 Forwarding yes. Save the file and exit.
4. In your own system's terminal, type: sudo systemctl restart sshd
5. For a Mac, download and install XQuartz.
6. Connect to OSC using ssh -XY username@hostname.
7. You can test that this works by typing 'xeye' or 'xclock' in the terminal when logged into OSC. 

Now you should be able to use OSC's GUIs.

PC USERS: 

As a PC user, there are a few things you need to do:

(1) Enable the Windows Subsystem for Linux on your device (https://developerinsider.co/stepwise-guide-to-enable-windows-10-subsystem-for-linux/#:~:text=To%20enable%20the%20Windows%20Subsystem,list%20here%20and%20click%20OK.)

(2) Get Bash on Ubuntu installed on your machine. You are going to need to have Bash on Ubuntu installed prior to starting these trainings; otherwise Bash commands will not be recognized.

•  https://ubuntu.com/tutorials/install-ubuntu-on-wsl2-on-windows-10#1-overview

(3) Get X11 forwarding working. Otherwise, when you go to plot via SSH, they will not display (they don't forward from OSC to your machine).

• Follow these steps here.  
  • http://courses.cms.caltech.edu/cs11/misc/xwindows.html
• Change your config file. To do so, in terminal (with Bash on Ubuntu open) 
  • type: sudo vi /etc/ssh/sshd_config
  • Then change '#X11Forwarding no' to '#X11Forwarding yes' and save it.
    • https://www.businessnewsdaily.com/11035-how-to-use-x11-forwarding.html
• Make sure Putty and Xming are running every time before you ssh in. Use Xlaunch application to run Xming and press yes to everyrthing.
  • Xming should now be running as a background task.
  • Open PuTTY and open the pre-saved like you saved in PuTTY. (This will open a Bash terminal that asks for your OSC login password.)

Here are the following things we need to work on:
 

1. Paper (Julie and Suren)
2. Get loop working-- specifically XF (Cade)
3. Test GA (Suren)
4. Gain pattern plots -- Make them 3D (Evelyn) & put on GitHub
5. Insert LR plot & Gain plot to loop (?)
   1. Edit bash script
   2. Update Github
6. AraSim plots (Julie and Max) -- see #2 and #3 here: http://radiorm.physics.ohio-state.edu/elog/GENETIS/7 for details. 
7. Update manual with things from Suren's last post (?)

We began a new run using the same parameters as the previous one (see the last ELOG entry). The previous one ran for 8 generations. The difference between this run and the last one is that we have added in the polarity fix for AraSim Brian and Jorge gave us (copying the correct Report.cc and Report.h files into our AraSim version). Here are the run parameters:

Run Paramaters:
50 individuals
80/20 Roulette/Tournament
6% Reproduction, 72% Crossover, 22% Mutation
Run for 10 generations, then implement Jorge and Brian's fix for polarization in AraSim for the next run

This is a multigenerational tracing of our second-best individual's parents and children:

The second-best individual in this evolution was Bicone 22 from generation 40. This individual is, in fact, a fascinating case as we shall see. But to start the story of this individual we will go back to generation 38 in order to demonstrate some of the peculiarities. 

In generation 38 there were two bicones of no renown,  Bicone 11 and Bicone 17. Bicone 11 was a fairly average individual that was ranked 23rd with a fitness score of 4.72016. Its DNA was {*6.16084, ***79.663, 0.0015434, -0.107765} for side one , and {0.308809, 39.6742, -0.0084247, 0.40629} for side two. One day, by chance it met Bicone 17, another average bicone ranked 24th with a score of 4.71877 and DNA {**2.32499, 79.663, -0.00224948, 0.192602} for side one, and {0.308809, 39.6742, -0.0084247, 0.40629} for side two. These two individuals eventually became the parents of two antennas: Bicone 16 and Bicone 17 of generation 39.

Bicone 16 eventually grew to have been ranked 3rd in fitness score of 4.95323. It’s DNA ended up being a complete balance of its parents sharing side one with Bicone 38.11{2.32499, 79.663, -0.00224948, 0.192602} and side two with bicone 38.17 {0.308809, 39.6742, -0.0084247, 0.40629}. Bicone 16 was an individual with high aspirations and hoped to be reproduced. But alas, it was not meant to be. But bicone 16 came upon some amazing luck, it was selected with itself for crossover. This meant that bicone 16 was able to provide two identical surrogates to survive into the next generation. This is where this Bicone fulfilled its full potential. 

The twin bicones were named Bicone 22 and Bicone 23 in generation 40. Being clones, they shared all their DNA with Bicone 39.16. However, due to some circumstances, they had slightly different fitness scores. Bicone 23 managed a very respectable 5.0117 fitness score and was ranked 2nd in the generation. But not to be outdone Bicone 22 managed to score a 5.17014 and ended up being the second-best performing individual of all time. Being so successful, the two bicones ended up producing 8 children between the two groups.

Bicone 23 was the first to crossover and had 2 children with its partner. These were Bicones 4 and 5. Bicone 4 was ranked 39th with a fitness score of 4.60251 and still shared the DNA of its second side with its grandparent Bicone 38.17 as well as most of its first side with Bicone 38.11 {2.32499, 79.663, -0.00213879, 0.192602} {0.308809, 39.9608, -0.0084247, 0.40629}. Bicone 5 on the other hand, was ranked 14th with a fitness score of 4.80535 and it still shared a lot of DNA with its grandparents {6.16084,53.0851,-0.00224948,0.0534469} {0.966617,39.6742,-0.0084247,0.40629}.

Bicone 22 had 6 children of its own with various other Bicones. These were; Bicone 8, ranked 11th with a fitness Score of 4.81784 and DNA {2.32499, 75.9855, -0.00224948, 0.192602} {0.966617, 39.6742, -0.00320023, 0.213833}; Bicone 9, ranked 7th with a fitness score of 4.88966 and DNA {6.10508, 79.663, -0.000594616, 0.0351901} {0.308809, 42.4246, -0.0084247, 0.40629}; Bicone 12, ranked 12th with a fitness score of 4.81705 and DNA {6.42695, 75.9855, 0.0015434, -0.107765} {0.308809, 39.6742, -0.00320023, 0.213833}; Bicone 13, ranked 2nd with a fitness score of 5.0344 and DNA {2.32499, 79.663, -0.00224948, 0.192602} {0.966617, 42.4246, -0.0084247, 0.40629}; Bicone 34 ranked 3rd with a fitness score of 4.99864 and DNA {0.66148, 73.5522, -0.000594616, 0.00582814} {0.966617, 39.6742, -0.0084247, 0.40629}; and finally Bicone 35, ranked 22nd with fitness score 4.74955 and DNA {2.32499, 79.663, -0.00224948, 0.192602} {0.308809, 42.4246, -0.0084247, 0.40629}

Bellow, I have attached the rainbow plot with the parameters occupied by individual 4 in gen 41 which was again ranked 39th in that generation. From this, we can see that while in its own generation it was a poor performer, overall it was upper middle of the pack. However, because of the density of other better performing antennas in this region, it is hard to distinguish which genes in this antenna are contributing the most to the drop in fitness score compared to its siblings and parents. 


 

*Gene originating from Bicone 38.11

**Gene originating from Bicone 38.17

***Gene originating from Bicone 38.11 and 38.17 that is shared between the two.

The second-best individual in this evolution was Bicone 22 from generation 40. This individual is, in fact, a fascinating case as we shall see. But to start the story of this individual we will go back to generation 38 in order to demonstrate some of the peculiarities. 

In generation 38 there were two bicones of no renown,  Bicone 11 and Bicone 17. Bicone 11 was a fairly average individual that was ranked 23rd with a fitness score of 4.72016. Its DNA was {*6.16084, ***79.663, 0.0015434, -0.107765} for side one , and {0.308809, 39.6742, -0.0084247, 0.40629} for side two. One day, by chance it met Bicone 17, another average bicone ranked 24th with a score of 4.71877 and DNA {**2.32499, 79.663, -0.00224948, 0.192602} for side one, and {0.308809, 39.6742, -0.0084247, 0.40629} for side two. These two individuals eventually became the parents of two antennas: Bicone 16 and Bicone 17 of generation 39.

Bicone 16 eventually grew to have been ranked 3rd in fitness score of 4.95323. It’s DNA ended up being a complete balance of its parents sharing side one with Bicone 38.11{2.32499, 79.663, -0.00224948, 0.192602} and side two with bicone 38.17 {0.308809, 39.6742, -0.0084247, 0.40629}. Bicone 16 was an individual with high aspirations and hoped to be reproduced. But alas, it was not meant to be. But bicone 16 came upon some amazing luck, it was selected with itself for crossover. This meant that bicone 16 was able to provide two identical surrogates to survive into the next generation. This is where this Bicone fulfilled its full potential. 

The twin bicones were named Bicone 22 and Bicone 23 in generation 40. Being clones, they shared all their DNA with Bicone 39.16. However, due to some circumstances, they had slightly different fitness scores. Bicone 23 managed a very respectable 5.0117 fitness score and was ranked 2nd in the generation. But not to be outdone Bicone 22 managed to score a 5.17014 and ended up being the second-best performing individual of all time. Being so successful, the two bicones ended up producing 8 children between the two groups.

Bicone 23 was the first to crossover and had 2 children with its partner Bicone 25 (4.64409). These were Bicones 4 and 5. Bicone 4 was ranked 39th with a fitness score of 4.60251 and still shared the DNA of its second side with its grandparent Bicone 38.17 as well as most of its first side with Bicone 38.11 {2.32499, 79.663, -0.00213879, 0.192602} {0.308809, 39.9608, -0.0084247, 0.40629}. Bicone 5 on the other hand, was ranked 14th with a fitness score of 4.80535 and it still shared a lot of DNA with its grandparents {6.16084,53.0851,-0.00224948,0.0534469} {0.966617,39.6742,-0.0084247,0.40629}.

Bicone 22 had 6 children of its own with various other Bicones. These were; With Bicone 8.40 (4.82423) Bicone 8, ranked 11th with a fitness Score of 4.81784 and DNA {2.32499, 75.9855, -0.00224948, 0.192602} {0.966617, 39.6742, -0.00320023, 0.213833}; Bicone 9, ranked 7th with a fitness score of 4.88966 and DNA {6.10508, 79.663, -0.000594616, 0.0351901} {0.308809, 42.4246, -0.0084247, 0.40629}; With Biocone 16.40 (5.01137) Bicone 12, ranked 12th with a fitness score of 4.81705 and DNA {6.42695, 75.9855, 0.0015434, -0.107765} {0.308809, 39.6742, -0.00320023, 0.213833}; Bicone 13, ranked 2nd with a fitness score of 5.0344 and DNA {2.32499, 79.663, -0.00224948, 0.192602} {0.966617, 42.4246, -0.0084247, 0.40629}; With Bicone 4.40 (4.66578) Bicone 34 ranked 3rd with a fitness score of 4.99864 and DNA {0.66148, 73.5522, -0.000594616, 0.00582814} {0.966617, 39.6742, -0.0084247, 0.40629}; and finally Bicone 35, ranked 22nd with fitness score 4.74955 and DNA {2.32499, 79.663, -0.00224948, 0.192602} {0.308809, 42.4246, -0.0084247, 0.40629}

Upon inspection of the children in generation 41, you will see that none of the children surpassed their parent individual with the highest fitness score. They do however tend to have fitness scores around or above their second parent's score. Therefore, it is likely that the children's lower fitness score is due to a mismatch of genes. If we look at the rainbow plots provided, we can see that the genes of the worst-performing child are not dissimilar to those of individuals that scored higher than 5. This would imply that the fitness score found could be due to variations in Arasim similar to those seen from individuals 16.39 to 22/23.40 despite them being identical. Though it also could be possible that the graphs do not have a fine enough display to be able to clearly tell some of the genes apart. 

*Gene originating from Bicone 38.11

**Gene originating from Bicone 38.17

***Gene originating from Bicone 38.11 and 38.17 that is shared between the two.

Amy: We want to nail things down in places where we still have hypotheses for what could have happened.  For example, "This would imply that the fitness score found could be due to variations in Arasim similar to those seen from individuals 16.39 to 22/23.40 despite them being identical. Though it also could be possible that the graphs do not have a fine enough display to be able to clearly tell some of the genes apart."  These two hypotheses can easily be distinguished by running with higher statistics and looking at the numbers rather than relying on the resolution of the display.

Amy: Also, at each stage, I was looking for the random numbers that were generated that led to the selection of the fractions of the population that were used for each type of selection and operator.  

The story of individual 8 from generation 18. (draft)

Once, there was a curved bicone named individual 8 who was from generation 18. In many ways, it was similar to many other bicones but in fact, this individual was the best bicone that ever lived with a fitness score of 5.22097. It achieved this by having the following parameters; an inner radius of 6.31483, and 3.97024; lengths of  77.6017 and 40.9244; quadratic coefficients of, 0.00260615 and, -0.0103313; and finally, a linear coefficient of -0.197494 and

0.36119. Unfortunately, individual 8’s lineage ended with it as it never met that special someone (crossover) and didn't survive into the next generation (reproduction) despite having a 77.7% of doing at least one of these. 

This bicone had a sibling with a fitness Score of 4.65861 and had the following parameters: 6.05614,79.663,-0.000353038,0.0331969 (side 1)

1.92878,37.9849,-0.00213265,0.156724  (side 2).

Unfortunately, this bicone also suffered the same fate as its sibling and failed to leave a lasting mark on this hypothetical world. 

These two individuals had parents from generation 17 whose name’s were individuals 39 and individual 8. After having individuals 8 and 9 in generation 18, individuals 39 and 8 from generation had a nasty divorce (possibly leading to 8 and 9’s aversion to marriage) and remarried. In these marriages, they both produced two more children that would be our previous antenna’s step-siblings. 

Antenna 8 from gen 17 married antenna 46 to produce antenna 20 with a fitness score of 4.57122,  and antenna 21 with a fitness score of 4.71056. Individual 20 shared the same genes for the second set of linear and quadratic coefficients as its more successful step-sibling individual 8. 21 had no similarities to individual 8 but had 4 children in generation 19. 

Antenna 39 from gen 17 married antenna 5 to produce antenna 28 with a fitness score of 4.71808 and antenna 29 with a fitness score of 4.90119. Antenna 28 shared had an identical side 1 to individual 8 and also shared the same inner radius and length on the second side, and had 2 children in generation 19. Individual 29 had no similarities to individual 8 but really got around and had 8 children with various partners in generation 19. 

I've started a run of the PUEO loop to get some data for evolving the antenna width and height. We are running with 24 individuals, each with 400k neutrinos at an energy exponent of 19. Here is the output directory: /fs/ess/PAS1960/HornEvolutionOSC/GENETIS_PUEO/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop/Run_Outputs/2023_05_01_Test_5

It's very important to note that the cross-pol data being used still comes from the data already stored in pueosim. This is because our extremely low cross polarizations were causing us to have no effective volumes. There's likely some issue in what values from XF we are choosing as our crosspols.

Slides contatining my notes on matching circuits.

https://docs.google.com/presentation/d/1x25nhiqaW7LvPZ1pNZ5O4ZzsWZbtgqxBQ5haB9uWgQY/edit?usp=sharing

Attached is a spreadsheet with the information on the parts we need for the N=14 matching circuit board.

https://docs.google.com/spreadsheets/d/1x8dX3tNE-WSHjH_slj_EH4XsHcAnCVC05XiRBFPtIUc/edit#gid=0

We presented at a workshop put on by the MODE collaboration. MODE is a collaboration dedicated to applying Automatic Differentiation to detector design. Here's the website: https://mode-collaboration.github.io/#:~:text=MODE%20(for%20Machine%2Dlearning%20Optimized,in%20space%2C%20and%20in%20nuclear

• Run Type
  • Main Arasim Loop
• Run Date
  • 02/04/2022
• Run Name
  • 2022_02_04_Rank
• Why are we doing this run?
  • To test rank selection in main loop
• What is different about this run from the last?
  • Rank Slection is being used.
  • Parents.csv introduced.
  • Elite is being turned off.
• Symmetric, asymmetric, linear, nonlinear (what order):
  • Non-linnear asymetric
• Number of individuals (NPOP):
  • 50 individuals
• Number of neutrinos thrown in AraSim (NNT):
  • 300,000
• Operatiors used (% of each):
  • 06% Reproduction
  • 72% Crossver
  • 22% Immigration
  • 1% M_rate (unused)
  • 5% sigma (unused)
• Selection methods used (% of each):
  • 0% Elite
  • 0% Reproduction
  • 10% Tournament
  • 90% Rank
• Are we using the database?
  • No.

Directory: /fs/ess/PAS1960/BiconeEvolutionOSC/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop/Run_Outputs/2022_02_04_Rank

Here's the video I made a few months ago demonstrating how to run the loop. There might be some changes that have been made since then, but they'll generally be minor so it should still be representative of what running the loop looks like. It might need to be shared with you through the Microsoft drive service OSU gives us in order to view it, so either message me on Slack or email me at machtay.1@osu.edu and I'll share the link through there.

https://drive.google.com/file/d/1yL_eH_w2hXNt7J7YOwS5xl3ZHwGyl5JR/view

As mentioned in the manual, using ssh to login to Nutau causes delays; the XFdtd GUI cannot be surpress and must be forwarded via X11 forwarding, and this is extremely slow. We are currently looking into using a newly suggested option, XRDP. Information will be added as this process continues. 

Previously, we have tried to straighten the sides of the best-evolved curved antenna in elog 229. However, there were potential issues with how closely this line resembled the curve of the antenna.

So, I attempted to create another straightened sides antenna using a linear regression model to find the best fitting line for the curve to create an antenna.

I made a Python notebook to separate the equations for each curve into 1000 discrete points. Then I ran a linear regression model to fit a curve of the points 1 - n, and a second curve of the n - 1000 points, looping from n = 2 to n = 999.

These results are from the combined output with the least squared error compared to the original.

Pictured is a plot showing the two sides of the curved bicone in red and blue, with the best fitting lines for each in black as well as a model in XF.

Results:

The antenna has a fitness score of 3.80627 with an error of 0.0759725.

This is much lower than the 5.71 of the curved antenna and 5.11 of the other attempt at straightening.

For the next attempt, we could consider constraining the endpoints to be the same as the original antenna to conserve the radius, and/or adding an extra line to fit the curve.

I've also attached a picture of what my notebook found for fitting 3 lines to the curve (not modeled or tested).

Professor Chen recommended using 3 sides and constraining the outer radii of the cones to match the original curved design.

Path to linear regression notebook: /users/PAS1960/dylanwells1629/developing/notebook.ipynb

Path to XF project: /users/PAS1960/dylanwells1629/straightened.xf

OSC managed to figure out our issue with running XF from an interactive job on Slurm. Previously, we were losing our connection to x11 forwarding. The solution is to use sinteractive instead of srun to obtain an interactive job. Here's the syntax:

sinteractive -A PAS0654 -t <time> -N 1 -n 8 -p serial

The -p serial flag denotes the type of partition to request. It's important to specify this, as otherwise it will default to the debug node, which has a limit of 1 hour.

We're going to begin a new run. The title is Machtay_2021_1_15_NPOP50_Asym . It is using the latest version of the GA Ryan has been working on (Latest_GA_Asym.cpp). We're using 6% reproduciton, 72% crossover, and 22% mutation. We are using 80% roulette selection. This correlates to the three numbers passed into the GA being 3 36 8 (since we're using 50 individuals).

Running IceMC:

Go into the directory ../anitaBuildTool/build/components/icemc/

And run the command

• ./icemc -i {inputFile} -o {outputDirectory} -r {runNumber} -n {numberOfNeutrinos} -t {triggerThreshold} -e {energyExponent}

*May need to chmod -R 775 ../anitaBuildTool/comonents/icemc/ if you get a permissions error

inputFile:

Must be the full path to the file

Config files are found in ../anitaBuildTool/components/icemc

Ex: /users/PAS1960/dylanwells1629/anitaBuildTool/components/icemc/inputs.anita4.conf

Config files are found in ../anitaBuildTool/components/icemc

outputDirectory:

Will be made in ../anitaBuildTool/build/components/icemc/ by default, specify full path otherwise.

runNumber: 

The run number.

numberOfNeutrinos:

The number of neutrinos generated in the simulation. 

Can be found in inputs.conf

Default is 2,000,000.

#How many neutrinos to generate

triggerThreshold:

Threshold for each band for the trigger. 

Default is 2.3

#thresholds for each band- this is only for the frequency domain voltage trigger.  If using a different trigger scheme then keep these at the default values of 2.3 because the max among them is used for the chance in hell cuts

energyExponent:

The exponent of the energy for the neutrinos

Can be found in input.conf 

Default is 1020

# Select energy (just enter the exponent) or (30) for baseline ES&S (1) for E^-1 (2) for E^-2 (3) for E^-3 (4) for E^-4 (5) for ES&S flux with cosmological constant (6) for neutrino GZK flux from Iron nuclei (16-22)not using spectrum but just for a single energy (101-114)use all kinds of theoretical flux models

Installing IceMC:

I: Getting anitaBuildTool

Clone the anitaBuildTool repository (https://github.com/anitaNeutrino/anitaBuildTool) to your user.

• git clone https://github.com/anitaNeutrino/anitaBuildTool

II: Get the Anita.sh file onto your user.

 Either copy the file from /users/PAS1960/dylanwells1629/Anita.sh

• cp /users/PAS1960/dylanwells1629/Anita.sh ~

or create a new file Anita.sh with the following code

# .bashrc

# Source global definitions

if [ -f /etc/bashrc ]; then

. /etc/bashrc

fi

#this modules were originally loading in both the env.sh, and bashrc_anita.sh files. This was redundant so it was added here, and removed from the others.                                          

 module load gnu/7.3.0

module load gnu

module load mvapich2

module load fftw3

#module load python/3.6-conda5.2

module load cmake

PATH=$PATH:$HOME/.local/bin:$home/bin

export PATH

export CC=`which gcc`

export CXX=`which g++`

export FFTWDIR=/fs/project/PAS0654/shared_software/fftw3/gnu/6.3/mvapich2/2.2/3.3.5

export ANITA_SOURCE_DIR=~/anitaBuildTool/

export ANITA_UTIL_INSTALL_DIR=~/anitaBuildTool/

export ICEMC_SRC_DIR=~/anitaBuildTool/components/icemc/

export ICEMC_BUILD_DIR=~/anitaBuildTool/build/components/icemc/

export DYLD_LIBRARY_PATH=${ICEMC_SRC_DIR}:${ICEMC_BUILD_DIR}:${DYLD_LIBRARY_PATH}

export ROOTSYS=/fs/project/PAS0654/shared_software/anita/owens_pitzer/build/root

# User specific aliases and functions

#This env.sh is for running the BiconeEvolution GENETIS software. This should only be un-commented if you are running GENETIS software. When you do this, comment out env.sh.                      

#source ~/new_root_setup.sh

source /fs/project/PAS0654/shared_software/anita/owens_pitzer/build/root/bin/thisroot.sh

#source /cvmfs/ara.opensciencegrid.org/trunk/centos7/setup.sh

#module load python/3.6-conda5.2

#BiconeGENETIS directory shortcut SHARED                                                                                                                                                           

alias GE='cd ../../../fs/project/PAS0654/BiconeEvolutionOSC/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop/'

#emacs Alias                                                                                                                                                                                       

alias emacs='emacs -nw'

#root alias                                                                                                                                                                                        

alias root='root -l'

#Alias                                                                                                                                                                                             

alias l="ls"

alias python='/cvmfs/ara.opensciencegrid.org/trunk/centos7/misc_build/bin/python3.9'

Then source the file

• source Anita.sh

Note: You will need access to PAS0654 for this step or you will get a permissions error.

III: Running the build tool.

Go into the anitaBuildTool directory

• cd anitaBuildTool

And run the building script

• ./buildAnita.sh

Note: There will be an error if you source files for running Ara in your .bashrc

Comment these out and restart your terminal before running the build. (remember to source Anita.sh before running the build tool. You could also source Anita.sh in your .bashrc)

Error if you source files for running Ara:

CMake Error at components/libRootFftwWrapper/cmake_install.cmake:238 (file):

  file INSTALL cannot copy file

  "/users/PAS1960/dylanwells1629/anitaBuildTool/components/libRootFftwWrapper/include/AnalyticSignal.h"

  to

  "/cvmfs/ara.opensciencegrid.org/v2.0.0/centos7/ara_build/include/AnalyticSignal.h":

  Read-only file system.

Call Stack (most recent call first):

Will sent me some info on how to install PUEOsim. I attached a markdown file he sent for how to install PUEOsim. He also sent the path to pre-compiled installation of PUEOsim. Here's the path to the pre-compiled installation: /users/PAS0654/wluszczak/PUEO_shared

Here is his full message, which details that you must also run set_env.sh before running PUEOsim:

a pre-compiled version is located at /users/PAS0654/wluszczak/PUEO_shared . Inside that directory, there's a set_env.sh script that you will need to source before running anything (hopefully this works, though there's a high probability you might run into permissions issues with the dependencies).

To run PUEOsim, use the following (see here for instructions: https://github.com/PUEOCollaboration/pueoSim):

./simulatePueo -i {inputFile} -o {outputDirectory} -r {runNumber} -n {numberOfNeutrinos} -e {energyExponent}

The outputs will be root files, but it should be possible (either modifying our own version directly or with a script) to print out the effective volume to a .txt to read.

This is a duplicate post of a previous post from the end of 2020 where I listed the important runs with some of their details in a table (as below). I am extending this table to include the important runs that have been conducted since this post. This includes the run used for the paper as well as the curved run done earlier this year.

In order to access the data for these runs, you can find them by going to this directory: /fs/ess/PAS1960/BiconeEvolutionOSC/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop/Run_Outputs

Some of these runs can also be accessed in the old project space directory, though they should all be contained in the above directory. Here's the path if interested: 

The runs are contained in directories available in the above path. Use caution when listing files in some of these directories--some contain many files (primarily the .uan files -- more recent runs are better organized), which means it may take a long time to list files/directories.

Today I removed some of the run directories which had very little or no data or weren't worth keeping around. There are still a few that I think can be removed, but I'm keeping them until we can get a consensus that they can definitely be removed. Below I listed the names and descriptions of the runs that I think we should definitely preserve going forward. In general, the more data contained in the run directory, the more important it is to keep around. 

We are trying to do a short run of the asymmetric bicone so that we can see how it tries to evolve the antenna when the minimum length is lowered from 37.5 cm to 10 cm. Currently, there is a problem with the loop in the asymmetric version. Attached is the run detail file.

Here is our current assumption of the antenna angles needed for the icemc inputs. 

Here is a PDF with a summary/log of my work trying to integrate an ice shell surrounding the bicone antennas in the GENETIS loop. I made gain patterns of the ARA bicone with the ice shells and without them, showing the impact of the ice on the antenna's gain. Having an ice shell changes these patterns, which are also dependent on the radius of the shell. I showed that convergence on the gain patterns and expected features start ocurring for shells with radii larger than 10 meters. I also keep track of the computation time which increases with bigger shells.

Note: I also add PowerPoint slides that contain all the graphs that I made. The PDF does not contain some of those since I didn't think they were coherent with my write-up, but I may be making an omission. 

 

In this post, I'm going to give step by step instructions on everything you need to do to run the loop. See Julie's thesis for the most recent iteration of this manual. Also refer to entry 123 for a list of some errors we have encountered historically in the loop and how to resolve them, though some fixes may be outdated.

Running the loop

Getting started

Assuming you have followed the instructions from the onboarding tutorials, you should be ready to access OSC through the interactive website. Instead of using ssh to access OSC, the loop requires that you go to this website: https://ondemand.osc.edu/ . Login with your usual OSC credentials. Once there, you will need to request a Lightweight Desktop by clicking on "Interactive Apps" at the top of the page. See the screenshots attached for how to do this. Old documentation may refer to the Lightweight Desktop as a VDI.

You can request up to 24 hours for a Lightweight Desktop, and it should begin almost immediately. After you request it, the website will take you directly to the waiting page. You can navigate to this page yourself by clicking "My Interactive Sessions" at the top of the ondemand page. When the job begins, you will see a blue buttong that says "Launch Lightweight Desktop". Click it and you should have a new tab open that brings you to a remote desktop environment on OSC that you interact with through your browser. 

New runs

If you're starting a new run for the first time (that is, beginning from generation 0), you will need to edit the script that runs the loop. Here is the path:  /fs/ess/PAS1960/HornEvolutionOSC/GENETIS_PUEO/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop/Loop_Scripts/Asym_XF_Loop.sh . At the top of the script is a list of variables that can be changed. See the first screenshot attached here.
For a new run, you must change the variable <RunName>, which is the first variable in the list. It should be in quotes and the convention is to start the name off with the date of the run, using YYYY/MM/DD format--this will always keep everything in chronological order when listing directories with <ls>. Other variables are explained in the comments next to them in Asym_XF_Loop.sh and you should change them depending on the type of run you are conducting. 

To begin the loop, type in the command ./Loop_Scripts/Asym_XF_Loop.sh from the Evolutionary_Loop directory.

When the loop begins, you will be asked to press any key. After you hit a key, the loop will run Part A, which executes the GA using the <start> mode. It will then automatically move on to Part B, which creates the antennas in XFdtd. XF will ask you if you want to specify a location for the XF file to be saved. Click <No>. The loop will automatically save the XF directory into the run directory when it finishes with the first part of part B. In principle (that is, barring errors that stall the loop or interruptions due to the VDI job ending), you will never need to manually enter anything to the loop for it to run beyond this point. 

However, if it is your first time using XF, you will need to set the max GHz setting upon startup to 2 instead of the default 10. 

Continuing Runs

After starting a new run, the loop will run automatically. When your Lightweight Desktop job ends, you'll need to get another one repeat the steps above. You will not be prompted for any input. 

Stepping Back

At times, you may want to continue the loop from an earlier point or interrupt the loop and step back. Usually you'll only want to do this within a generation, but it's possible that you will encounter an error or bug at the end of a generation and wish to return back to that point to remedy it before continuing forward in the next generation. This is a very tricky and delicate process that requires a lot of attention to detail. In the future, we'd like to make an option for the loop to do this automatically when given input at the beginning, but implementing that will take a long time and will be very sophisticated.

The state of the loop is recorded in the savestate file located here: /fs/ess/PAS1960/HornEvolutionOSC/GENETIS_PUEO/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop/saveStates . The file will be named <Run_Name>.savestate.txt . In it, you will see three numbers. The first one represents the current generation of the loop. The second one represents the state of the loop, that is, which part in the generation it is currently on. The third one represents the individual it is processing. In practice, the third number is rarely relevant, and is almost always 1, though it can be used for Part B, where the loop is processing individuals sequentially, so if it was interrupted on individual eight (for example) you can change the number to be 8 so that it will pick up where it left off without needing to redo the first seven individuals.

If you want to step back to an earlier state in a generation, you'll need to interrupt the loop and revert the state by changing the second number in the savestate file. This is NOT all you will have to do, however. Because of the way files are edited and moved around during states, you will need to manually revert files to be in the format or location needed by the earlier state. 

For example, you may need to revert from state 2 (when XF is creating the antenna geometries) to state 1 (when the GA generates new individuals). In this case, you'll need to edit the savestate file to change the second number from a 2 to a 1. Then, you'll need to change the generationDNA.csv file in Generation_Data to use the values from the previous generation, since the GA will have overwritten that file and uses it to determine what individuals are to be selected from. To do this, you'll need to locate the <gen>_generationDNA.csv file from the previous generation in the <RunName> directory in Run_Outputs. This is should be found here:  /fs/ess/PAS1960/HornEvolutionOSC/GENETIS_PUEO/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop/Run_Outputs/<RunName>/Generation_Data . 

The above is a relatively simple example of how to step back in the loop. Stepping back from a late state to an early state will be more complicated, though if you're willing to sacrifice run time you can pretty easily follow the above to just restart a generation. Stepping back to a previous generation will be explained in a future update to this post.

One of our foundational questions tied to the optimization of the GA has been "How many individuals should we simulate". Up to now, our minds were made up for us by the speed of arasim being great enough that the time cost of simulating individuals was great enough that the improvements made from having more were not enough to justify the slowdown. However, with the upgrade to the faster, more recent version of arasim, I decided to re-examine this. This was also spurred on by the fact that the last time I ran this test we were testing GA performance by final generation metrics rather than by how many generations it took to reach a benchmark. So in one of my optimization tests, I tracked this data. 

To start, using the same run proportions, using a .5 chi-squared benchmark, the average time across all 89 run types used in this run was 25.4 generations for 50 individuals as compared to 8.3 generations running the same test for 100. Furthermore, the minimum number of generations for 50 individuals was 4.8 while using 100 individuals yielded 2.4. So on average running 100 individuals was about 3 times fast at reaching this benchmark than with 50. And when comparing the best result regardless of run type, 100 individuals was still 2 times quicker than the min for 50 individuals. Finally, the run that yielded an average of 2.4 generations for 100 individuals took an average of 29.2 generations with 50 individuals or roughly 12 times the generations. 

For the test we will discuss, we ran 89 different run types that all used 60% rank, 20% roulette, and 20% tournament selection respectively. These test had the following ranges:

6-18% of individuals through reproduction (steps of 3%)

64-88% of individuals through crossover (steps of 12%)

0-10% mutation rate (steps of 5%)

1-5% sigma on mutation (steps of 1%)

These tests also used our fitness scores with simulated error of .1 to imitate arasim's behavior and as such we used the chi-squared value to evaluate these scores as there is no error on those values. 

Comparing this same test with a tighter chi-squared benchmark of .25, we see similar results. On average 50 individuals took 37.1 generations to reach this point while 100 individuals took 16.0 generations. Similarly, the minimums amount of gens for 50 individuals was 15.4 while 100 individuals was 5. Finally, the corresponding run for the 5 generation min with 100 individuals took 41.8 generations with 50 individuals. These correspond to speed up's of 2.3, 3.08, and 8.36 respectively. 

This data implies that on average, independent of run type, we should expect to have to use 2-3 times fewer generations while running 100 individuals than we would running 50 individuals but we could see up to 8-12 times fewer generations to reach benchmarks. Another data set using a different set of selection methods was also tested for this and again yielded similar results, though overall the runs from the first batch were better across both 50 and 100 individuals and so those results are likely to be more indicative of the parameters we use in a true run. 

The data being examined in these results can be found here: https://docs.google.com/spreadsheets/d/1GlfnjQSO6VI8MuUGYTUcLkjwDZU98nyFFysgTTfVFOE/edit?usp=sharing  

Work in progress. Update on H-pol XF design. The first plot is the gain pattern with ferrite rod sets (blue) vs. with nothing but the copper plates. 
The second plot is the gain pattern with everything but the ferrite rod sets vs. with nothing but the copper plates.
The ferrite rod sets function to narrow the gain pattern.

How To Update PueoSim For GENETIS:

First, whoever updates pueoSim needs access to pueoBuilder, pueoSim, and niceMC on GitHub (ask Will for permissions).

Once you do, go into the peuoSim directory at /fs/ess/PAS1960/buildingPueoSim/

and source set_env.sh

`source set_env.sh`

Then, we want to make copies of the files we are currently modifying for the GENETIS loop.

For PueoSim v1.1.0 these are:

1. nicemc/src/EventGenerator.cc  # Modified to create custom output root files to calculate errors

2. pueosim/test/simulatePueo.cc # Modified to stop the simulation of the LF antenna which is not needed for GENETIS

3. pueosim/src/Seavey.cc # Modified to read in custom gain files

4. pueosim/src/pueo.cc # Modified to read in custom gain files

5. pueosim/include/Seavey.hh # Modifies to read in custom gain files

Currently, I store copies of these in the `/fs/ess/PAS1960/buildingPueoSim/backups` directory in case somebody accidentally overwrites the files in pueoBuilder. 

Once you’ve made the copies, you can run `./pueoBuilder.sh` from the `/fs/ess/PAS1960/buildingPueoSim/pueoBuiler` directory. This will rebuild pueoSim and niceMC, pulling the latest updates from GitHub. 

You may need to delete the pueoSim and niceMC directories in order for the builder to pull the latest version from GitHub. Or, if it’s being really stubborn, move the whole pueoBuilder directory to a temporary location and run the builder from scratch with 

`git clone git@github.com:PUEOCollaboration/pueoBuilder` and then `./pueoBuilder.sh`

Then, you will need to reference the copies of the changed files to make changes to the new version of pueoSim. Hope this doesn’t cause too much of a headache, and when you’re done, return to the /fs/ess/PAS1960/buildingPueoSim/pueoBuiler directory. 

Then you simply type

`make install`

then

`make`

And now, pueoSim should be ready to run!

EventGeneratior.cc Changes and Rationale:

PueoSim’s default output ROOT files are very large and therefore time-consuming to parse through to get the information we need to calculate effective volume and errors. So, we want to create a custom ROOT file with only the variables we want, greatly increasing the speed of the analysis.

To do this, we want to create a new TFile with corresponding TTree and TBranches that will store the loop.positionWeight, loop.directionWeight, and neutrino.path.weight. Then, we want to fill the Tree when a detector is triggered and write the results to the file at the end of the loop.

Sample code for initializing the TFile:

simulatePueo.cc Changes and Rationale:

By default, pueoSim v1.1.0 runs a simulation for the normal antenna and a low-frequency (LF) antenna. As GENETIS is evolving for the main antenna, we are not interested in using computation time to simulate the LF antenna. So, we comment out the lines that initialize the LF detector in this file.

Seavey.cc, Seavey.hh, and pueo.cc Changes and Rationale:

As of pueoSim v1.1.0, the simulation software only has the ability to read in one antenna. This is a problem, as we want it to be able to read in files for any antenna we make. So, we need to change these scripts to be able to read in whatever gain files we want.

The convention the loop uses is to input gain files in the format of 

vv_0_Toyon{runNum}

hh_0_Toyon{runNum}

…

In the ./pueoBuilder/components/pueosim/data/antennas/simulated directory

So, the main change is getting the runNum variable into Seavey.cc file where the gains are read. Currently, we define a global variable at the stary of pueo.cc and pass it into where Seavey is called. This means we have to add runNum as a parameter in Seavey.cc as well as Seavey.hh. Finally, we change the name of the read-in files to be in the above format AFTER dividing runNum by 1000 (this is because pueoSim uses the run number as a random number generating seed, so we divide runNum by 1000 to be able to read in the same gain patterns for multiple seeds of the same individual).

Note: We used to change pueoSim to output a veff.csv file. We now handle this calculation by analyzing ROOT files, so this change is no longer necessary. 

Attached is a .txt file you can find in the Evolutionary_Loop directory as Loop_Error_List.txt. It's a list of the current errors we sometimes experience in the loop, along with how to fix them if you encounter them while running. If people encounter errors that aren't in the list, let everyone know in the #genstudents chat on slack and update the file with the error message and when it was encountered (what state the loop was in) and the possible cause and solution if you know it. 

We want to automate tests in pull requests in GitHub to run unit tests on individual functions to ensure they return the expected results. This will help prevent bugged code from being pushed to working branches. We will add a list of unit tests to this ELOG as we design them.

Here are a collection of antenna gain patterns with the antenna power source located at various heights at 500 MHz. The heights are indicated in the name of the images and were located at 3cm, 6cm, 12cm, and 24cm from the base. One thing to note is that the power source becomes longer as a function of height (since it's connected to the ridges, which slope back). 

The 3D gain plots attached show the gain becoming more uniform until we get to the last one, where it seems to reinvert and be much stronger in the upward direction than below. I'll update this post with some polar slices for a more complete comparison between them.

EDIT: 07/17/23

I've made a few adjustments and added plots of them.

• First, I separated the ridges. Previously, the ridges touched at their inner corners. This electrically connected them and may be why we saw very high peaks in the first set of impedance plots. You can see the plots named Separated_Impedance_<x>_cm.png.
  • This appears to have lowered the peaks considerably and also shifted them to lower frequencies. The peaks are still high when the source is placed high.
  • We seem to be developing another peak at higher frequencies, although this also existed in the previous tests for the high sources.
• After this, I added a waveguide and positioned the source between the extensions of the ridges into the waveguide. I've also added a picture that shows the waveguide in the geometry. 
  • The sources here are positioned from 1 cm below the ridges to 8 cm below the ridges. The naming convention is the same as above, except it includes "Waveguide" in the name.
  • The peaks are considerably lower here and seem consistent as we place sources further down. They are all below 500 Ohms, compared to as much as 4000 Ohms in the previous tests.

EDIT: 08/10/23

I've been trying more tweaks to the design to get rid of the peaks we see in the plots below. I'm having trouble actually getting the impedance to match close to the 50 Ohm value we expect from the cable. The best case situations seem to come from when the ridges are still touching, which we don't want. Here are the properties I've been changing:

• Ridge positions (gene in GA)
  • We don't want the ridges to touch, but I can't get them to be very far from each other anyway. The minimum wavelength in our bandwidth would be 20 cm, but that's close to the size of the bottom of the antenna for when we make it large, so the separation is still small.
  • When the ridges do touch is actually when I get the best matching,
• Bottom size (gene in GA)
  • I've been changing the size of the bottom of the antenna. This is something that can change substantially between individuals in the GA, so even if I find a good value for this, the impedance it gives wouldn't be the same for all individuals we see in an evolution.
  • This does seem to move the location of the impedance peak (in frequency) but I can't seem to get it fully out of band.
• Waveguide length
  • I've been changing the length of the waveguide between 3 cm and 10 cm. We said that somewhere in the middle should be the most reasonable (the example antenna we saw looked to have a waveguide of ~5 cm long). It seems like the source position is more important.
• Source position
  • I've been moving the position of the power source inside the waveguide. This seems to move around the peak and can lower its height, but I haven't been able to get peaks lower than ~400 Ohms, except in cases where the ridges are in contact.

I've added a picture of the best impedance I managed to get, but this involved an antenna with a small base and ridges touching. My next step is to just try a grid search by making many antennas automatically and simulating them all to look for any that have a nice impedance in our band. 

As of 01/30/2020, the proposal was submitted! The project description and summary are attached below. 

Bicone Loop Status Update: 

The loop currently runs; however, it does seem that if AraSim jobs are left to run while the user is away, it doesn't properly write one or more of the flags, and makes the loop get stuck at AraSim. We must then re-run from that state by manually editing the Save State text file. 

The loop currently has the following additions implemented.

• Functionalized loop (XF_Loop_AraSeed.sh)
  • Cleaner bash script which runs smaller bash scripts, instead of one massive script. 
  • Submits multiple AraSim jobs in parallel for each individual to increase speed. 
    • Ie if 100k neutrinos are thrown per individual ($NNT=100000, $Seed=10) we can, say, break it up into a "seed" of 10 -- which then submits 10 jobs with NNT=10k for each individual. 
  • Still needs this to be fixed: The AraSim run in Gen 0 that gets Veff for the ARA actual bicone input file is not currently being seeded. It is just submitting one job the following way.
    • If NNT=100000 and Seed=10 (ie 10 jobs with 10k neutrinos), it will only run AraActualBicone as 1 run with 10k neutrino number (ie NNT/seed=NNT_New).
• Save State 
  • After every "part" -- ie part a, part b, etc -- of the loop, we save state by writing a series of numbers to a text file. 

    • ${gen}, ${state}, {indiv}

  • If the loop is killed for any reason during the process, the text file is read in, and the loop can continue forward from the last completed item/section. 

  • This saves the state after every individual runs through XF. 

• Antenna sizing

  • The larger the antenna, the longer XF takes to run. A true-to-size ARA antenna takes anywhere from 1.5 to 2 hours per individual. 

  • What we've done: Allowed us to scale the antenna by a factor of "x".

    • If we make the gaussian centered around something, say, 5 times smaller than the ARA antenna size, we then make the frequencies 5 times larger. This should have the same effect. 

    • This also means we've had to scale these sizes back up when plotting. 

    • Still needs this to be fixed: all of this has been hardcoded in! We need to make these variables in the MANY places we need to adjust for this scaling. See the to-do list below for details. 

Moving forward, I will be doing the following:

1. Getting our less coding-savvy teammates familiar with bash, python, and c++. This will be crucial for us when we start adding more parameters to our GA. 
2. Mastering XF (along with Alex, and Alex). 
3. Getting the team to clean up some messy coding done during the proposal hustle. 
4. Fixing some errors. 
5. Teach the ENTIRE team git. 

TO-DO List

The following is our coding to-do list:

1. Get our Ara sim error fixed (Alex, Julie) 

   1. It seems that if AraSim jobs are left to run while the user is away, it doesn't properly write one or more of the flags, and makes the loop get stuck at AraSim. We must then re-run from that state by manually editing the Save State text file.

   2. We are going to try to get XF to run at the command line so that we can submit it as a job. This would alleviate us using an interactive job. We *hope* this would fix the error since the user wouldn't be "gone" while utilizing a GPU in an interactive job. 

      1. Alex emailed XF to ask a few questions! Stay tuned!

2. Make Hardcoded directories variables.

   1. Xmacros (use sed command) 

   2. All over bash script functions.

   3. AraSim job files (sed command?)

3. Make the following variables in the bash script:

   
   In xmacros/simulationPECmacroskeleton_prototype.txt (use sed command):

   Grid spacing=0.1
   Frequency= factor of n higher 
   This factor of "n" higher in frequency has to be the same as the Multiplier_factor in the roulette algorithm AND the same as GeoFactor in fitnessFunction_ARA_amy.cpp.
   
   
   

   In roulette_algirithm.cpp

   Length = 50cm/Multiplier_factor; 
   STD= 15cm/Multiplier_factor 
   Radius = 1.5cm/Multiplier_factor; 
   STD= 0.75cm/Multiplier_factor 
   
   Also the "Multiplier_factor" line 89.
   
   This "Multiplier_factor"has to be the same as the GeoFactor in fitnessFunction_ARA_amy.cpp AND the same as the factor higher in frequency in the xmacro.
   ​NOTE:THIS MUST BE RECOMPILED AFTER CHANGING.
   
   
   
   

   In fitnessFunction_ARA_amy.cpp:

   GeoFactor=2 
   This GeoFactor has to be the same as the Multiplier_factor in the roulette algorithm AND the same as the factor higher in frequency in the xmacro
   NOTE: THIS MUST BE RECOMPILED AFTER CHANGING.
   
   
   

4. Get rid of all code we are not using anymore.  

   1. Currently using: XF_loop_AraSeed.sh

      1. Look at what items are being called in here, and remove old versions such as XF_loop_Prototype.sh, XF_loop_Functionized.sh, and the related scripts. 

      2. These can temporarily moved to on "old code" directory and we can delete once we finalize the loop. 

5. Turn tournament on in the GA. 

6. Parallelize the AraActualBicone job.

   1. The AraSim run in Gen 0 that gets Veff for the ARA actual bicone input file is not currently being seeded.

Atttendance: Julie, Max, Cade, Suren, Evelyn, Sophie

Today's tasks:

Max works on AraSim section (needs to do more research and reading)

Julie, Suren, Sophie, Evelyn work on plotting software to plot gain patterns of indiviudals

Cade preps impuse XF code to be implemented

To-Do list:

I. Divide up sections on paper, and work on them. See http://radiorm.physics.ohio-state.edu/elog/GENETIS/1 for section assignments. 

II. Implement impuse code. 

• This will require us to change XFintoARA.py so that it takes less output files. 
• Is there anything else different with the output files?

III. Finish Evelyn and Sophie's plotting software.

• This means also puting it onto Nutau, and adding in lines to the bash script (both to move and save the images, and to run the code!)

Here are things we can do if we are waiting for Remcom:

I. Change sides of bicone to be sinusoid instead of lines.

II. Take known bicone parameters and put it through AraSim to check if it's working properly.

• This also means we should make plots with it to triple check. 

III. Test GA and make it maximize other paramaters to see how it does. 

This is an entry for the usual Wednesday working meeting GENETIS holds. We are largely carrying on from last week's hackathon hammering out the remaining pieces that need to be fixed. See the previous ELOG post for a list of those.

Alex

Amy pointed out on the Monday call that the proper way to simulate the ANITA antenas is to use one source at a time. I modified the script I had so that it makes two simulations per antenna. The first one will use the power source connected to the "VPol" ridges and the second one uses the one connected to the "HPol" ridges. After checking this, it looks like the asymmetry I had pointed out before fell away. In other words, I think XF only wanted to use a single power source anyway. So now we should be able to produce the VV, HH, VH, and HV data that icemc and PUEOsim need.

Ryan and I also worked a bit on the slopes of the ridges. We experimented with the parameter that affects the curvature, which seems to be able to be too large. But many of the designs still look to have almost no slope and many still caved inward rather than sloping outward. We're still working on figuring out what causes that, but I think it means we need another constraint on distance of the top of the inner ridges from the center. 

We also tried figuring out why the sides of the curved part of the ridges don't stay in the same plane as the straight part of the ridges. I'm confident that they do stay in a single plane (I attached a Mathematica notebook that shows as much by finding that the torsion of the curve is zero; for some reason I couldn't print it to a PDF). We think that we should be able to rotate the curve into the same plane as the straight part of the ridge, but I haven't figured out how yet.

 Nick and I continued working on the HPol antenna. We want to check if there are any other things in Steph's example XF file that are significant to the results of the simulation (versus just being there as a representation of the model). It looks like there are, and Nick is working to figure out what they are by deleting them (in a copy) and generating simulation results for them to compare to the original. Otherwise, the most significant parts (the copper plates) are now being made in a script and just need to be connected to one another.

Dylan

I tried to build root from source again and ran into an error 3 hours into the build. I messaged Will and asked if he knew anything about the error. I then reran the build with Will's suggested change and ran into the same error 3 hours later. I did, however, manage to build pueo by sourcing the installation of root in Will's shared pueo directory. I'm now testing getting a txt output file from the simulation as it turns out my initial solution didn't work.

Update:

I got a working version of root installed. Also, I was able to modify pueoSim to output a veff.txt file. Now, I need to modify where it reads in the gain files, and then modify the icemc specific bash scripts to work for pueosim.

Attendance: Julie, Mitchell, Ryan, Alex P., Leo, Alex M.

Today's Tasks

Mitchell: Worked on Python Project and helped with Ryan's final bash project questions

Ryan: Finished the bash project, and is about to start the Python project. 

Leo: Worked on lectures, which will restart on March 17th.

Julie, Alex P, Alex M: Worked on fixing the job XF submission scripts. Worked on making it so that it saves state after each individual's XF simulation is complete. We tried to parallelize as many of the XF scripts as possible, but if we have more than one it asks us to choose a license for each individual -- ie requires user input each time. Since we haven't found a way to get around it AND since running with 1GPU is super fast (under 5 min per individual), we are going to run them in series.

TO-DO List

1. Scaling test run: Do a test run to see if the scaling (when we also scale the grid spacing) actually affects the run time for XF. (PASSED THIS TASK TO JULIE. WILL DO TOMORROW 3/4)
2. Evolve: Right now, we are using an interactive job with only a CPU running at half scale, and have just edited our software so that we submit jobs for the XF simulations requesting 1GPU (could be 2GPU if we choose to edit it so). 
   1. WHY WE ARE DOING THIS:
      1. Remember that we only need an interactive job when XF initializes and reads in the DNA for each individual. The GUI closes once the simulations start and we no longer need an interactive job. 
      2. We were requesting a 1GPU interactive job and just running XF from the command line; however, OSC has been SUPER busy lately and we haven't been getting them (especially since we need long wall times if we are running the full simulations over many generations within the interactive job). However, we realized that if we submit an interactive job at a CPU (so that XF can allow the GUI to pop up), and then run the simulations by requesting 1-2GPUs we can then request a much smaller wall time and it will be quicker for the simulations to run --- ie we would be requesting a wall time of about 10 minutes using 1-2GPUs per simulation for each job submission versus 5+ hour wall time of an interactive job using 1-2GPUs.
      3. WE HAVE TESTED THIS, AND WE HAVE ERRORS. WE WORKED ON FIXING THEM TODAY, AND HOPE TO FINALIZE THEM TOMORROW (3/4) AND RUN!

3. Parallelize the AraActualBicone job.

   1. The AraSim run in Gen 0 that gets Veff for the ARA actual bicone input file is not currently being parallelized. (ALEX M)

   2. As of 3/3, I have not gotten an update on this. I will check on the status of this tomorrow. 

4. Comment code thoroughly (Everyone needs to help with this)

   1. ​Julie made some progress on this over the weekend. 

      1. Made comments on what each bash script version does, and cleaned up our directories from old code so that it doesn't confuse the others when they start helping 

   2. WAY more needs to still be done. 

5. Get new proposal results for March 12th deadline.

Today's Tasks

Julie: Write bash practice assignment, assist others when they have questions, work on Git lecture. (Will finish bash assignments early next week)

Evelyn, Mitchell, Eliot: Make progress on the "Learn the Command Line" course on Code Academy. (Should be done by the end of the week?)

Leo: Start gathering information for the "physics of antennas" lecture series! Note: Remember that Leo wants to be an educator, and would like us to find opportunities to work on "teaching" items. Most of the students don't have any background on how antennas work. He started learning about them and is going to do a lecture series (likely the first 20 minutes of every meeting) starting in a few weeks! (Should start series by hopefully 2/21)

Alex M: Clean up unused code. Fix hard-coded directories (likely finished & tested by end of day 2/11)

Alex P: Fix hard-coded variables (likely finished & tested by end of day 2/11)

I've tacked on our to-do list in the meantime so we can keep carrying the items still pending along with us. As we complete them, I will drop them off the list! 

TO-DO List

The following is our coding to-do list:

1. Get our Ara sim error fixed (Alex, Julie) 

   1. It seems that if AraSim jobs are left to run while the user is away, it doesn't properly write one or more of the flags, and makes the loop get stuck at AraSim. We must then re-run from that state by manually editing the Save State text file.

   2. We are going to try to get XF to run at the command line so that we can submit it as a job. This would alleviate us using an interactive job. We *hope* this would fix the error since the user wouldn't be "gone" while utilizing a GPU in an interactive job. 

      1. Alex emailed XF to ask a few questions! Stay tuned!

2. Make Hardcoded directories variables.

   1. Xmacros (use sed command) 

   2. All over bash script functions.

   3. AraSim job files (sed command?)

3. Make the following variables in the bash script:

   
   In xmacros/simulationPECmacroskeleton_prototype.txt (use sed command):

   Grid spacing=0.1
   Frequency= factor of n higher 
   This factor of "n" higher in frequency has to be the same as the Multiplier_factor in the roulette algorithm AND the same as GeoFactor in fitnessFunction_ARA_amy.cpp.
   
   
   

   In roulette_algirithm.cpp

   Length = 50cm/Multiplier_factor; 
   STD= 15cm/Multiplier_factor 
   Radius = 1.5cm/Multiplier_factor; 
   STD= 0.75cm/Multiplier_factor 
   
   Also the "Multiplier_factor" line 89.
   
   This "Multiplier_factor"has to be the same as the GeoFactor in fitnessFunction_ARA_amy.cpp AND the same as the factor higher in frequency in the xmacro.
   ​NOTE:THIS MUST BE RECOMPILED AFTER CHANGING.
   
   
   
   

   In fitnessFunction_ARA_amy.cpp:

   GeoFactor=2 
   This GeoFactor has to be the same as the Multiplier_factor in the roulette algorithm AND the same as the factor higher in frequency in the xmacro
   NOTE: THIS MUST BE RECOMPILED AFTER CHANGING.
   
   
   

4. Get rid of all code we are not using anymore.  

   1. Currently using: XF_loop_AraSeed.sh

      1. Look at what items are being called in here, and remove old versions such as XF_loop_Prototype.sh, XF_loop_Functionized.sh, and the related scripts. 

      2. These can temporarily moved to on "old code" directory and we can delete once we finalize the loop. 

5. Turn tournament on in the GA. 

6. Parallelize the AraActualBicone job.

   1. The AraSim run in Gen 0 that gets Veff for the ARA actual bicone input file is not currently being seeded.

       

Today we continued with the to-do lists items previously posted. I marked all the to-do list items that have been completed with green and a slash. Note that I have also added a new item at the bottom of the to-do list.

Today's Tasks

Julie: Write bash practice assignment (almost done!)

Evelyn, Mitchell, Eliot, Ryan: Make progress on the "Learn the Command Line" course on Code Academy. (Mitchell is done, and the others are close.)

Leo: Start gathering information for the "physics of antennas" lecture series! Note: Remember that Leo wants to be an educator, and would like us to find opportunities to work on "teaching" items. Most of the students don't have any background on how antennas work. He started learning about them and is going to do a lecture series 

Alex M: Make sure antenna scaling is made a variable in the bash script everywhere that it exists, start mass commenting all of the code so it reads as cleanly as the GA does. (Finished making the antenna scaling a variable. needs to be tested, but we waited forever for wall time)

Alex P: Fix hard-coded variables (done but tried to test it. Waited for wall time and never got it), and looked into ways to get around requesting an interactive job. For some reason, we don't need an interactive job to forward things like plots, but we do for the XF UI. We were trying to figure out why this is, and if there's a way to fix this. So far, no luck. 

AMY QUESION: Could we actually consider running on NASA computers? I think Alex P. found a way the NASA computing centers let you adjust for this issue, and it didn't work on OSC. If we end up not finding a solution with OSC, is it at all possible to have XF installed with NASA computing centers? (NOTE: I JUST ASKED ALEX FOR MORE DETAILS AND WILL ADD THEM HERE TOMORROW -- IE WEDNESDAY)

TO-DO List

The following is our coding to-do list:

1. Get our Ara sim error fixed (Alex, Julie) 

   1. It seems that if AraSim jobs are left to run while the user is away, it doesn't properly write one or more of the flags, and makes the loop get stuck at AraSim. We must then re-run from that state by manually editing the Save State text file.

   2. We are going to try to get XF to run at the command line so that we can submit it as a job. This would alleviate us using an interactive job. We *hope* this would fix the error since the user wouldn't be "gone" while utilizing a GPU in an interactive job. 

      1. Alex emailed XF to ask a few questions! Stay tuned!

2. Make Hardcoded directories variables

3. Make the following variables in the bash script:

   
   In xmacros/simulationPECmacroskeleton_prototype.txt (use sed command):

   Grid spacing=0.1
   Frequency= factor of n higher 
   This factor of "n" higher in frequency has to be the same as the Multiplier_factor in the roulette algorithm AND the same as GeoFactor in fitnessFunction_ARA_amy.cpp.
   
   
   

   In roulette_algirithm.cpp

   Length = 50cm/Multiplier_factor; 
   STD= 15cm/Multiplier_factor 
   Radius = 1.5cm/Multiplier_factor; 
   STD= 0.75cm/Multiplier_factor 
   
   Also the "Multiplier_factor" line 89.
   
   This "Multiplier_factor"has to be the same as the GeoFactor in fitnessFunction_ARA_amy.cpp AND the same as the factor higher in frequency in the xmacro.
   ​NOTE:THIS MUST BE RECOMPILED AFTER CHANGING.
   
   
   
   

   In fitnessFunction_ARA_amy.cpp:

   GeoFactor=2 
   This GeoFactor has to be the same as the Multiplier_factor in the roulette algorithm AND the same as the factor higher in frequency in the xmacro
   NOTE: THIS MUST BE RECOMPILED AFTER CHANGING.
   
   
   

4. Get rid of all code we are not using anymore.  

   1. Currently using: XF_loop_AraSeed.sh

      1. Look at what items are being called in here, and remove old versions such as XF_loop_Prototype.sh, XF_loop_Functionized.sh, and the related scripts. 

      2. These can temporarily moved to on "old code" directory and we can delete once we finalize the loop. 

5. Turn tournament on in the GA. 

6. Parallelize the AraActualBicone job.

   1. The AraSim run in Gen 0 that gets Veff for the ARA actual bicone input file is not currently being seeded.

7. Comment code thoroughly 

   1. The Gen alg is commented so thorougly that it can read like a book. We want this with ALL of our code; it will help all the less familiar people get comfortable. 

Today's Tasks

Julie: Write Python project! I have officially finished the bash course for the students. It can be found here: https://docs.google.com/document/d/1nGdPrwfYJOrO6NBT76Wsh6T0hDmGHo2-23oJ1WyeNcs/edit?usp=sharing. Mitchell is currently trying it out and will give me comments. 

Mitchell: Work through my bash project and give me comments!

Evelyn, Eliot, Ryan: Make progress on the "Learn the Command Line" course on Code Academy. 

Leo: continuing gathering information for the "physics of antennas" lecture series! These will take place every Friday meeting starting next week. 

Alex M: Try to get scaling into the xmacro, comment the code better.

Alex P: Add on the AraSim fixes mentioned below!

TO-DO List

The following is our coding to-do list:

1. Get our Ara sim errors fixed

   1. Explanation of issue (1): Usually 1 of ~100 AraSim jobs never completes. OSC shows "completed", but the flag is never written and the output file is not filled in entirely. The error file shows the following.

      1. Carl believes this is due to the way we are saving our AraSim outputs. Instead of saving the files directly to our desired location, we are going to save them to $TMPDIR and use "pbsdcp" to copy them from $TMPDIR to our desired location. This may alleviate the problem since it will not be writing all the files to our desired location at once. 

DATA_LIKE_OUTPUT=1,2 doesn't work with DETECTOR=0,1,2
DATA_LIKE_OUTPUT controls data-like output into UsefulAtriStationEvent format; without a real station selected (using DETECTOR==3,4), the mapping to the data-like output will not function correctly
There are 1 errors from settings. Check error messages.
SysError in <TFile::Flush>: error flushing file outputs/AraOut.setup.txt.run29.root (Stale file handle)

1. Explanation of issue (2): We keep timing out of the interactive job when AraSim is running (if the user isn't remaining active on their terminal).
   1. OSC suggested that we use a virtual desktop (via ondemand) insead of using an interactive job. We will try this. 
      • Note: We should now be requesting 2GPUs instead of 1GPU. 

1. Make the following variables in the bash script. 

   1. Alex M is working on this. We can't directly make this a variable. We need to take the array of frequencies and multiply it by some value and THEN feed it into the xmacro. He is looking into whether of not he can directly do this in bash. If not, I have written a small python script that does this, and he can write it to that file; however, he can't append it at the end so it's not the ideal way. We are going to continue to try to do it through bash first. 

In xmacros/simulationPECmacroskeleton_prototype.txt (use sed command):

Grid spacing=0.1
Frequency= factor of n higher 
This factor of "n" higher in frequency has to be the same as the Multiplier_factor in the roulette algorithm AND the same as GeoFactor in fitnessFunction_ARA_amy.cpp.

1. Turn tournament on in the GA. 

2. Parallelize the AraActualBicone job.

   1. The AraSim run in Gen 0 that gets Veff for the ARA actual bicone input file is not currently being parallelized.

3. Comment code thoroughly

   1. The Gen alg is commented so thorougly that it can read like a book. We want this with ALL of our code; it will help all the less familiar people get comfortable. 

Today's Tasks

Julie: Continue writing Python project! 

Mitchell, Evelyn, Eliot (not here on Tuesdays), Ryan: Work through bash project

Leo: continuing gathering information for the "physics of antennas" lecture series! We start this Friday at 4pm. 

Alex M: Try to get scaling into the xmacro, comment the code better.

Alex P: Add on the AraSim fixes mentioned below! He is testing this now. We are waiting for updates. 

TO-DO List

The following is our coding to-do list:

1. Get our Ara sim errors fixed

   1. Explanation of issue (1): Usually 1 of ~100 AraSim jobs never completes. OSC shows "completed", but the flag is never written and the output file is not filled in entirely. The error file shows the following.

      1. UPDATE: Alex P. has edited our bash job script to save the outputs to $TMPDIR and then to pbscp (submit a "job" to copy them over) them into our correct directory that the bash script needs them in. He is testing this now and will report back. 

      2. Note: we still need to make sure that every time we run that we divide AraSim runs into MANY more jobs (ie further parallelize by dividing into, say, 100 instead of 10). 

DATA_LIKE_OUTPUT=1,2 doesn't work with DETECTOR=0,1,2
DATA_LIKE_OUTPUT controls data-like output into UsefulAtriStationEvent format; without a real station selected (using DETECTOR==3,4), the mapping to the data-like output will not function correctly
There are 1 errors from settings. Check error messages.
SysError in <TFile::Flush>: error flushing file outputs/AraOut.setup.txt.run29.root (Stale file handle)

1. Explanation of issue (2): We keep timing out of the interactive job when AraSim is running (if the user isn't remaining active on their terminal).
   1. OSC suggested that we use a virtual desktop (via ondemand) instead of using an interactive job. We will try this. 
      • Note: We should now be requesting 2GPUs instead of 1GPU. UPDATE: Alex P. requested one, but it took too long and he had to leave before he got it. 

1. Make the following variables in the bash script. 

   1. UPDATE: Alex M. figured out how to pass the scale factor into the xmacros. We used the "tr" command in bash. He will be testing this and can update us at the Thursday meeting. 

In xmacros/simulationPECmacroskeleton_prototype.txt (use sed command):

Grid spacing=0.1
Frequency= factor of n higher 
This factor of "n" higher in frequency has to be the same as the Multiplier_factor in the roulette algorithm AND the same as GeoFactor in fitnessFunction_ARA_amy.cpp.

1. Turn tournament on in the GA. 

2. Parallelize the AraActualBicone job.

   1. The AraSim run in Gen 0 that gets Veff for the ARA actual bicone input file is not currently being parallelized.

3. Comment code thoroughly

   1. UPDATE: Alex commented code that utilized the scale factor (Part_E.sh). More still needs to be done. 

Today's Tasks

Julie: I finished the Python assignment. I am making sure my solutions encompass all of the different ways to solve this, and will then start on the C++ project. 

Mitchell, Evelyn, Eliot, Ryan: Work through my bash project.

Leo: At 4 pm today Leo will begin his "physics of antennas" lecture series! 

Alex M: All scaling is corrected to be set at the bash script level. Alex is now going to work on commenting code!

Alex P: Still working on our AraSim edits!

TO-DO List

The following is our coding to-do list:

1. Get our Ara sim errors fixed

   1. Explanation of issue (1): Usually 1 of ~100 AraSim jobs never completes. OSC shows "completed", but the flag is never written and the output file is not filled in entirely. The error file shows the following.

      1. UPDATE: Alex still trying to figure out how to input Car's suggestion. He also needs to add this for the flag files that are being saved. Here is his most recent question:

Alright so I have pbsdcp working with generic files and I now have it working in the AraSim call but my current worry is that it wasn't creating a file unless I had

./AraSim setup.txt $runNum outputs/ a_${num}.txt > $AraSimDir/outputs/a_${num}.txt

So I just did this to have some a_1.txt, before we had the last line writing directly to our antennaPerformance metric (this makes the AraOut file typically I just kept it as this name for testing purporses), now if I get rid of the portion after the > or the part after outputs/ it doesn't seem to make a file that we need. I think I don't understand to get the full output of AraSim without using this. Because I don't understand how using this is and then using pbsdcp is any more efficient.

Now maybe if I had the portion right after the > go directly into TMPDIR instead of using pbsdcp to both scatter to and gather from TMPDIR that would be better

1. Make the following variables in the bash script. 

   1. UPDATE: Alex M. finished editing the scaling factor to be at the bash level for every piece of code that uses it. 

2. Turn tournament on in the GA. 

3. Parallelize the AraActualBicone job.

   1. The AraSim run in Gen 0 that gets Veff for the ARA actual bicone input file is not currently being parallelized.

4. Comment code thoroughly

   1. UPDATE: Alex commented code that utilized the scale factor (Part_E.sh). He is now going to begin commenting the other code to clean it up. We want it to be read as cleanly as our GA. 

Attendance: Julie, Mitchell, Ryan, Evelyn, Eliot, Alex M.

Today's Tasks

Julie: Worked on C++ project, and helping others with the Python project; I also made additions, clarifications, and edits to the Bash project (also updated the solutions and pushed them to gitHub). I still need to do the following:

• Add the instructions from the google doc to the GitHub pages for both the Python and the Bash projects.
• Work on the C++ project!
• Run a test evolution this weekend to make sure AraSim stuff is fixed and that XF job submissions work. 
• EVOLVE

Mitchell: Started Python Project.

Evelyn, Eliot, Ryan: Work through my bash project. 

Leo: Worked on lectures. 

Alex M: Helped make XF job scripts. Needs to do the following:

• Parallelize the AraActualBicone properly. 
• EVOLVE!

Alex P: Needs to do the following:

• Do a test run to see if the scaling (when we also scale the grid spacing) actually affects the run time for XF.

TO-DO List

1. Scaling test run: Do a test run to see if the scaling (when we also scale the grid spacing) actually affects the run time for XF. (ALEX P)
2. Evolve: Right now, we are using an interactive job with only a CPU running at half scale, and have just edited our software so that we submit jobs for the XF simulations requesting 1GPU (could be 2GPU if we choose to edit it so). 
   1. WHY WE ARE DOING THIS:
      1. Remember that we only need an interactive job when XF initializes and reads in the DNA for each individual. The GUI closes once the simulations start and we no longer need an interactive job. 
      2. We were requesting a 1GPU interactive job and just running XF from the command line; however, OSC has been SUPER busy lately and we haven't been getting them (especially since we need long wall times if we are running the full simulations over many generations within the interactive job). However, we realized that if we submit an interactive job at a CPU (so that XF can allow the GUI to pop up), and then run the simulations by requesting 1-2GPUs we can then request a much smaller wall time and it will be quicker for the simulations to run --- ie we would be requesting a wall time of about 10 minutes using 1-2GPUs per simulation for each job submission versus 5+ hour wall time of an interactive job using 1-2GPUs.
      3. WE HAVE NOT TESTED OUR .SH JOB SUBMISSION SCRIPTS! JULIE WILL DO IT THIS WEEKEND!

3. Parallelize the AraActualBicone job.

   1. The AraSim run in Gen 0 that gets Veff for the ARA actual bicone input file is not currently being parallelized. (ALEX M)

4. Comment code thoroughly (Everyone needs to help with this)

5. Get new proposal results for March 12th deadline.

Attendance: Julie, Evelyn, Mitchell, Ryan, Alex M.

Since we have been having issues with the lack of availability of GPUs on OSC, we have come up with a new to-do list:

1. Try to keep running on Pitzer with 1 core instead of 40. Based on first attempts, it is saying this will reduce wait times to ~3 hours (will vary based on usage), and after that 3 hours, human interaction will be needed. So the plan is, someone requests a job in the morning, and when the interactive job starts, the person is responsible for doing the thing which will allow the generation to run. We will aim for one generation per day under this plan, taking turns. Better than no evolving.
2. I have asked Heechang at OSC if we can install the latest version of XF on Ruby and/or Owens. That would be better. There seems to be more available on Owens at least.
3. Our initial attempt to run on Ruby is giving us an error which is probably to do with different inputs required in different versions. Will attempt to pursue this as a last resort, but the older version of XF does require lots more human interaction so until other options are exhausted it's not worth it.

Today's Tasks

Mitchell: Worked on Python Project, learned to evolve (note: Mitchell may leave the group at the end of the semester)

Ryan: Starting the Python project

Evelyn: Finish bash project.

Julie: C++ project, look at Amy's proposal draft. 

Alex M:  Fix XF solver to submit each individual as a separate job (thus we can request a smaller wall time for each job)

TO-DO List

1. Scaling test run: Do a test run to see if the scaling (when we also scale the grid spacing) actually affects the run time for XF. 
   1. We are going to pause on working on this, because we are struggling to get GPU time. We will test this whenever we see OSC less busy (ie if we log on at a lucky time). 
2. Evolve: Try to keep running on Pitzer with 1 core instead of 40. Based on first attempts, it is saying this will reduce wait times to ~3 hours (will vary based on usage), and after that 3 hours, human interaction will be needed. So the plan is, someone requests a job in the morning, and when the interactive job starts, the person is responsible for doing the thing which will allow the generation to run. We will aim for one generation per day under this plan, taking turns. Better than no evolving.

3. Parallelize the AraActualBicone job.

   1. The AraSim run in Gen 0 that gets Veff for the ARA actual bicone input file is not currently being parallelized. (ALEX M)

   2. As of 3/5, I have not gotten an update on this. I will check on the status of this. 

4. Comment code thoroughly (Everyone needs to help with this)

   1. ​Julie made some progress on this over the weekend. 

      1. Made comments on what each bash script version does, and cleaned up our directories from old code so that it doesn't confuse the others when they start helping 

   2. WAY more needs to still be done. 

5. Get new proposal results for March 12th deadline:

   1. ​Julie is looking at what Amy wrote here 

I just wanted to give an update on what I've been doing today for GENETIS. I went ahead and recorded myself running the loop. It should be useful for new people we're onboarding but I also went through a whole generation, which I think is interesting for everyone to understand the timescale of each part of the loop. I'm making a small edit (I have no clue how to edit videos haha) so once I'm done I'll try to send it via email and post it on slack. I also started a long run Amy requested using antennas which are asymmetric in length and opening angle and 50 individuals per generation. Since it has so many individuals it will take a long time to go through each generation. I also had to decrease the number of AraSim jobs (thus increasing the number of neutrinos per job) because otherwise we'd be submitting too many jobs and have a lot of them blocked, holding us back from running. If anyone has problems with me using up too many jobs let me know and I can try to decrease the numer of jobs per individual futher (right now I have 10 jobs per individual with 3000 neutrinos per job, which gives us 500 AraSim jobs per generation).

Attendance: Julie Rolla, Alex M. 

Today Alex and I met to discuss how to move forward given the COVID-19 restrictions. We have resent out a message polling people's schedules to create online working sessions. Here is what we are planning:

Once we collect times for each group, we will have separate meetings for all of these entities. This way it's easier for us to meet without all having different items to discuss/work on. All of these will be zoom meetings moving forward. Meeting info will be announced shortly!

Tasks for "the Loop Group":

1. Make wall time a variable for XF sim.
2. Test current software edits.
   1. Such as item #1 above
   2. Make sure the bottleneck function is working.
      1. Right now it seems to be working; however, we keep hitting a wall time error. If we can make the wall time a variable then we can be sure the bottleneck function is working, and the wall time is truly the error. 
3. Get Ruby working on XF
   1. when I type:
      /usr/local/xfdtd/7.8.1.4/remcom/bin/xfdtd        I get:

      Info: Using latest available version (7.8.1.4)
      Info: Running /usr/local/xfdtd/7.8.1.4/remcom/XFdtd_7.8.1.4/bin/xfdtd 
      Error: Unable to find executable to run.  This command must exist

4. Get Alex's account of Ruby working. 
   1. He received an email and is still unable to log in. He is touching base with them again. 

Tasks for "the Training Group":

1. Julie and Alex need to finish the C++ project! 

The current working draft of the GENETIS paper can be seen here: https://www.overleaf.com/6783528497tvqbjphsgvzn

Additionally, here's our outline for the paper. The section assignments are as follows:

I. Intro (Julie)

II. Types of Algorithms (Suren)

III. XFdtd (Cade)

IV. AraSim -- will be called something else... (Max)

V. Fitness function (Suren)

VI. Results (Julie)

VII. Conclusion (We will see..)

Note that the order of the sections may change.

This is a general update. Most of our recent meetings have been to teach the students to use Git properly, or to answer questions on the training modules. 

Update

1. Git Update: I recently cleaned up our Git and GitHub so that we have a better way to develop this code. Once I cleaned it up the goal is to educate all of the students to properly use Git so we develop code correctly moving forward. The repository had become quite a mess. The students were constantly developing on a branch, and never merging it back to the main branch. This means that we never had a working copy that we could run on while edits were being made in parallel. I attempted to merge the edits to the main branch with the hopes of getting this functioning so we could evolve while working on the bottleneck function; however, it looks like someone with poor knowledge of Git edited the main branch and we had an abundance of merge conflicts. 

What I've done now is reset the repository so that all merge conflicts are resolved and a working copy exists on the main branch. I've also added a .gitignore that will ignore all data files. Git doesn't like these because (1) it hates when tracked files disappear -- ie when we delete trial run data, if someone accidentally pushed it then Git complains-- and (2) it means it we constantly are adding more untracked files (every single new file we make is untracked and then gets tracked, like the flag files). So, I've had it only track useful code. 

The goal once I train the other on Git is to move the place of the data files and make it it's own git repository (once we start doing fewer trial runs so we won't lose important data). If the others learn to check the status of files before pushing them, they can make sure they don't push useless trial data sets, and either (1) delete them, or (2) add them to the .gitignore.

As for now, we successfully have the working, run-able copy on the main branch now. When any more revisions are made (e.g. the talk of adding the wall time variable), we can branch and edit on that so we still run. 

2. Evolution/bottleneck update: We got the loop working to the point that it can run all the way through a generation without fatal problems. The only potential problem we can see right now is if we need to go backward in the savestate. We're working on making that not a problem--the solution is to better organize the output files and data, which we were interested in anyway. Right now we are running with 10 individuals in each generation. Alex got through 2 generations without a problem, so he'll keep running but now we're trying to make some of these more minor fixes so we may have to stop the run. Alex does have plots (they don't look great, but at least they're being made correctly without error). At the moment we are swimming in GPUs. We haven't had much trouble at all this week with getting them, so that's why we were able to get 2 generations done between around midnight last night and 2ish today; not sure how long that will persist, but Pitzer has had relatively little usage whenever we've looked lately.

Comments on the plots: We're discussing this now, we don't think the individuals should really be color coded/separate on the legend because it is not like they follow one individual across multiple generations; however, we do need to label the dotted line. Another thing we were considering is decreasing the penalty factor for when the cones are too big for the hole. If we get something with a much much better Veff but it's only slightly larger than the hole, it might be worth it to help that one stick around a little bit more.

3. Wall time Variable Update: Also, last week Carl and Alex talked about the variable wall time again. We decided the best idea is to start by implementing a variable wall time based on the cylindrical volume that he mentioned before, but in the future, we might also be able to check the time of each GPU job and then plot it against the dimensions of the cone/cylindrical volume to get a better estimate of the wall time for each job

Meeting Attendance: Julie, Alex M., Alex P, Eliot, Leo, Evelyn, Mitchell, Ryan. 

Today is the deadline for confirming our attendance at the virtual APS meeting. Alex M. will be confirming and we will be orienting ourselves so that we can have some results. Here is our to-do list for today:
 

1. Create a method for automatically adding our plots to dropbox each day. 
   • The way we are going to do it is by adding the command: mail -s "Test Subject" someemailaddress@gmail.com < testScript
     • We will create a random email to use for this that will only be receiving emails from this command. 
     • We are trying to find and set up a client that as soon as this random email address receives an email, it uploads it to the Dropbox here. 
   • This is still being worked on. 
2. Start developing a database of old antenna runs and parameters, and pull data from that for any repeat dimensions. 
   • This should speed up some of our run time. 
   • We want to create a database of old antenna dimensions and their associated UAN files. That way, if we repeat an antenna geometry very closely, instead of re-submitting a job to produce the gain patterns, we can pull from one that has already been created.
   • This should limit:
     • The number of jobs we have to submit.
     • The number of GPU hours/minutes we have to request and wait for.
   • These should probably still be run through AraSim, however, since the energy could change from run to run.
3. Keep running and collecting data. 
   • Do we want to run and have data for 20ish generations with 10 individuals for APS?
     • We have already made some progress on this so we could likely make more generations for APS if we keep going. 
   • Do we want to instead switch to doing something like 20 individuals even if we get fewer generations completed?

Notes apropos developing our database for antenna info. This is how we are going to do it, and what we need to develop in our code:

1. Add all info to csv files.
   • The csv files will have columns: length, radius, theta, uan file directory and name (ie /file/location/database#/).
   •  # will be the row that this matches the L, R, Theta within the csv file
   • Inside /file/location/database#/ will be all the uan files for each frequency for these dimensions. 
     • We need an "if/else" statement in the bash script that tells the code to look in this csv file (read it in). 
       • IF the produced L, R, Theta match any of those that currently exist, save the 4th column (file location) to a variable (say we call it ExistingUan=/file/location/database#/). 
         • Then cd ExistingUan
         • Make a loop that:
           • does CP /file/location/database#/gen_individual_freq.uan BiconeEvolutionOSC/BiconeEvolution/current_antenna_evolution/XF_Loop/Evolutionary_Loop/Run_Outputs/$RunName/ (ie to the file name and location that XFintoAra.py reads. This will make sure that it has the data as if we actually completed the XF simulations for the antennas that had repeat sizes from our past runs (I mean past by runs we may have EVER done -- not past generations). 
             • We need to copy ALL of the frequency uan files for that individual into Run_Outputs/$RunName 
       • Else fill another line in that csv file with the L, R, Theta of this individual so we can keep it for later. 
         • mkdir the new directory /file/location/database#/
         • Add the name of this new directory (ie database#) to the 4th column on the csv file next to the correlating L, R, Theta.  
         • Then submit the XF job for this individual. 
         • Once it finished the XFSolver job and produces the .uan files (for each frequency), do a loop that:
           • cp individual_freq.uan /file/location/database#/gen_individual_freq.uan
             • Note that I wrote to cp individual_freq.uan instead of gen_individual_freq.uan. Note that XF produces it as individual_freq.uan, and it is changed to gen_individual_freq.uan when we move it into the Run_Outputs/$RunName directory.
             • So you can do it either (1) after it moves it to Run_Outputs/$RunName with the name gen_individual_freq.uan, or (2) before it moves the file with the name individual_freq.uan. 
2. We need to make a new directory that holds all of these database# directories and their uan files, and the csv file the "if/else" statement checks. 
   • We need to add this new directory to the .gitignore!

Note that Pitzer usage over the weekend was much better! Maybe weekends are our best time to run? Today (Monday) has been a little bit worse. 

Over the past few days the following items have been accomplished:

• We got the jobs to stop canceling on OSC.
  • Issue was the destination command for the error and output files for the GPU jobs inside the GPU job script; we fixed it by taking it out.
    • This issue was causing jobs to fail. They are no longer failing, and the loop is working again.
• Set up a way to be able to automatically upload both the L, R, Theta plots and the Fitness Score plot to dropbox as we work.
  • Set up an email that automatically uploads to dropbox under the folder GP_Antennas/Updates/DailyFitnessScoreImages when you send an attachment with that email.
  • We added this command into the loop:
    mail -s "Subject" dropbox.2dwp1o@zapiermail.com < FScorePlot2D.png
    • It automatically has it upload an image when the loop finishes a generation. It's currently so that the file will be named whatever the subject of the email is so we can control that by whatever is after the -s command. It's currently set to not overwrite.
  • Still need to add the Veff plot to dropbox. 
• Made gitpush.sh and gitpushforce.sh
  • Each time one of us pushes changes to GitHub, it makes a permssion lock on the directories. From now on, after looking at what files are changed (using git status. Remember to make sure to add any data files missed to the .gitignore)users should do:
    • git add .
    • git commit -m "add message" or git commit --amend --no-edit
    • Then run: ./gitpush.sh if you made a full new commit, or use ./gitpushforce.sh if you amended this edit to the previous commit. They both exist in: /fs/project/PAS0654/BiconeEvolutionOSC
      • These files run the git push or git push -f commands and then immediately run a chmod to change permissions after. 
• Alex is starting the APS slides!
  • He should be sharing them shortly. I told him to look at Suren's from last year for guidance. 
• We have started working on adding the feature that pulls non-unique (ie repeat parameter) individuals from an antenna database. 
  • It's currently being developed on the Antenna_Database branch in git. 
  • Alex has started writing c++ code that reads in the csv file and appends unique items to it. 
  • Comments from Amy we need to remember: 
    • Doing "==" on doubles is bad!
    • We want this "likeliness" to be between 1 and 10% (start with 10%). 
      • Remember: is this difference between L stored and new L a fraction of the wavelength?

The students have been meeting to work out the summer schedule. We are dividing into to groups:

(1) Paper writing: Julie, Alex M., Alex. P

• We need Amy to start an overleaf draft (she has the premium version that allows you to share with more than 1 person)

  • https://www.overleaf.com/5612932179jzykrkvztqnb

• In tomorrow's collaboration meeting (Now Fridays at 3pm) we are going to request discussion on an outline

(2) Adding wall time variable structure: Eliot, Evelyn, Leo (Alex P. will help oversee)

• In order to prepare group (2) Alex M. has been going over the edits he started and is discussing his code. We also are taking some time to catch them up on the current code -- ie how it runs, what all of the pieces involved do -- so that they can know how to test the code after edits are made, and where each essential piece may lie. We will be going over this for the next few days in good detail. 
• Group (2) will then be coming up with a scheduled time each day that they will be coworking on zoom. This will work the same as if they were sitting in an office together. They'll announce this time to everyone once it is set.

Alex P, Alex M, and I are going to push to rapidly finish this antenna database so that the others can create a new branch to work on the wall time variable with. This will leave us very busy for the next few days. 

We met today for a long time and pushed to catch up. The following is what we have done:

(1) Worked for a bit on our current tasks. They will work on these current tasks before they start on their summer projects. 

• Antenna database (Julie, Alex P., Alex M.)
• Worked on a paper outline (Julie, Alex M.)
• Worked on wall time variable (Eliot, Evelyn, Leo)

       Note I also have to: (1) Fix merge conflict and make sure students are good about only editing the dev branch moving forward. (2) Get branches and GitHub set up for both the asymmetric bicone and the non-linear bicone. 

(2) Went over the details of how the loop works as a group.

• Went over how each piece works.
• Went over how to run it. 
• Went over best ways to test new edits. 

(3) Went over summer tasks.

• Eliot: Add parameters to bicone (ie make bicone asymmetric)
• Julie & Alex M.: Paper, and helping others
• Alex P.: Frequency database for AraSim (to speed up AraSim)
• Evelyn & Leo: Make non-linear bicone (wiggly sides) 

GENETIS Loop Work: Fixing Consol Errors By Julie Rolla

1. Python programs are not running: I think either (1) the user isn't module loading python, or (2) they are loading the wrong version of Python (remember there's a discrepancy for Python2, and Python3). 

Solution: I added a line "module load python/3.6-conda5.2" in our araenv.sh script to address the issue. All of the users are sourcing the araenv.sh script in their .bashrc when they login. It's in the directory: /fs/project/PAS0654/BiconeEvolutionOSC. I also added a line in the XF_Loop.sh script (at the top in the "Initialization of Variables" section) that sources araenv.sh just in case the user hasn't put that in their .bashrc. The line reads: source /fs/project/PAS0654/BiconeEvolutionOSC/araenv.sh

Error message fixed:

 0 
Data successfully read. Data successfully written.
Fitness scores successfully written.
Fitness function concluded.
rm: cannot remove 'runData.csv': No such file or directory
Unable to parse the pattern
^CTraceback (most recent call last):
  File "LRPlot.py", line 99, in <module>
    plotLR(generations, lengths, radii, g.numGens, g.destination)
  File "LRPlot.py", line 45, in plotLR
    plt.show()
  File "/usr/lib64/python2.7/site-packages/matplotlib/pyplot.py", line 143, in show
    _show(*args, **kw)
  File "/usr/lib64/python2.7/site-packages/matplotlib/backend_bases.py", line 108, in __call__
    self.mainloop()
  File "/usr/lib64/python2.7/site-packages/matplotlib/backends/backend_gtk.py", line 83, in mainloop
    gtk.main()
KeyboardInterrupt

2. rm runData.csv error: The following error was printed to consol

rm: cannot remove 'runData.csv': No such file or directory

I found these line in section (E) of the loop.

cd "$WorkingDir"
rm runData.csv
python gensData.py 0
cd Antenna_Performance_Metric
python LRPlot.py "$WorkingDir" "$WorkingDir"/Run_Outputs/$RunName 1 $NPOP
cd ..
# Note: gensData.py floats around in the main dir until it is moved to         

 I think this error is fine. This is how it works: (Step 1) removes runData.csv, if it exists. (Step 2) Runs gensData.py, which creates a new runData.csv. We wanted to delete it before the .py program was run at the start of the run, because gensData.py appends data to runData.csv, if it already exists. (alternatively if runData.csv doesn't exists it creates it-- which is what we want in this case). That's also why we don't delete it later in the "looping part" of XF_Scripts.sh (part G). We want it to append at this point, so we only delete it in the very beginning to make sure it doesn't append to an old run.

3. cp: cannot stat 'Antenna_Performance_Metric/1.uan': No such file or directory (similarly for 2.uan, 3.uan...etc): This error exists just as the fitness score is being calculated -- ie part E. Here is the print:

cp: cannot stat 'Antenna_Performance_Metric/1.uan': No such file or directory
cp: cannot stat 'Antenna_Performance_Metric/2.uan': No such file or directory
Congrats on getting a fitness score!
  File "FScorePlot.py", line 1
    -*- coding: utf-8 -*-
    ^
IndentationError: unexpected indent

Note that we have a few errors here. Right now I am directly addressing the error with the "cp" command. It looks like these files are actually being saved as gen_individual#.uan. For example 0_1.uan. The current line in part E says:

for i in `seq 1 $NPOP`

do

    #Remove if plotting software doesnt need                                                                                                                                                             

    #cp data/$i.uan ${i}uan.csv                                                                                                                                                                          

    cp Antenna_Performance_Metric/$i.uan "$WorkingDir"/Run_Outputs/$RunName/${i}.uan

done

However, this is not how these files are being saved. I corrected the first iteration of the loop (ie Part E for generation 0) by editing the line to say:

for i in `seq 1 $NPOP`

do

    #Remove if plotting software doesnt need                                                                                                                                                             

    #cp data/$i.uan ${i}uan.csv                                                                                                                                                                          

    cp Antenna_Performance_Metric/0_$i.uan "$WorkingDir"/Run_Outputs/$RunName/0_${i}.uan

done

And for the looping part of XF_Loop.sh (ie part G) it has been edited to say:

for i in `seq 1 $NPOP`

        do

    # Gens data used to create a .csv file for the uan file for gain plotting                                                                                                                            

    # cp Antenna_Performance_Metric/$i.uan ${i}uan.csv                                                                                                                                                   

                cp Antenna_Performance_Metric/${gen}_${i}.uan "$WorkingDir"/Run_Outputs/$RunName/${gen}_${i}.uan

        done

4. Made Number of Neutrinos thrown (NNU) in AraSim's setup.txt file a variable: I've added the following line in part D -- as well as the looping section in part G. 

These lines replace the number of neutrinos thrown in our setup.txt AraSim file with what ever number you assigned NNT at the top of XF_Loop.sh in the "variables" section. "setup_dummy.txt" is a copy of setup.txt that has  NNU=num_nnu instead of a number. NNU is the number of neutrinos thrown (as seen in setup.txt). This "sed" line finds every instance of num_nnu in setup_dummy.txt and replaces it with $NNT (the number you assigned NNT in the variable section at the top of XF_Loop.sh). It then pipes this information into setup.txt (and overwrites the last setup.txt file allowing the number of neutrinos thrown to be as variable changed at the top of this script instead of manually changing it in setup.txt each time. Command works the following way: sed "s/oldword/newwordReplacingOldword/" path/to/filewiththisword.txt > path/to/fileWeAreOverwriting.txt. Both setup.txt and setup_dummy.txt can be found in the directory: /fs/project/PAS0654/BiconeEvolutionOSC/AraSim/

cd "$AraSimExec"

for i in `seq 1 $NPOP`

do

#This next line replaces the number of neutrinos thrown in our setup.txt AraSim file with what ever number you assigned NNT at the top of this program. setup_dummy.txt is a copy of setup.txt that has \

NNU=num_nnu (NNU is the number of neutrinos thrown. This line finds every instance of num_nnu in setup_dummy.txt and replaces it with $NNT (the number you assigned NNT above). It then pipes this infor\

mation into setup.txt (and overwrites the last setup.txt file allowing the number of neutrinos thrown to be as variable changed at the top of this script instead of manually changing it in setup.txt e\

ach time. Command works the following way: sed "s/oldword/newwordReplacingOldword/" path/to/filewiththisword.txt > path/to/fileWeAreOverwriting.txt                                                      

        sed "s/num_nnu/$NNT/" /fs/project/PAS0654/BiconeEvolutionOSC/AraSim/setup_dummy.txt > /fs/project/PAS0654/BiconeEvolutionOSC/AraSim/setup.txt

        #We will want to call a job here to do what this AraSim call is doing so it can run in parallel                                                                                                  

        cd $WorkingDir

        qsub -v num=$i AraSimCall.sh

        rm outputs/*.root

done

5. rm cannot remove 'simulation_PEC.xmacro': No such file or directory error appearing in generations after the first (ie not seen in gen 0, but seen in gen 1+).

Statistics initialized.
Tournament parents selected.
Tournament complete.
rm: cannot remove 'simulation_PEC.xmacro': No such file or directory

I found that it tries to remove it twice. See here:

########  Loop (G)  ######################################################################################                                                                                                                                                                                                                                                                                        

#     1. Does steps A-F for each generation                                                                                                                                                                                                    

###########################################################################################################                                                                                              

for gen in `seq 1 $TotalGens`

do

# used for fixed number of generations                                                                                                                                                                   

#gen=0; while [ `cat highfive.txt` -eq 0 ]; do (( gen++ ))                                                                                                                                               

# use for runs until convergence                                                                                                                                                                         

#should the line below be double indented?                                                                                                                                                               

        read -p "Starting generation ${gen}. Press any key to continue... " -n1 -s

#Part A                                                                                                                                                                                                  

        cd $XmacrosDir

        rm simulation_PEC.xmacro                                               

        cd "$WorkingDir"

        ./roulette_algorithm.exe cont $NPOP

        cp generationDNA.csv Run_Outputs/$RunName/${gen}_generationDNA.csv

    ##chmod -R 777 $WorkingDir                                                                                                                                                                           

#we can make a function that does this with the flag $gen                                                                                                                                                

#doing this in part A would mean setting $gen=0                                                                                                                                                          

#Part B                                                                                                                                                                                                  

                # First, remove the old .xmacro files                                                                                                                                                    

        #when do that, we end up making the files only readable; we should just overwrite them                                                                                                           

        #alternatively, we can just set them as rwe when the script makes them                                                                                                                           

        cd $XmacrosDir

        #I'm commenting out the next two lines. They are written to below (around lines 154-170)                                                                                                         

        #If we keep them this way then the files have restricted permissions                                                                                                                             

        rm output.xmacro

        rm simulation_PEC.xmacro

I have commented the first removal of simulation_PEC.xmacro. If there are any issues with it running, we can go ahead and comment it back in; however, I suspect this is not a necessary line to have added in. A run has not been performed since these edits. So, once it has been tested, we can delete the comments out:

#Part A                                                                                                                                                                                                  

        #I think these next two lines can be deleted. Its repeated again just below...11/30/19 comment by Julie  #cd $XmacrosDir #rm simulation_PEC.xmacro                                               

        cd "$WorkingDir"

        ./roulette_algorithm.exe cont $NPOP

        cp generationDNA.csv Run_Outputs/$RunName/${gen}_generationDNA.csv

6. Python Error from FScorePlot.py: The error can be seen in the line below.        

Congrats on getting a fitness score!

  File "FScorePlot.py", line 1

    -*- coding: utf-8 -*-

    ^

IndentationError: unexpected indent

Congrats on getting some nice plots!

I've commented out this strange line at the start of this python program:   -*- coding: utf-8 -*-. This fixed the immediate error, but threw another one when I tried to run it outside of the loop with the command 

python FScorePlot.py /fs/project/PAS0654/BiconeEvolutionOSC/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop/Run_Outputs/Julie_11_30/ /fs/project/PAS0654/BiconeEvolutionOSC/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop/Run_Outputs/Julie_11_30/ 2

The next error it threw can be seen below. 

Traceback (most recent call last):

  File "FScorePlot.py", line 79, in <module>

    fScores.append(fScorei[2:])

AttributeError: 'numpy.ndarray' object has no attribute 'append'

            This is Alex P. I edited the append() and changed it to np.append(fScores, fScorei[2:]) on line 79 in FScorePlot.py

7. rm: cannot remove 'outputs/*.root': No such file or directory: This error comes up right after the AraSim job is submitted in part D. 

1011007.pitzer-batch.ten.osc.edu
rm: cannot remove 'outputs/*.root': No such file or directory
Waiting for AraSim jobs to finish...

It's definitely coming from the lines:

for i in `seq 1 $NPOP`
do        #We will want to call a job here to do what this AraSim call is doing so it can run in parallel
        cd $WorkingDir
        qsub -v num=$i AraSimCall.sh        rm outputs/*.rootdone

We are currently in $WorkingDir at this point-- which is: WorkingDir= /fs/project/PAS0654/BiconeEvolutionOSC/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop. There is not directory called "outputs" in this location. This is why we are getting an error.Can someone explain what .root files we are trying to remove, and why? I cannot proceed unless someone makes this clear to me, as I do not know what files we are looking to remove (and where they exist/when they are created).

Immediately after that, I get the following error:

Waiting for AraSim jobs to finish...
rm: remove write-protected regular file 'AraSimFlags/1.txt'? y
rm: remove write-protected regular file 'AraSimFlags/2.txt'? y
mv: cannot stat '*.root': No such file or directory

It looks like it is coming from the following line:

cd "$WorkingDir"/Antenna_Performance_Metric
mv *.root "$WorkingDir/Run_Outputs/$RunName/RootFilesGen0/"

However-- again-- it looks like it's trying to move .root files from the directory: /fs/project/PAS0654/BiconeEvolutionOSC/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop/Antenna_Performance_Metric and put them into: /fs/project/PAS0654/BiconeEvolutionOSC/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop/Run_Outputs/$RunName/RootFilesGen0/

It can't move these, because they do not exist where it is trying to take them from. Similarly, the question in bold above would help me with this a lot. From what I can see, we are only utilizing an .txt file output from Arasim. Does Arasim output this to .txt, or are we somehow converting from a .root to a .txt? 

 This error has not yet been corrected!!!!       

Cade's comment here:

The .root files live in BiconeEvolutionOSC/AraSim/outputs/ folder. Thus, we were looking at the wrong directory. They were being removed because it is a giant file (GB size) and are created everytime AraSim runs. It just overwrites itself everytime though so it doesnt add up too much. We can have them deleted later if needed. It's our choice on whether we take it out completely, or redirect to delete the .root files in the correct directory.

8. New errors found after running with these corrections: These need to be fixed!

cp: cannot stat ‘Antenna_Performance_Metric/0_1.uan’: No such file or directory

cp: cannot stat ‘Antenna_Performance_Metric/0_2.uan’: No such file or directory

Congrats on getting a fitness score!

Traceback (most recent call last):

  File "FScorePlot.py", line 86, in <module>

    Gen = np.zeros(fScores.shape[0]*fScores.shape[1])

I found the cp error. It looks like I was incorrect in #3 about this cp command for the .uan files. I moved all of the currently existing .uan files into a new directory in

 /fs/project/PAS0654/BiconeEvolutionOSC/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop/Antenna_Performance_Metric/UAN

With all of them out of the Antenna_Performance_Metric directory, I did a run from scratch to see what it creates. Here's the outcome: So on generation 1, for individual one its going

1_1.uan
1_2.uan
1_3.uan

ie-- > individual_frequencyStep.uan

So once it hits all 60 frequencies for individual 1, it does all 60 for individual 2 by saying:
2_1.uan
2_2.uan
2_3.uan

...etc.This isn't great because it never actually cares about generation number. In section E (so generation 0) we currently have it as:

for i in `seq 1 $NPOP`
do
    #Remove if plotting software doesnt need                                                                                                                                                             
    #cp data/$i.uan ${i}uan.csv                                                                                                                                                                          
    cp Antenna_Performance_Metric/0_$i.uan "$WorkingDir"/Run_Outputs/$RunName/0_${i}.uan
done

and in section G of the looping part (gens 1+) we have:

    for i in `seq 1 $NPOP`
        do
    # Gens data used to create a .csv file for the uan file for gain plotting                                                                                                                            
    # cp Antenna_Performance_Metric/$i.uan ${i}uan.csv                                                                                                                                                   
                cp Antenna_Performance_Metric/${gen}_${i}.uan "$WorkingDir"/Run_Outputs/$RunName/${gen}_${i}.uan
      done

This doesn't seem like it's being output from XF the way we thought it is. I think the easiest way to do this is to edit the xmacros to have it output like:

generation_individual_frequency.uan

and thus

cp Antenna_Performance_Metric/${gen}_${i}_${frequency}.uan

 If so, we'd have to make a new variable in the bash script for $frequency to go from 1-60, and have it loop moving all of those. I will be looking into this with Cade tomorrow 12/3/19 (since he is the xmacro script expert). Stay tuned!

Phew....thanks for sticking with me on this long update.

This entry will serve to summarize what tasks we've completed and still need to complete after last week's GENETIS hackathon.

Alex

My focus was on the XF scripting for making the PUEO antenna. Before last week, we had a script which made the geometry of a PUEO-like antenna. The xmacro script has been modified in the following ways:

• Now includes power sources connecting the ridges that are opposite each other. 
• Generalized to use variables without hardcoding.
• Reads in data from the output files of the PUEO GA.
• Loops over all read in antennas to set up all of the simulations at once.

At this point, the script for setting up the simulations of the PUEO loop is just about ready for use. The only change to be made is to the frequency list. The script for printing out the gain data is also ready, though it only prints out the gain in theta and phi, not the cross polarization.

Here are some outstanding points that need to be hammered out, though they impede the quality and accuracy of a run rather than the actual functioning of the loop.

• The shape of the ridges in the generated antennas from the GA are considerably different from the PUEO antenna.
  • This is likely something that can be fixed by adjusting the constraints--the subspace of the parameter space that leads to PUEO-like ridges may be very small as-is.
  • You can see an example of an antenna from the GA attached (viewed from slightly below). It looks dramatically different. This seems to be due to constraints on the height and width being too small, but the ridges are also clearly misshapen. 
• The curve on the ridges looks like it needs some work.
  • For very PUEO-like antennas, the slopes look reasonable, but they do flare in or out slightly. This is more dramatic for ridges that deviate from PUEO's significantly.
  • My best guess is that the parametric form of the curves is placing the curves in a plane that is different from the rest of the ridge. This should be fixable by somehow constraining the curve to a given plane.
• The simulation still only outputs polarization data for theta and phi, not the cross polarization.
  • It looks like XF naturally gives us the theta and phi polarization. It's not clear how to get the cross polarization gain (which we think means the gain when the signal is at 45 degrees).
  • One idea is to redo the simulation but with the antenna rotated 45 degrees in the azimuth. But I'm not certain that that iwll work.
  • Amy indicated that the correct way to do this is by simulating the antenna with just one power source at a time.
    • Leaving the VPol source would produce the VV and VH data and leaving the HPol would produce the HH and HV data (HV and VH could be flipped!).
• Currently, the antennas have a fixed opening angle. We'd like to add this as a parameter, though at the moment it might be ok to forego that.  
• We need to make sure that the power sources aren't too close together (fixed!).

To do for this week:

1. Modify the script to make two simulations for each antenna, each with only one power source.

2. Adjust the constriants on the parameters to fix the curvature.

3. When we begin a run, we will want to just evolve the height and the inner side length (these determine both the inner side length and the outer side length).

Dylan and Jacob

Dylan and Jacob focused on getting PUEOsim/icemc working, which will be the simulation software needed for the PUEO loop. Here's a summary from Dylan:

• Plotting and bash scripts should be mostly good to go.
• Currently have 4/5 of the PUEO prerequisites for pueoBuilder, and the GENETIS-specific code is ready to implement and test when it's built. I’m trying to build the correct root version one more time before today’s meeting, and if it fails again I’m going to ask Will.
• The main thing left is to get the correct version of root installed, which while annoying, shouldn't take too long

To do for this week:
1. Once PUEOsim is working, run it with the usual settings to get the default performance. It will need to be submitted as a job array (ask Alex for help with this if you need; ask Will for advice on how many neutrinos to run).

Today was day two of the GENETIS hackathon. See the previous entry for details on day one. 

Alex

I continued working on the XF scripting for the PUEO horn antenna. Amy showed me an actual horn antenna from ANITA in Lisa's office so I could get the dimensions and see how the loads (power sources in XF) were offset. Offsetting the power sources led to fixing the asyemmtry problem from yesterday. I tested offsetting the loads by various heights and found that they all give the same gain patterns except for when they overlap, proving that they were interfering with each other when they crossed.

There is still, however, an asymmetry between the x and y axes (but the gain patterns are now symmetric across either of them). This is more reasonable, though I'm not positive that it should be this way. I attached an image of the gain pattern to show this asymmetry. Note that it is still at 83 MHz, which is well outside the PUEO bandwidth, but higher frequency gain patterns show the same thing (but they have more nodes so they're harder to look at). I attached images from the x-axis, y-axis, and above of the gain pattern. One way to check this will be to compare to the data stored in PUEOsim/icemc and see if they demonstrate similar asymmetries.

My next step is to test the constraints we've decided on using the script. I'll try out a few different values of the parameters and then if everything is working I'll use the GA Ryan and Dylan wrote to generate many individuals and have XF make the models for them. We don't necessarily need to see the simulation outputs from those XF runs, we just need to see if we run into error. Beyond that, we need to figure out how to extract the proper gain data from XF to create the same format that PUEOsim and icemc use. 

Amy also mentioned that we should add the opening angle of the horn antenna as a gene. This may be a little tricky but should be doable. Once we are set with the current XF script and changing the data formatting to match what PUEOsim and icemc need, I'll work on adding that as a gene.

Dylan and Jacob

We tried to get IceMC running and able to read in gain files -  we failed. Something was wrong with my make files and Jacob’s c++ compiler, so neither of us got a working version of IceMC. We then switched gears to focus on pueoSim. I created a branch on the nicemc github to output a veff.txt in the format of IceMC, so we can reuse the already-made code. Jacob and I also have been trying to install pueoSim, but we’ve been running into many errors and issues from the “C++ Gods”. For the next steps, I will finish the veffective branch and create a pull request. Then, I will work on a branch to make it so pueoSim can read in a variable path  to the gain files.

Ryan

Tuesday: Created a new branch on the PEUO repository, version0.0,  and placed the repository in the same location as the Ara one for familiarity. I also added an alias to this directory onto the genstudents channel if people would like to add the path to their .bashrc's. I then went through the the Evolutionary_Loop directory and cleaned up any residual Ara based files that were not needed to run this loop. This will make navigating and troubleshooting easier as we finish building. This branch was then commited and pushed to the github and submitted a pull request.

Today was the first day of the GENETIS Hackathon. We are spending several days this week specifically tackling the PUEO project to get it into full form (or close to it). Here is a progress update after day 1.

Alex

I am working on the xmacro script for the ANITA antenna. Before today, we had a working xmacro script to make the geometry of the ANITA horn antenna. Now I am trying to modify it to make it 1) more true to the actual ANITA antenna (in terms of which parts are electrically connected) and 2) add in the power source and sensors so that we can get simulation results. I've attached the xmacro script here along with a picture of the geometry and some gain patterns. There are four ridges in the antenna. They are all connected by the outer walls of the antenna. However, only the ones directly across from each other are connected by a load (in this case, a power source, since we simulate as a radiator). So we have two power sources, one connected each "ridge bicone". 

In making this, I have generated some gain patterns. I want to test a few cases to see if the results coming from XF seem reasonable, though I don't know what I should expect the gain pattern to look like when I've done things correctly. Here are the options I'll compare:

1. Just use one "ridge bicone".
2. Use two "ridge bicones" with the walls but just one power source.
3. Use two "ridge bicones" without the walls and two power sources.
4. Use the full form, with two "ridge bicones", walls, and two power sources.

Case (4) is the one I expect to be correct. However, when I simulate that, I get a slightly asymmetric gain pattern compared to what I expect (I expect symmetry across the "ridge bicones"). You can see from the attached image that there is a strange preference in one corner. My guess is that this has to do with which ridges are set to be ground and which are set to 1V (the potential). Feel free to ask me for more details about these figures/scripts on slack and I'll fill them in on here. I'll clean up the script and add in full test cases later this week.

The image of the antenna I attached is a bird's eye view. The green lines represent the power sources I've connected between the ridges. They do appear to overlap, but my understanding is that they are not actually affecting each other (they should just be a representation of the power sources). 

The image of a gain pattern (which is 3D and also a bird's eye view) is for the case of the antenna with both power sources and the walls remaining (the antenna is still shown too, to demonstrate the asymmetry of the gain pattern). This is well outside of the frequency range ANITA cares about (83 MHz), and I haven't looked closely at the higher frequencies yet, but I still am surprised by the asymmetry. I'll update tomorrow with more gain patterns to compare.

Dylan and Jacob

Dylan and Jacob are working on the bash scripts for the PUEO loop. They are taking the parts of the bicone loop that are specific to ARA/bicones and changing them to be specific to the PUEO loop. 

All the bash scripting outside of implementing the new software we’re developing should now be complete (we just need to debug when we do test runs). The new GA is implemented into the PUEO loop. We tried and failed to build a version of pueoSim on our own. Will has provided us with a precompiled version and instructions for installing it on our users as well as access to the necessary pueoBuilder repo. Jacob and Dylan are scheduled to go through the installation with him on Thursday at 4pm in his office, M2024.

We have also found that peuoSim currently only outputs root files. So, we will need to create a conversion script or modify pueoSim for when we implement it into the loop. Currently, using IceMC looks to be the easiest way to get a test run going. For the next steps, we have a plan for making IceMC read in a variable file path name to avoid recompiles and issues with parallelization (Alex has given us the path to look at how ARA does it). Then, we will work on installing pueoSim and testing its current viability.

Bryan

Upon checking the existing plotting scripts used for previous GENETIS work, it appears that they can be used for PUEO with minimal changes. In order to test the plotting scripts, sample output data from the PUEO loop is needed, or at least test data that mimics the form of the output for the PUEO loop. After more progress is made with installing an MC package and the PUEO loop can be run to produce test outputs, the plotting scripts can be tested and implemented in the bash script.

https://drive.google.com/drive/folders/1iDamk46R2_oOLHtvsOg4jNy05mCiB7Sn?usp=sharing

Today's Summer 2020 daily update:

As a note, today OSC was down so productivity was more limited

d

Today's Summer 2020 daily update:

Julie: Since I  won't be doing GENETIS work every day necessarily, I'll add myself down here with a blurb when I am working with the others. I will also be continuing my daily updates on Dropbox for non-GENETIS specifics. 

Today's Summer 2020 daily update:

Learn how to use the final part of the loop

Today's Summer 2020 daily update:

Here's the presentation we gave for the April APS meeting today. Atttached is the power point of the presentation and a pdf version. 

The following plots are ittereations if the test loop that add increasing improvements to the GA.

The first plot shows the GA's behavoir unaltered from our previous runs (80% roulette, 20% tournament elite selection on).

The second plots shows when we use 90% rank selection and 10% tournament, elite selection off.

Plot 3 shows when we add an offset to restrict the values of the fitness function to be more within the range of the main loop.

Plot 4 shows when we add a gaussean mutation function that is applied to crossover individuals (rate and gaussean width chosen by guess).

The following are papers I have looked at while modifing the GA (not nessisaraly recently).

https://pdfs.semanticscholar.org/5733/418cbf21dedc9e5c04351ded4a989f1ff67e.pd

https://www.sciencedirect.com/science/article/abs/pii/0165607493902157

https://www.scientific.net/AMM.340.727 

http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.28.1400&rep=rep1&type=pdf

https://arxiv.org/pdf/2010.04340.pdf

https://d1wqtxts1xzle7.cloudfront.net/30694440/10.1.1.34.9722.pdf?1361979690=&response-content-disposition=inline%3B+filename%3DUsing_genetic_algorithms_with_asexual_tr.pdf&Expires=1612908683&Signature=X93Gsc47AS0xqWf1SPLjG~7sNkoXSOXfnq1GpZ2QaPrYw9x9mWwASStW2IWexo7QBzbkhzcE5tZ~CmQA1MHN-paiNFIx2ed8VNS3IhesMnotKM0mSgUZ37BCleHT9BgGkUUum8mTJBAzCUaECn6RYjm1CZpfwVPC9zwuA~DnXBST4pGlQdna22D--sHwXgX~3U3gDUSxqk8mLI0gtn~Xued3XqsTGuMUKwJ2D9UpD5yp42-3IrH6d5CZREjEfXY2geTopQ-uNkr3eOriDj0UZqSrDw5mczmod3kQrQncgd~G2Kyda4RlIs8VDzQs~BGgszHJhSDAuKDrXr8P--9tVg__&Key-Pair-Id=APKAJLOHF5GGSLRBV4ZA

http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.438.7389&rep=rep1&type=pdf

https://arxiv.org/pdf/2102.01211.pdf

I'm making this entry so that I can record some interesting papers we find on genetic algorithms. Feel free to update this list with links to papers and maybe make a description of what was interesting/of note in the paper.
 

Bellow lies plots testing different scores and comparing them using a chi^2 score.

The functions used are as follows

Gaussian: e^(-2) (Red)

Inverse: 1/(1+(O-E)^2) (Purple)

Algebraic: 1/(1+(chi^2) )^(3/2)) (Green)

Chi: 1/(1+chi^2) (Blue)

Which are plotted here:

Here's a power point I showed today at the GENETIS meeting for the two most recent runs we did. The first run was symmetric and we ended it at 15 generations. The second one is ongoing and is shown up through generation 9 on the power point.

At the main GENETIS meeting yesterday, we collected the last few things that need to be worked out in order to get the PUEO loop in working order. Here's a list

• Skeletonize the xmacros (Alex)  04/10/23 In progress

  • This involves substituting some of the text to be variables that are replaced by the loop for each generation. 

  • Requires some slight modification of Part B1 in the loop.

• We need to modify PUEOsim to read in different gain files (Dylan) Changes made 04/10/23  Testing to be done.

  • Currently, PUEOsim reads in files like vv_0, vh_el, etc. It needs to be modified to take in the gainfiles as arguments and read different ones for each individual.

  • Requiers slight modification of the batch job scripts, but these are relatively small

• Slight modification on the fitness score read in script (Dylan)  Changes made 04/10/23  Testing to be done.

  • Now that PUEOsim is working and has been modified to output txt files with the effective volume, we just need to slightly amend the the fitness score script to read from those files.

• Fix Ezio's comments on XFintoPUEO.py and modify to read in outputs from two files (Jacob) 04/10/23 in progress, further work this week

  • Since we are running XF twice (once for each power source), we need the conversion script to read in from two files and print out to 8 files (vv_0, vh_el, etc)

We are off again with a new team. The new team consists of Alex Machtay, Alex Patton, Cade Sbrocco, Eliot Ferstl, Evelyn Shank, Mitchell Halley, Scott Janse. A spreadsheet of tasks and deadlines can be found here: https://docs.google.com/spreadsheets/d/1CkcdbediFbripdlwPw6UmhAHdAl5IdkL2efaseJkTTM/edit#gid=0

Updates:

We are now running on OSC. We have acquired a version of XF on OSC and have now transferred the loop to OSC. All of the xmacros for XF should be functioning, and we have managed to make the bash script auto import the xmacro scripts to XF when it boots. The loop exists on /fs/projects/PAS0654/BiconeEvolutionOCS. This is a shared copy so that everyone can run off of the same installation. This is important so that we can make sure everyone is pushing edits regularly, and so that everyone is always running with the most up-to-date edits. 

The loop had not yet been tested for it's current status. Alex M. is testing it and we are waiting to hear back. 

Tasks:

• Check continuity of loop
  • Update: Alex M is testing this
• Install Arasim to /fs/projects/PAS0654/BiconeEvolutionOCS to be edited and messed with
  • Update: this has been completed as of 11/5/19
• Test Alex M's code for editing Arasim's Setup.txt variables
• Clean bash script
  • Update: Mitchell is working on this.
• Make version of bash script which writes .job scripts for Arasim
  • This must write a different .job for each individual so it parallelizes Arasim running for each individual
  • The number of .jobs written must depend on $NPOP
    • ie this bash script should be writing .jobs from scratch each time as (1) the number of them we need depends on $NPOP, (2) the names of the input and output files depends on $NPOP. 
      • It should also delete .jobs at the end -- ie when all generations have completed so that we don't have a random number of .jobs floating around. For example, say we run with 10 individuals one time, and it writes 10 .jobs ($NPOP=10), and then the next run we only have $NPOP=5, we will have 5 extra .jobs floating around not being used unless we clean them all at the end line of the run in the bash script. 
  • Update: Work on this has begun by Alex M, Cade, and Evelyn (Alex P has also been messing with Arasim).
• Make XF be submitted as a job
  • Cade has ideas for this.

Students have been posting individual updates here: https://www.dropbox.com/home/GP_Antennas/Updates

Paperclip Antenna (Ryan and Evelyn working on it):

• https://github.com/gp-antennas/Sph._Harmonics_And_Paperclip

Asymmetric Bicone (Leo and Eliot working on it):

• https://github.com/JulieRolla/GENETISBicone

AREA (Ben working on it):

• Github? We need to ask Steph where their past work exists.
• See AREA thesis attached

Symmetric Bicone(Alex M. and Alex P. working on it):

• https://github.com/JulieRolla/GENETISBicone

Attached is a plot containing bar graphs with error bars representing the average number of generations it took for the GA to achieve a chi-squared value of 0.25 (roughly equated to a 0.8 out of a max 1.0 fitness score). Unlike the fitness scores used by the GA, these values do not have simulated error attached to them and are therefore a better measure of how well the GA is optimizing. These results were obtained by running 10 tests in the test loop for each design, population, and error combination and solving for the average number of generations to meet our threshold and the standard deviation of those scores. From this, a few things immediately pop out, I will address the more obvious one later. But essential for this test, we can see that increasing our population size seems to have a more direct impact on the number of generations needed to reach our threshold than decreased error does in both designs. My best guess regarding this is that the GA depends on diversity in its population in order to produce efficient growth, and an increase in the number of individuals contributes to this, allowing the GA to explore more options.

This leads me to the more easily spotted trend; PUEO is much slower than ARA. This presents an anomaly as this is the opposite of what we would expect from this test loop as PUEO has fewer genes (7) to optimize than ARA (8). It is also important to note that no genes are being held constant in this test for either design, so both designs have the full range of designs provided they are within the constraints. With that in mind, my guess as to what causes this phenomenon is that PUEO's constraints are much stronger than ARA's. How this may affect the growth is that it more heavily bounds the possible solutions, which makes it harder for the GA to iterate on designs. It is possible that during a function like a crossover, the only possible combinations for a pair of children are identical to their parents, effectively performing reproduction. This could limit the genetic diversity in a population and therefore cause an increase of generations needed to reach an answer. We could in theory test this by relaxing the constraints done by PUEO and then running the test again to see how it compares.

Finally, You will notice that no AREA designs are shown. This is because, under current conditions, they never reached the threshold within 50 generations. However, Bryan and I think we know what is happening there. AREA has 28 genes, about four times the amount of genes that our other designs use. Given that our current test loop fitness measure is dependent on a chi-squared value given by: SUM | ((observed(gene) - expected(gene))^2) / expected(gene) |, we can see that given more genes, the harder it gets to approach zero. For example, we can imagine in an ARA design if each index of the sum equals 0.1, you would get a total value of 0.8, while AREA would get a value of 2.8, which seems considerably worse, despite each index being the same. Upon further thinking about it, Bryan and I do not think that a chi-squared is the best measure of fit we could be using in this context. Another thing we thought about is that we have negative expected values in some cases. We have skirted around this by using absolute values, but upon reflection believe this to be an indicator of a poor choice in metric. Chi-Squared calculations seem to be a better fit for positive, independent, and normally distributed values, rather than our discrete values provided by our GA. With this in mind, we propose changing to a Normalized Euclidean Distance metric to calculate our fitness scores moving forward. This is given by the calculation: d = sqrt((1/max_genes) * SUM (observed(gene) - expected(gene))^2). This accomplishes a few things. First, it keeps the same 0 -> infinity bounds that our current measure has, allowing our 1/(1+d) fitness score to be bounded between 0 and 1. Second, it forces all indices to be positive so we don't need to worry about negative values in the calculation. Third, this function is weighted by the number of genes present for any given design, making them easier to compare than our current measure. Finally, as our GA is technically performing a search in an N-dimensional space for a location that provides a maximized fitness score, it makes sense to provide it the distance a solution is from that location as a measure of fit in our test loop. We created a branch on the test loop repository to test this and the results are promising as results from the three designs are much more comparable for the most part (though we still see some slowdown we think is contributed to constraints as mentioned above). Though some further input would be appreciated before we begin doing tests like the one we have done in the plot below.