We ran valgrind (a profiler) on AraSim to find out where it was spending most of its time, we ran it with callgrind to give an output file of the time and opened that file with qcashegrind to display it in a legible format. This let us know that the function Param_RE_Tterm_approx was taking up about 75% of the time because it was being called hundreds of millions of times. Now more specifically the standard math functions pow() and exp() were being called in this function every time and took up about 44% and 20% respectively. They were called about the same number of times as Param_RE_Tterm_approx but they are only called in the else block in an if-else statement.

JULY 31st Update:

I used the C++ library future and set up iterations of a loop to run asynchronously as they do not rely on each other's calculations. This launches a seperate thread whenever I call one. The struggle with this is that launching a thread and getting its results takes time as well, so there had to be enough calulations in a thread in order for it to make up any time. The purpose of the function GetVm_FarField_Tarray is to create arrays of doubles called Earray and Tarray and those are the only values that get altered in this function. Tarray is created with simple calculations so I left that out of threads but making Earray calls the other function that takes up so much time.

I edited the files signal.cc and signal.hh to work with 4 threads and ran 10,000 neutrinos with 4 cores. The cpu time was 1:48:41 but the real walltime it used only took 00:54:40. The time these tests for 10,000 neutrinos with base AraSim normally take anywhere from 1:35:00 to 1:45:00 or in that range. It should still work with less threads available it would just run slower and probably at the cpu time, which might be slightly slower than base AraSim but as soon as more threads are available it runs much faster as we see here. I still need to check for the accuracy of these edits, the number passed is in a reasonable range, but I will need to investigate further by giving the exact same neutrinos to both programs and making sure their outputs are identical.

I also have some further ideas to speed it up even more. Currently every iteration of the loop calls a thread but what would be faster would be to either

A: Set it up so it can call more threads at a time and thus go faster, which would be an easy edit but only work if more cores are given

B: Set it up so the current threads run more than one iteration at a time, I've started experimenting with this but having trouble making sure that everything involving pointers stays safe, but the benefit of this is that the threads would get called and joined significantly less and would cause a great speed increase without adding more threads.

All these developments are new after me spending a while trying many different ways to speed it up or run parts in parallel like this, so even if this isn't accurate or valid I believe I would be able to make it so with more time and work.

Attached I have a zip that contains the two files I edited as well as an output file from my first test. I will continue to test and improve this but am unable to work on this again until August 19th so I wanted to share my current status before I am unavailable.

Because of issues we were having with XF when running at the command line, we've shifted to running through a Pitzer VDI (see here: https://ondemand.osc.edu/pun/sys/dashboard/batch_connect/sys/bc_desktop/vdi/session_contexts/new).

The VDI will give you a mirror of a desktop running on OSC's side, so things like XF will run much faster. You can open a terminal by clicking the black box at the bottom of the screen and then run the loop like normal (./Asym_XF_Loop.sh). This has the added benefit of making it possible to exit out of your browser, close your laptop, or turn off your computer without interrupting the run, since the process is running on OSC's side rather than piping to your computer.

For PAEA Algorithm:

Part I: Complete as soon as the run starts

Run details: Please answer all of the questions below!

• Run Type
  • Answer here whether or not it's AREA (Gain pattern evolution) or PAEA (Bicone evolution)
• Run Date
  • Add answer here
• Run Name
  • Add answer here
• Parameters evolved
  • Add answer here
• Why are we doing this run?
  • Add answer here about what we are testing
• What is different about this run from the last?
  • Add answer here with info on what we are doing differently from the last run
  • Example: We are testing a new operator, we are changing the percentages of each selection method used, etc.
• Symmetric, asymmetric, linear, nonlinear (what order):
  • Add answer here (say N/A if this is an AREA run)
• Number of individuals (NPOP):
  • Add answer here
• Number of neutrinos thrown in AraSim (NNT):
  • Add answer here
• Operatiors used (% of each):
  • Add answer here
• Selection methods used (% of each):
  • Add answer here
• Are we using the database?
  • Add answer here. This can be found in the main bash script within the variables section. (Answer N/A for AREA run)

Please upload the text file with all run details before closing this!

Part II: Complete as soon as the run ends

Results: Once this run completes, please upload the plot(s) to this post as an attachment, as well as a general explanation of the results. 

• Summary and comments on results
  • Add thoughts on what we should do next here!
  • Example: "It seems like we are seeing minimal evolution. We should take a step back and try to see why we don't see improvement. Once we trouble shoot, we can start a new run to investigate."
  • Example 2: "Run looks good! Our best individual is in generation #12. Attached are the CAD drawings of that individual."

**Upload all plots, run parameter text files (file that has run settings saved), CAD models of best indivduals, etc**

For AREA Algorithm:

Part I: Complete as soon as the run starts

Run details: Please answer all of the questions below!

• Run Type
  • Answer here whether or not it's AREA (Gain pattern evolution) or PAEA (Bicone evolution)
• Run Date
  • Add answer here
• Run Name
  • Add answer here
• Parameters evolved
  • Add answer here
• Why are we doing this run?
  • Add answer here about what we are testing
• What is different about this run from the last?
  • Add answer here with info on what we are doing differently from the last run
  • Example: We are testing a new operator, we are changing the percentages of each selection method used, etc.
• Single frequency run or run with broadband frequency dependence:
  • Add answer here
• Number of individuals (NPOP):
  • Add answer here
• Number of neutrinos thrown in AraSim (NNT):
  • Add answer here
• Operatiors used (% of each):
  • Add answer here
• Selection methods used (% of each):
  • Add answer here
• Any other things to note?
  • Add answer here

Please upload the text file with all run details before closing this!

Part II: Complete as soon as the run ends

Results: Once this run completes, please upload the plot(s) to this post as an attachment, as well as a general explanation of the results. 

• Summary and comments on results
  • Add thoughts on what we should do next here!
  • Example: "It seems like we are seeing minimal evolution. We should take a step back and try to see why we don't see improvement. Once we trouble shoot, we can start a new run to investigate."
  • Example 2: "Run looks good! Our best individual is in generation #12. Attached are the CAD drawings of that individual."

**Upload all plots, run parameter text files (file that has run settings saved), gain patterns of the two best and two worst individuals, etc**

This run was started on 10/23/2020. The purpose was to attempt to demonstrate evolution by beginning from 50 identical individuals in the initial generation (which had previously produced a low score).

This run was the first time we removed the penalty on fitness scores for antennas which exceeded the borehole radius. It was also the first time we increased the number of neutrinos thrown up to 300,000. 50 fully asymmetric individuals were evolved over 25 generations.

There was a 50/50 split for roulette/tournament selection and 75%/10%/15% for crossover/reproduction/immigration. While the evolution was somewhat flat, we do believe we demonstrated evolution because the average score rose as the run evolved. 

(Note that the final generation was interrupted but the plot was still made, hence the drop to 0 for all scores on the plot)

This was a symmetric run that began on 9/11/2020. This run used fewer neutrinos to determine how significant the number would be on the performance of the evolution. 30,000 neutrinos were used per individual.

10 individuals were evolved over 35 generations, using 100% roulette selection and the old mutation algorithm, where genes had a 40% chance of mutating. Mutations changed the gene by reselecting the value from a guassian with a mean and sigma equalt to the mean and sigma of that gene's value in the generation.

This run was conducted beginning on August 31, 2020.

Both the angle and length were asymmetric for the two cones. It was run with 10 individuals over 42 generations using 150,000 neutrinos. A penalty was still implemented on bicones exceeding the borehole width.

As this was before the substantial rewrite of the genetic algorithm, 100% of selection was done using roulette. Genes had a 40% chance to "mutate," which was done by selecting a new value from a guassian, with a mean and sigma equal to the mean and standard deviation of that gene's value in that generation.

This run followed the first symmetric run. It had an asymmetric length, but the opening angle and inner radius were kept symmetric. 

10 individuals were evolved over 17 generations using 100,000 neutrinos. All selection was done through roulette, and all individuals were formed through crossover and the old mutation method (where individual genes had a 40% to be mutated, and mutated genes were chosen from a gaussian based on the mean and standard deviation of that gene's value in the generation).

This is considered to be the first run of "good" data. Prior to this run, we ran with few neutrinos and had errors which needed to be resolved in AraSim and with the way we handled XFdtd's simulation results being passed to AraSim. Those were resolved in the summer before this run.

This was a symmetric run of 10 individuals over 15 generations using 100,000 neutrinos per individual. This preceded the substantial modifications made by Ryan to the GA. All individuals were formed through crossover, but there was a probability of 40% for genes to be mutated. In this case, mutation was a complete change to the gene, though it was based on the average value of that gene in the generation. There was no reproduction or mutation. All individuals were selected through roulette.

Green individuals indicate that there was no penalty applied, while red individuals had a penalty factor multiplied by their effective volumes. The penalty was of the form exp(-(R-7.5)^2), where R is the outer radius, in cm, of the bicone.

This is the run that is discussed in the GENETIS paper submitted to Physical Review D in December of 2021. It was conducted beginning on March 19, 2021. The preprint can be found here. It ran for 31 generations with 50 individuals per generation. Each individual was run through AraSim for 300,000 neutrinos.

The ratio of generative operators was: 72% crossover, 22% immigration, 6% reproduction. 

The ratio of selection operators was: 80% roulette, 20% tournament, 0% rank.

The highest scoring individual had a fitness score of 5.24. However, when it was rerun with many more neutrinos (3*10^7), it had a score of 4.90, an 11% improvement over the ARA bicone.

This run was conducted before the introduction of the automatic run_details.txt generator. Instead, there is a file called Run_Notes.txt (attached) which details some of the errors that were encountered. These errors were remedied during runs by removing the offending AraOut file (which had too high of a fitness score due to a weight being calculated to be greater than 1, which shouldn't be possible) and have since been prevented by modifying the data file AraSim uses to calculate the weights.

We began a new run of the curved sided GA. Previously, we conducted a run just like this in July of 2022 (titled 2022_07_15_Latest_Greatest). In that run, we made substantial updates to the loop, including using the latest version of AraSim. However, we discovered after the run (which had a very flat fitness score growth curve) that we had not modified AraSim to take in gainfiles appropriately (so it was just evaluating the ARA bicone repeatedly). We've fixed this issue and a few more (see the github repository for the issues that exist and have been resolved) and are redoing this run. The run details will look much the same as that one and are visible in the attached file. Here is the location of the directory where the data for this run will be stored: /fs/ess/PAS1960/BiconeEvolutionOSC/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop/Run_Outputs/2022_12_29

For a list of changes made in this run compared to previous ones, see this ELOG post on the previous run: http://radiorm.physics.ohio-state.edu/elog/GENETIS/186

We have started a new run. Here is the directory: /fs/ess/PAS1960/BiconeEvolutionOSC/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop/Run_Outputs/2022_07_15_Latest_Greatest

This run is a substantial overhaul. It is a curved run using the latest parameters from Ryan's optimization loop. Here is a list of important changes:

1. The newest version of AraSim has been implemented.
2. The minimum length has been decreased to 10 cm per side.
3. The GA is recording the parents and operators used to generate individuals.
4. The number of individuals has been increased to 100 per generation.
5. The number of neutrinos per individual has been increased to 600k.

This run may take longer than previous runs due to the increased number of individuals. There may need to be modifications made to resolve or work around AraSim errors that delay the loop (due to the auto-resubmit function). Find the details of the run attached.

This run was a short run (stopped due to resource limits) that was conducted to test the minimum length constraint that had been applied in the past. Previously, the minimum length was set to be 37.5 cm for each cone; in this run, it was lowered to 10 cm. It ran for 60 generations with 50 individuals per generation, each evaluated with 300,000 neutrinos. To compare to the paper run, this run was performed using the asymmetric algorithm, meaning each individual had 6 genes (length, inner radius, and opening angle for each cone). The initial generation began with identical individuals, with genes matching the genes of the best performing individual mentioned in the GENETIS paper.

The ratio of generative operators is as follows: 72% crossover, 22% immigration, 6% reproduction. Mutation was also used, with 1% of genes produced by crossover mutated by adding a number chosen from a Gaussian centered at 0 with a width of 5% of the gene's value.

The ratio of selection operators was: 20% roulette, 80% tournament, 0% rank, 0% elite.

Attached are the run details, violin plot, and "rainbow" plot.

This run was started on February 2, 2022. It is a full run of 50 generations with 50 individuals per generation using the quadratic version of the loop. This means that each individual is defined by 8 genes (length, inner radius, linear coefficient, and quadratic coefficient for each cone). Attached is the txt file run_details.txt that is automatically generated when the loop is run. Each individual was run for 300,000 neutrinos.

This run used the usual ratio of generative operators: 72% crossover, 22% immigration, and 6% reproduction. It also used the proper mutation function: 1% of genes created through crossover were mutated by adding a number chosen from a Gaussian distribution centered at 0 with a width of 5% of the gene's value (MUTATION WAS NOT USED BUT WAS AVAILABLE).

This run was the first full run in which the rank selection operator was used. The ratio of selection operators was: 0% roulette, 10% tournament, 90% rank, 0% elite. 

This run used the script fitness_check.py to average the scores of identical individuals that appeared across multiple generations. This gives us a more accurate measure of those individuals' scores. It may also explain why this run appears so flat (on the violin plot): fluctuations in the scores should die down as repeated individuals have more accurate scores, and newly generated individuals that perform highly regress to their mean (actual) score when they are reproduced/recreated through crossover. This has led us to question our GA parameters, as Audrey and Autumn demonstrated that roughly 1/4 individuals in the run are repeat individuals.

Here are the changes that have been made in this (and the preceding) run relative to what we have been doing before.

1. The newest version of AraSim has been implemented.
   1. This was done because AraSim has gone through significant changes since 2019 that make it more accurate and makes it run faster.
2. The minimum length has been decreased to 10 cm per side.
   1. This was done because we previously tested shorter lengths for a response to the reviewer when we submitting the paper to Phys Rev D. Previously, we thought that going below 37.5 cm would lead to unreliable results, but we are reassured that that is not the case now.
3. The GA is recording the parents and operators used to generate individuals.
   1. This should allow us to look back and make a "history" of the evolution by seeing where genes come from in each generation.
4. The number of individuals has been increased to 100 per generation.
   1. Ryan demonstrated that a higher number of individuals does lead to a substantial improvement in the speed at which the evolution converges. 
      1. We previously thought that it did not make a big difference. However, that was done by measuring the average maximum fitness score. If we measure how many generations it takes for the evolution to reach a benchmark score, we find that more individuals helps significantly.
5. The number of neutrinos per individual has been increased to 600k.
   1. This was done to make the fitness scores more precise. At 300k neutrinos, we expect an error of around 0.2. This will cut things down closer to 0.1, which may help us decrease the number of outlier measurements without increasing run time too much because of the AraSim speed up.

This run may take longer than previous runs due to the increased number of individuals. There may need to be modifications made to resolve or work around AraSim errors that delay the loop (due to the auto-resubmit function). Find the details of the run attached.

The following is a run summary of an AREA test using 100 individuals per generation, without the linear frequency dependence on the gain, with the frequency fixed (i.e. the same gain pattern is produced for all frequencies). This run was a preliminary test of the AREA algorithm with a larger NPop (100 individuals per generation) after seeing promising results in the previous test that only used 20 individuals per generation. The percentages of selection methods/operators used attempted to mimic the optimal percentages used for the PAEA Bicone Loop, but because the algorithms do not set these in the same manner, these are highly unlikely to be the best percentages to use. The results show a very quick plateau in fitness score with a loss in diversity of solutions after a low number of generations, potentially meaning that a poor breakdown of selection methods/operators was chosen. This result seems to underscore the need for a test loop-style optimization for AREA to determine the best breakdown of selection methods and operators to use for AREA.

Run details:

• Run Type
  • AREA
• Run Date
  • January 18, 2023
• Run Name
  • 20230118breynoldsrun_noLinearDependAREATest_38RCO_2RM_58TCO_2TM_150000NNU_2Seeds
• Why are we doing this run?
  • This test increased the number of individuals to 100 and attempted to mimic the percentages of selection methods and operators found to be optimal for the PAEA loop.
• What is different about this run from the last?
  • The previous test showed slow/minimal evolution after ~12 generations, and it was speculated that this was due to a small NPop of 20 individuals. This test uses an increased NPop of 100 individuals.
• Symmetric, asymmetric, linear, nonlinear (what order):
  • N/A
• Number of individuals (NPOP):
  • 100
• Number of neutrinos thrown in AraSim (NNT):
  • 300,000 Total (150,000 NNU x 2 Seeds)
• Operatiors used (% of each):
  • 96% Cross-Over, 4% Mutation
• Selection methods used (% of each):
  • 40% Roullette, 60% Tournament
• Are we using the database?
  • N/A

Results:

• Summary and comments on results
  • The fitness score quickly plaeaued and diversity of solutions was lost, potentially due to a unsuccessful choice of selection method and opterator percentages.
  • An important future step is to work with Ryan to create a test loop optimization for the AREA algorithm, as it is written differently in how it is passed selection method and operator percentages and requires its own optimization to find the optimal ratios of these. (Work on creating a test loop optimization for AREA is currently underway).

EDIT 4/10/23
Re-uploading gain pattern associated with most fit individual from this run

The following is a run summary of the AREA test without the linear frequency dependence on the gain, with the frequency fixed (i.e. the same gain pattern is produced for all frequencies). This run was a preliminary test of the AREA algorithm after fixing long-standing errors in the algorithm and finding that the initial constaints implemented did not work with the frequency dependence on gain. Results from this run did show a "proof-of-concept" that the AREA algorithm seems to evolve single frequencies in a promising way, however a low number of individuals may have stalled the evolution.

Run details:

• Run Type
  • AREA
• Run Date
  • December 26, 2022
• Run Name
  • 20221226breynoldsrun_noLinearDependAREATest_8RCO_8RM_2TCO_2TM_30000NNU_10Seeds
• Why are we doing this run?
  • We tested the AREA algorithm set to evolve at a single frequency in order to see if it would evolve a gain pattern at that frequency within the required constraints.
• What is different about this run from the last?
  • This run only evolves gain patterns at a single frequency, allowing us to investigate the evolved gain patterns and check that the required constraints on the power conservation are met, without yet fixing the constraints function to work with the linear frequency dependence relationship. This is also the longest test run since the AREA algorithm was fixed and gotten into working order.
• Symmetric, asymmetric, linear, nonlinear (what order):
  • N/A
• Number of individuals (NPOP):
  • 20
• Number of neutrinos thrown in AraSim (NNT):
  • 300,000 Total (30,000 NNU x 10 Seeds)
• Operatiors used (% of each):
  • 50% Cross-Over, 50% Mutation
• Selection methods used (% of each):
  • 80% Roullette, 20% Tournament
• Are we using the database?
  • N/A

Results:

• Summary and comments on results
  • The evolution appeared to be successful in that the most fit gain pattern found (attached) seems to make sense with our understanding of the underlying physics. Additionally, after inspecting the outputs, conservation of power was confirmed to be properly constrained. However, in the attached violin plot, the fitness score plateaued quickly and never evolved further, with the best individual coming in generation 12. This was speculated to be due to the small population used in this test (only 20 total individuals per generation).

The reviewer asked us to run the loop again but for just symmetric antenna designs. We have completed that run and have results ready. We still need to resimulate the top five individuals, which I'll update this post with later. I attached the violin plot (though it should be fixed to not let the legend overlap the data). See this previous entry for the details of the run: http://radiorm.physics.ohio-state.edu/elog/GENETIS/198

Requesting an interactive job:

qsub -I -X -l walltime=1:00:00 -l nodes=1:ppn=40

(ppn=24 makes it quicker FYI)

Note that walltime is the length of time you'll need it. You will likely need it for 4+ hours if you are running the loop... Probably longer. Note that before you enter this command into your command line, you must be logged into OSC.

For submitting a job with GPUs (which we will be doing to speed up XF now as of 1/24/20), use this command:

qsub -I -X -l walltime=1:00:00 -l nodes=1:ppn=40:gpus=1:default      OUTDATED

srun -A PAS0654 -t 1:00:00 -N 1 -n 40      CURRENT

I fixed a bug in the loop, so we started another rank test run. Run details in the attached file.

The bug was searching for the generationDNA.csv file in the wrong place, meaning that it wasn't able to copy it to the run directory. That meant we didn't have a record of the generation data in the usual format. I don't think that this explains the flatness, since the the generationDNA.csv file was still created every generation correctly, so the GA knew where it was. But this test now corrects that problem and tests the usage of the rank selection method.

The Physics of Results Plots have been added to the Pueo Loop. The current version of the plotter is built for pueoSim v1.0 and located in ${WorkingDir}/Antenna_Performance_Metric (Hasn't been pulled into the loop directory yet).

The pueoSim v1.0 IceFinal files were missing information on the RF direction and information needed to see an amplitude spectrum. I asked Will, and he said that the new version of pueoSim (v1.1.0) outputs the needed information.

I created an updated version of the plotter, and have a pull request here: https://github.com/osu-particle-astrophysics/GENETIS_PUEO/pull/40

However, before this can be implemented in the loop, we need a way to get errors from pueoSim v1.1.0.

Currently, there is a version of rootAnalysis.py here that can analyze the new root files and output fitness scores and errors, but instead of taking 20 minutes for a generation, it now takes 20 minutes per individual to run.

This is because the update splits the IceFinal file into IceFinal_allTree, IceFinal_passTree0, and IceFinal_passTree1. IceFinal_allTree and IceFinal_passTree1 are of comparable size to the previous IceFinal file, but the passTree0 final is

about 10-20 times larger, causing a slowdown in calculations. If we calculate without this file, it takes about 1 minute per individual, still a bit slower than before.

Ideas to solve this issue:

Don't use the passTree0 files (I have asked Will what they're for, and if we need them. Hopefully he responds after the Holiday.)

Instead of running through a Python for loop, call a C++ script that creates a CSV file the Python program can easily load in.

Submit a batch job to run the analysis in parallel before combining the outputs in the correct format. (We should maybe do this anyways?)

Continue using pueoSim v1.0 (I think we can still retain the Theta graphs for the incoming neutrinos, but there won't be any RF graphs or the possibility of amplitude spectrum plots) 

Attached is a preview of plots the v1.1 plotter is capable of making.

Add weights.