We are almost to where we can start the physical building of the antenna! 

I've attached all the information I currently have regarding the building project. Some of it is messy work notes and some is well-structured.

I’ve attached the following files for the GENETIS building project:

• Building Dump.txt 
  • My working notes that I used while trying to simulate the antenna in XFdtd (very messy) 
• Building Dump of Useful Materials.txt
  • List of materials that I found regarding the building project like slides, elogs, etc.
• Simulating Building Model.txt
  • A writeup I made describing my process for simulating the antenna in XFdtd 
• Done with change materials.zip
  • Solidworks model of antenna

I also made a slide deck that contains the directory locations + has graphs HERE.

Here are some scholarships available to physics majors I've been lucky enough to receive throughout undergrad that allowed me to work on this project unpaid without needing supplemental income from a separate job or loans. 

1. OSU Arts and Sciences Merit Scholarship Pooled Application 

• 500 word personal statement
• 1 letter of recommendation from a faculty member (Amy wrote mine)
• Varying amounts awarded (I got the $3,300 David and Velva Zarley Scholarship)
• Can be used for any education related expensed (tution, rent, food, ...)

2. James L. Smith Scholarship for Physics Majors

• 3 short essays (~200 words)
• Probably varying amounts (I got $4,000)
• Can be used for any education related expensed (tution, rent, food, ...)

3. Physics Class Awards 

• No application, chosen by department (just get good grades, honors might help. I didn't really talk to any of my professors or go to office hours, so milage might vary there)
• Freshman: $50, Sophomore: $250, Junior: $500, Senior: $?
• Can be used for any education related expensed (tution, rent, food, ...)

4. OSU Scholarship Universe

• A couple ~500 word essay
• Very few awarded, I've applied twice: $1,000 the first time and completely ghosted the second
• Can be used for any education related expensed (tution, rent, food, ...)

5. Licking County Foundation (If anyone is from licking county, I HIGHLY RECOMMEND APPLYNG TO THIS. I think there are similar ones for other counties) 

• ~500 word essay
• Varying awards (I've applied 3 times and gotten $2,375, $5,000, and $7,500) 
• Can be used for any education related expensed (tution, rent, food, ...)

Here is a link to a google drive with all my winning scholarship essays:

https://drive.google.com/drive/folders/1qJKpBf4by9wlReU5l_XjxjVsDIfXR4Jc

Copy this slide deck to use when presenting on run updates!

https://docs.google.com/presentation/d/1Tk-B1QbFTP_5pQZovfn0_wbTswZTmOrJURpi374xJWo/edit?usp=sharing

# AraSim CSE Spring 2024 Work

## Goals 
The main goal was to get a working multithreaded version of the AraSim codebase working. Doing this, the hope was to learn how to multithread the code and get it in a good place to hopefully also integrate GPU's at a later date.

## Where is it at currently? 
A bulk of the work done was to functionalize the Connect_Interaction_Detector_V2 to allow for multithreading and to cleanup codebase. 

### Going through code added/changed
Helper functions for multiple parts of code. They went through and split it up into multiple parts. Putting line numbers when needed

Part 1: Clearning Antenna Data
Part 2: Determine gain channel
Part 3: solve ray tracing (Not Done)
Part 4: Process Ray Tracing Solution (Not Done)
Part 5: Calculate Signal Factors (Not Done)
Part 6: Calculate Antenna Gain Factors (Not Done)
Part 7: Process Frequency Domain Signal
Part 8A: Process Neutrino Events. Lines 908 - 1208 (Not Done)
Part 8B: Process Arbitrary Events. Lines 1209 - 1475 
Part 8C: Process Simple Pulser Simulation. Lines 1478 - 1747 (Not Done)
Part 8D: Process PVA Pulser Simulation. Lines 1751 - 2110 (Not Done)
Part 8E: Process Calpulser Event. Lines 2113 - 2445
Part 9A: Process Noise. Lines 2593 - 2955
Part 9B: Process Trigger and Mimic Waveforms. Lines 2994 - 3624

## What still needs to be done? 
- Multithreading still isn't working, multiple threads are writing data to the same place causing the program to crash. This need to be resolved to at least have a working prototype. 

- I believe for multithreading we need to mark explicitly where file/data writing is happening to be able to adjust to make thread-safe

- Double checking that new functions are passing variables in the correct way. The CSE students had this has a slight fear. 

- Some helper functions are still not completed (3-6, 8A, 8C, 8D) 

- Current completed parts of code are all in separate branches and need to be merged after double checking that variable passing is correct

We are beginning a new run with several improvements to the ARA VPol loop to try to evolve antennas that are optimally matched just by their geometry. To do this, we are evolving with the RealizedGain instead of just Gain in the Xmacros (thus taking into account impedance mismatch). We also have the speed up that parallelizes the AraSim jobs on each run. Attached is the run_details.txt file, but the GA parameters are subject to change. Here is what they are to begin:

We have a population size of 50 individuals per generation.

Selection operators (NUMBER selected by each):

• Roulette: 10
• Tournamente: 10
• Rank: 30

Genetic Operators (NUMBER generated by each):

• Reproduction: 6
• Crossover: 36
• Immigration: 8
• Mutation: 2
  • Sigma: 5%

Attached is the license agreement that each person should sign to be able to use XF on OSC.  You can sign it, send it to Amy, and she will return it to OSC with her signature on it.

Slides shown at experts building meeting Aug. 2nd, 2023.

Previously, we have tried to straighten the sides of the best-evolved curved antenna in elog 229. However, there were potential issues with how closely this line resembled the curve of the antenna.

So, I attempted to create another straightened sides antenna using a linear regression model to find the best fitting line for the curve to create an antenna.

I made a Python notebook to separate the equations for each curve into 1000 discrete points. Then I ran a linear regression model to fit a curve of the points 1 - n, and a second curve of the n - 1000 points, looping from n = 2 to n = 999.

These results are from the combined output with the least squared error compared to the original.

Pictured is a plot showing the two sides of the curved bicone in red and blue, with the best fitting lines for each in black as well as a model in XF.

Results:

The antenna has a fitness score of 3.80627 with an error of 0.0759725.

This is much lower than the 5.71 of the curved antenna and 5.11 of the other attempt at straightening.

For the next attempt, we could consider constraining the endpoints to be the same as the original antenna to conserve the radius, and/or adding an extra line to fit the curve.

I've also attached a picture of what my notebook found for fitting 3 lines to the curve (not modeled or tested).

Professor Chen recommended using 3 sides and constraining the outer radii of the cones to match the original curved design.

Path to linear regression notebook: /users/PAS1960/dylanwells1629/developing/notebook.ipynb

Path to XF project: /users/PAS1960/dylanwells1629/straightened.xf

I've been building in XF and have ran into some issues but currently have the following completed:

- Struts are placed, though they are not evenly spaced and the same. I have contacted XF to try and figure out how to get rotation/patterns to do what we need

- Circuit components and circuit board are in the xf file, though the components are not on the circuit board (I'm not totally sure how to put them on, that's the next step). The circuit components should be of spec, I looked up the part numbers from the previous ELOG post and grabbed the values from the manufacturer.

Here are the things that have not been completed/need answered:

- How to evenly space the struts connecting the shape (as stated previously, I've contacted XF to figure out how to do this easily. Im just bad at modeling in XF)

- Add circuit components to circuit board

- Add voltage and ground connections to the circuit board

- On the circuit diagram, figure out J1 and J2 connections. Meaning, which one connects to the signal from the cable and which connects to the other cone that is used as ground?

To see current antenna build go to: " /users/PAS1977/jacobweiler/GENETIS/XFzone/straightened_antenna_for_Jack.xf "

Attached are pictures in case there are issues pulling up model in XF. :)

Standard Loop Architecture:

Complete an evolutionary step FOR EACH antenna before continuing on with the next step.

Steps:

1. Run the Genetic Algorithm for the entire population.

2. Run the XF radio simulation for the entire population.

3. Run the neutrino simulation software for the entire population.

4. Run root analysis and plots for the entire population.

However, due to constraints on the amount XF keys we have, we can only run 4-5 XF simulations at a time. 

So, when the first antennas finish their XF simulations, their outputs will simply sit there until EVERY other antenna finished their XF simulations.

New Proposed Architecture:

Complete necessary evolutionary steps as a population, but string together those that don't rely on data from other antennas.

Steps:

1. Run the Genetic Algorithm for the entire population

2. Run the XF radio simulation for each antenna

      - When an XF simulation finishes for an antenna, submit the pueoSim / root analysis jobs for that antenna

3. Run the plots for the entire population

This will allow us to complete most of our pueoSim computation while the XF portion of the loop is still running, cutting down the time between the final XF job finishing and the final pueoSim job finishing.

Test run of this new architecture with 100 antennas, 7,840,000 neutrinos per antenna.

After the final XF job finished, the pueoSim simulations and analysis of outputted root files were completed in 954 seconds,

Comparison Between the Two Architectures (both using job optimization from Elog 232)

Notes on Chart:

Source of data located in /users/PAS1960/dylanwells1629/buildingPueoSim/testingouts/times.txt and /fs/ess/PAS1960/HornEvolutionTestingOSC/GENETIS_PUEO/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop/Run_Outputs/2023_07_24_test5/time.txt respectively)

Time from standard uses the time one of the 250 jobs running in parallel took in my testing of parallelizing processes inside of pueoSim jobs: 250 jobs * (40 * 40000 neutrinos per job) / 100 individuals = 4,000,000 neutrinos per individual)

The New Proposed time includes time spent analyzing the outputted root files to find fitness scores and errors, which would have taken around 100 seconds * population size for its number of neutrinos and files per individual (/fs/ess/PAS1960/HornEvolutionTestingOSC/GENETIS_PUEO/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop/Run_Outputs/2023_07_23_test5/time.txt for data on this number)

So, this new architecture can provide more improvements for the amount and speed of neutrino simulation in the loop on top of the methods discussed in Elog 232.

This architecture could also be applied to see improvements in the Bicone and Hpol loops which are both affected by the limited number of XF keys.

Additional Notes:

1. For this new architecture test, each antenna uses 49 jobs for neutrino simulation instead of 2.5 previously. (49 pueoSim + 1 XF for 50 jobs per antenna, 5,000 jobs per generation)

2. The time for each antenna to submit, queue, and finish its neutrino simulation jobs must be less than the length of the XF job, or the extra time will accumulate for each antenna, losing much of the time benefits. (As long as it is less, the time spent on just pueoSim should be invariant under an increase in population)

3. The number of core hours spent on pueoSim jobs will be roughly the same for the same number of neutrinos as each job is shorter (except for a small contribution of the job overhead taking a higher percentage of total time for shorter jobs)

4. Initially I had thought that maybe queuing pueoSim jobs while running the XF jobs could slow down the queue for the XF jobs. So, I made the loop wait to submit the batch pueoSim jobs until we had space for all of them to be active with the ~250 max jobs per user.

Additionally, while observing my many tests, I didn't notice any correlation between the number of CPU peuoSim jobs in the queue and the number of GPU XF jobs out of the queue.

5. Branch I'm developing this on is here

6. The total time for the test generation was 15.4 hours, which is slightly longer than the ~14 hours from the 2023_05_08 run. However, this test also used double the population size, larger values for the range of antenna heights (on average about 3x taller), and 20x more neutrinos simulated per antenna. So, the actual speed is better than it first looks.

Types of runs I need (May need some other people to help me run these):

1. Optimization runs

   1. Non-error

      1. Search over combinations of selection methods and genetic operators

         1. Fix Sigma at 10%.

      2. Do this for 10 runs for each 

      3. Re-run the best run for 100 tests to see if the results agree with 10 test results

      4. This will be the main talking point

      5. Save the plot for the best combination (proxy and metric)

   2. Error

      1. Re-run the best run for 100 tests with 3 different error amounts (0.1, 0.2, 0.3)

      2. We will compare these results to the non-error run to talk about how errors may affect consistency/speed. 

      3. Save an example plot of an error test for each error (both metric and proxy score)

   3. Population

      1. Re-run the best run for 100 tests with 3 different population sizes (100, 500, 1000)

      2. We will compare these results to the non-error run to talk about how population size affects consistency/speed. 

      3. Save an example plot of a population test for each size (just proxy score)

2. Demonstration runs (single runs, fitness score, no error population 100)

   1. Crossover only

   2. Mutation only

   3. Reproduction only

   4. Injection only

Currently, the PUEO loop runs pueoSim with 1 pueoSim process per job submitted.

Each of these jobs has 1 node and 8 cores, however, pueoSim only needs a single core to run.

Here is some data I collected by running the same seed of pueoSim with different numbers of cores in the job:

So, we should be able to run multiple processes of pueoSim without losing much performance.

Tests with more cores:

A test comparing the parallelization with the 40000 neutrinos per process we simulate in the loop:

So, these tests suggest we can simulate neutrinos ~20 times faster than currently done in the loop by utilizing more cores per job.

This method of multiple processes per job should be applicable to how we run other simulation software like AraSim as well.

(Note: both of these jobs had comparable queue times of 27 and 31 seconds respectively.

Also, I have tested and seen that the outputted root files are identical for the same seed regardless of if ran with multiple processes at the same time or with processes in serial)

(data found in /users/PAS1960/dylanwells1629/buildingPueoSim/testingouts/times.txt)

How to run multiple processes of pueoSim:

Previous Call:

./pueoBuilder/build/components/pueoSim/simulatePueo -i pueo.conf -o ${PSIMDIR}/outputs/${gen}_outputs/$SLURM_ARRAY_TASK_ID -r $run_num -n $NNT -e $Exp > $TMPDIR/out.txt

Multiple Processes Test Call:

for ((i=0; i<$threads; i++))

do

    ./simulatePueo -i pueo.conf -o $TMPDIR -r $i -n $NNU -e 19 > pueo.out &

done

wait

Instead of just calling peuoSim once per job, we can use the & symbol and a loop to run $threads number of pueoSim processes in the background and then wait for them to finish.

Recording of training on XFdtd by Alex Machtay on July 12, 2023.

https://osu.zoom.us/rec/share/FVKiTcQrRh8NLNYJADwugy0H2VjDo8ZlBsH16sJAGv2JINNwxxzyNfo5ueqVgOTO._XErF57Kz8KOM3a5

This link will expire in 120 days so note that we need to post a permanent recording.

Here's the GitHub repository with examples of Xmacros: https://github.com/Machtay/XFdtd_Scripts 

Attached are images of the matching circuit schematic and PCB design. A parts list is also attached in .pdf and .csv format. The .csv format can be imported into Digikey if necessary.  

Table of component values:

After discretizing and straightening the sides of the curved antenna that performed the best (Generation 13, individual 84) and running the antenna through AraSim, the following results and physics plots were generated. 

Here are a collection of antenna gain patterns with the antenna power source located at various heights at 500 MHz. The heights are indicated in the name of the images and were located at 3cm, 6cm, 12cm, and 24cm from the base. One thing to note is that the power source becomes longer as a function of height (since it's connected to the ridges, which slope back). 

The 3D gain plots attached show the gain becoming more uniform until we get to the last one, where it seems to reinvert and be much stronger in the upward direction than below. I'll update this post with some polar slices for a more complete comparison between them.

EDIT: 07/17/23

I've made a few adjustments and added plots of them.

• First, I separated the ridges. Previously, the ridges touched at their inner corners. This electrically connected them and may be why we saw very high peaks in the first set of impedance plots. You can see the plots named Separated_Impedance_<x>_cm.png.
  • This appears to have lowered the peaks considerably and also shifted them to lower frequencies. The peaks are still high when the source is placed high.
  • We seem to be developing another peak at higher frequencies, although this also existed in the previous tests for the high sources.
• After this, I added a waveguide and positioned the source between the extensions of the ridges into the waveguide. I've also added a picture that shows the waveguide in the geometry. 
  • The sources here are positioned from 1 cm below the ridges to 8 cm below the ridges. The naming convention is the same as above, except it includes "Waveguide" in the name.
  • The peaks are considerably lower here and seem consistent as we place sources further down. They are all below 500 Ohms, compared to as much as 4000 Ohms in the previous tests.

EDIT: 08/10/23

I've been trying more tweaks to the design to get rid of the peaks we see in the plots below. I'm having trouble actually getting the impedance to match close to the 50 Ohm value we expect from the cable. The best case situations seem to come from when the ridges are still touching, which we don't want. Here are the properties I've been changing:

• Ridge positions (gene in GA)
  • We don't want the ridges to touch, but I can't get them to be very far from each other anyway. The minimum wavelength in our bandwidth would be 20 cm, but that's close to the size of the bottom of the antenna for when we make it large, so the separation is still small.
  • When the ridges do touch is actually when I get the best matching,
• Bottom size (gene in GA)
  • I've been changing the size of the bottom of the antenna. This is something that can change substantially between individuals in the GA, so even if I find a good value for this, the impedance it gives wouldn't be the same for all individuals we see in an evolution.
  • This does seem to move the location of the impedance peak (in frequency) but I can't seem to get it fully out of band.
• Waveguide length
  • I've been changing the length of the waveguide between 3 cm and 10 cm. We said that somewhere in the middle should be the most reasonable (the example antenna we saw looked to have a waveguide of ~5 cm long). It seems like the source position is more important.
• Source position
  • I've been moving the position of the power source inside the waveguide. This seems to move around the peak and can lower its height, but I haven't been able to get peaks lower than ~400 Ohms, except in cases where the ridges are in contact.

I've added a picture of the best impedance I managed to get, but this involved an antenna with a small base and ridges touching. My next step is to just try a grid search by making many antennas automatically and simulating them all to look for any that have a nice impedance in our band. 

All data and plots discussed in this post are taken from this spreadsheet:

https://docs.google.com/spreadsheets/d/1BbpD81mugWQf10ozGDLI60takAn3tlvrq8ksT10yV5I/edit?usp=sharing

Run Details:

This optimization run was done to do a course search for the optimal set of parameters to run the GA with. This run used a fixed 100-individual population with a simulated error of 0.25 over 50 generations, to best replicate the current PUEO-loop environment. The selection methods have also been held fixed in this run. This run was tested using both the ARA and PUEO design to see if there were similarities in the best run type. This run encompassed 210 different run combinations that searched the following parameter space:

• 0-16 Reproduction, step sizes of 4
• 72-96 Crossover, step sizes of 4
• 4-16 Mutation, step sizes of 4
• 5-15 Sigma, step sizes of 5

Results:

Each run combination was run 10 times and we tracked the average number of generations it took for the distance metric to reach 0.05, which corresponds to a 0.95 true fitness score. The best runs for each design were as followed:

• ARA
  • Parameters
    • 12 Reproduction
    • 72 Crossover
    • 4 Mutation
    • 5 Sigma
  • Average gens to benchmark
    • 28.7 generations
  • Standard Deviation
    • 17.2 generations
• PUEO
  • Parameters
    • 0 Reproduction
    • 96 Crossover
    • 4 Mutation
    • 15 Sigma
  • Average gens to benchmark
    • 23.0 generations
  • Standard Deviation
    • 13.0 generations

We can see that these run types do not appear to have any strong correlations. In fact, looking at our complete data set, there does not appear to be a strong trend around any of the best runs. This issue became more prominent when we ran these best two run types for 100 tests rather than 10. When we did this, they returned averages of 44.4 generations for PUEO and 46.9 generations for ARA. For comparison, the average number of generations across all run types was 43.1 generations for PUEO and 42.8 generations for ARA. This seems to suggest that all of these run types are inherently inconsistent and regress to the mean given enough tests. Therefore, I do not believe we can draw any conclusions from this run.

Moving Forward:

Our ongoing hypothesis is that the size of the error could be causing inconsistency in how quickly our population can grow over time. Therefore, our next step is to repeat this experiment, but with zero error included, and see if we achieve results that show consistent behavior. If we can find more consistency in runs with no error, then we can more deeply explore the effects that error can have on GA growth. If there continues to be a lack of consistency, then we can look to other factors, such as population size, and try to find the root of the inconsistency.

To act as our comparison to the evolved antennas while plotting, we have done a simulation of pueoSim with 4,000,000 neutrinos for the measured toyon gains found in /fs/ess/PAS1960/buildingPueoSim/pueoBuilder/components/pueoSim/data/antennas/measured

In order to run the jobs, I used the runJobs.sh script found in /fs/ess/PAS1960/buildingPueoSim which submits job runs of 40,000 neutrinos at a time, mirroring what we do in the actual loop.

The resulting data is now located in /fs/ess/PAS1960/HornEvolutionOSC/GENETIS_PUEO/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop/Toyon_Outputs

Veff: 11750.0503856

Plus Error: 89.0051458227

Minus Error:88.9007824246

The fitness score plots have been updated to read this value in as a comparison.

Here are my recommended starting genes for the initial generation of the upcoming run (confirmed that these will produce a working antenna in XF). We want to use the lowest values we can because we expect that a larger (namely, taller) antenna will perform better:

S, H, x0, y0, yf, zf, beta
7.50000,50.0000,1.0000,1.00000,1.00000,50.0000,0.2

• S: Half the side length of the bottom of the antenna
• H: height of the antenna
• x0: distance of the bottom of the ridge from origin
• y0: width of ridge at the bottom
• yf: width of ridge at top
• zf: final height of the ridge (cannot exceed H)
• beta: a curvature parameter for the ridges

We want to automate tests in pull requests in GitHub to run unit tests on individual functions to ensure they return the expected results. This will help prevent bugged code from being pushed to working branches. We will add a list of unit tests to this ELOG as we design them.

Attached is a plot containing bar graphs with error bars representing the average number of generations it took for the GA to achieve a chi-squared value of 0.25 (roughly equated to a 0.8 out of a max 1.0 fitness score). Unlike the fitness scores used by the GA, these values do not have simulated error attached to them and are therefore a better measure of how well the GA is optimizing. These results were obtained by running 10 tests in the test loop for each design, population, and error combination and solving for the average number of generations to meet our threshold and the standard deviation of those scores. From this, a few things immediately pop out, I will address the more obvious one later. But essential for this test, we can see that increasing our population size seems to have a more direct impact on the number of generations needed to reach our threshold than decreased error does in both designs. My best guess regarding this is that the GA depends on diversity in its population in order to produce efficient growth, and an increase in the number of individuals contributes to this, allowing the GA to explore more options.

This leads me to the more easily spotted trend; PUEO is much slower than ARA. This presents an anomaly as this is the opposite of what we would expect from this test loop as PUEO has fewer genes (7) to optimize than ARA (8). It is also important to note that no genes are being held constant in this test for either design, so both designs have the full range of designs provided they are within the constraints. With that in mind, my guess as to what causes this phenomenon is that PUEO's constraints are much stronger than ARA's. How this may affect the growth is that it more heavily bounds the possible solutions, which makes it harder for the GA to iterate on designs. It is possible that during a function like a crossover, the only possible combinations for a pair of children are identical to their parents, effectively performing reproduction. This could limit the genetic diversity in a population and therefore cause an increase of generations needed to reach an answer. We could in theory test this by relaxing the constraints done by PUEO and then running the test again to see how it compares.

Finally, You will notice that no AREA designs are shown. This is because, under current conditions, they never reached the threshold within 50 generations. However, Bryan and I think we know what is happening there. AREA has 28 genes, about four times the amount of genes that our other designs use. Given that our current test loop fitness measure is dependent on a chi-squared value given by: SUM | ((observed(gene) - expected(gene))^2) / expected(gene) |, we can see that given more genes, the harder it gets to approach zero. For example, we can imagine in an ARA design if each index of the sum equals 0.1, you would get a total value of 0.8, while AREA would get a value of 2.8, which seems considerably worse, despite each index being the same. Upon further thinking about it, Bryan and I do not think that a chi-squared is the best measure of fit we could be using in this context. Another thing we thought about is that we have negative expected values in some cases. We have skirted around this by using absolute values, but upon reflection believe this to be an indicator of a poor choice in metric. Chi-Squared calculations seem to be a better fit for positive, independent, and normally distributed values, rather than our discrete values provided by our GA. With this in mind, we propose changing to a Normalized Euclidean Distance metric to calculate our fitness scores moving forward. This is given by the calculation: d = sqrt((1/max_genes) * SUM (observed(gene) - expected(gene))^2). This accomplishes a few things. First, it keeps the same 0 -> infinity bounds that our current measure has, allowing our 1/(1+d) fitness score to be bounded between 0 and 1. Second, it forces all indices to be positive so we don't need to worry about negative values in the calculation. Third, this function is weighted by the number of genes present for any given design, making them easier to compare than our current measure. Finally, as our GA is technically performing a search in an N-dimensional space for a location that provides a maximized fitness score, it makes sense to provide it the distance a solution is from that location as a measure of fit in our test loop. We created a branch on the test loop repository to test this and the results are promising as results from the three designs are much more comparable for the most part (though we still see some slowdown we think is contributed to constraints as mentioned above). Though some further input would be appreciated before we begin doing tests like the one we have done in the plot below.

How To Update PueoSim For GENETIS:

First, whoever updates pueoSim needs access to pueoBuilder, pueoSim, and niceMC on GitHub (ask Will for permissions).

Once you do, go into the peuoSim directory at /fs/ess/PAS1960/buildingPueoSim/

and source set_env.sh

`source set_env.sh`

Then, we want to make copies of the files we are currently modifying for the GENETIS loop.

For PueoSim v1.1.0 these are:

1. nicemc/src/EventGenerator.cc  # Modified to create custom output root files to calculate errors

2. pueosim/test/simulatePueo.cc # Modified to stop the simulation of the LF antenna which is not needed for GENETIS

3. pueosim/src/Seavey.cc # Modified to read in custom gain files

4. pueosim/src/pueo.cc # Modified to read in custom gain files

5. pueosim/include/Seavey.hh # Modifies to read in custom gain files

Currently, I store copies of these in the `/fs/ess/PAS1960/buildingPueoSim/backups` directory in case somebody accidentally overwrites the files in pueoBuilder. 

Once you’ve made the copies, you can run `./pueoBuilder.sh` from the `/fs/ess/PAS1960/buildingPueoSim/pueoBuiler` directory. This will rebuild pueoSim and niceMC, pulling the latest updates from GitHub. 

You may need to delete the pueoSim and niceMC directories in order for the builder to pull the latest version from GitHub. Or, if it’s being really stubborn, move the whole pueoBuilder directory to a temporary location and run the builder from scratch with 

`git clone git@github.com:PUEOCollaboration/pueoBuilder` and then `./pueoBuilder.sh`

Then, you will need to reference the copies of the changed files to make changes to the new version of pueoSim. Hope this doesn’t cause too much of a headache, and when you’re done, return to the /fs/ess/PAS1960/buildingPueoSim/pueoBuiler directory. 

Then you simply type

`make install`

then

`make`

And now, pueoSim should be ready to run!

EventGeneratior.cc Changes and Rationale:

PueoSim’s default output ROOT files are very large and therefore time-consuming to parse through to get the information we need to calculate effective volume and errors. So, we want to create a custom ROOT file with only the variables we want, greatly increasing the speed of the analysis.

To do this, we want to create a new TFile with corresponding TTree and TBranches that will store the loop.positionWeight, loop.directionWeight, and neutrino.path.weight. Then, we want to fill the Tree when a detector is triggered and write the results to the file at the end of the loop.

Sample code for initializing the TFile:

simulatePueo.cc Changes and Rationale:

By default, pueoSim v1.1.0 runs a simulation for the normal antenna and a low-frequency (LF) antenna. As GENETIS is evolving for the main antenna, we are not interested in using computation time to simulate the LF antenna. So, we comment out the lines that initialize the LF detector in this file.

Seavey.cc, Seavey.hh, and pueo.cc Changes and Rationale:

As of pueoSim v1.1.0, the simulation software only has the ability to read in one antenna. This is a problem, as we want it to be able to read in files for any antenna we make. So, we need to change these scripts to be able to read in whatever gain files we want.

The convention the loop uses is to input gain files in the format of 

vv_0_Toyon{runNum}

hh_0_Toyon{runNum}

…

In the ./pueoBuilder/components/pueosim/data/antennas/simulated directory

So, the main change is getting the runNum variable into Seavey.cc file where the gains are read. Currently, we define a global variable at the stary of pueo.cc and pass it into where Seavey is called. This means we have to add runNum as a parameter in Seavey.cc as well as Seavey.hh. Finally, we change the name of the read-in files to be in the above format AFTER dividing runNum by 1000 (this is because pueoSim uses the run number as a random number generating seed, so we divide runNum by 1000 to be able to read in the same gain patterns for multiple seeds of the same individual).

Note: We used to change pueoSim to output a veff.csv file. We now handle this calculation by analyzing ROOT files, so this change is no longer necessary. 

The Physics of Results Plots have been added to the Pueo Loop. The current version of the plotter is built for pueoSim v1.0 and located in ${WorkingDir}/Antenna_Performance_Metric (Hasn't been pulled into the loop directory yet).

The pueoSim v1.0 IceFinal files were missing information on the RF direction and information needed to see an amplitude spectrum. I asked Will, and he said that the new version of pueoSim (v1.1.0) outputs the needed information.

I created an updated version of the plotter, and have a pull request here: https://github.com/osu-particle-astrophysics/GENETIS_PUEO/pull/40

However, before this can be implemented in the loop, we need a way to get errors from pueoSim v1.1.0.

Currently, there is a version of rootAnalysis.py here that can analyze the new root files and output fitness scores and errors, but instead of taking 20 minutes for a generation, it now takes 20 minutes per individual to run.

This is because the update splits the IceFinal file into IceFinal_allTree, IceFinal_passTree0, and IceFinal_passTree1. IceFinal_allTree and IceFinal_passTree1 are of comparable size to the previous IceFinal file, but the passTree0 final is

about 10-20 times larger, causing a slowdown in calculations. If we calculate without this file, it takes about 1 minute per individual, still a bit slower than before.

Ideas to solve this issue:

Don't use the passTree0 files (I have asked Will what they're for, and if we need them. Hopefully he responds after the Holiday.)

Instead of running through a Python for loop, call a C++ script that creates a CSV file the Python program can easily load in.

Submit a batch job to run the analysis in parallel before combining the outputs in the correct format. (We should maybe do this anyways?)

Continue using pueoSim v1.0 (I think we can still retain the Theta graphs for the incoming neutrinos, but there won't be any RF graphs or the possibility of amplitude spectrum plots) 

Attached is a preview of plots the v1.1 plotter is capable of making.

Add weights.

Here are some preliminary results from testing the effect of error on growth in the GA. For this test, we start with a simulated error of 0.5 because our true fitness score is bounded between 0.0 and 1.0. From here, we simulate doubling the number of neutrinos by reducing the error by root(2), then root(4), and observed the growth on the fitness scores plots. We did these three tests with both 50 and 100 individuals (plots starting with 30 use 50 and starting with 60 use 100). Looking at the plots, we can see that the only plot that shows growth is 100 individuals with halved error (corresponding to 4 times the number of neutrinos). In fact, it took going to root(25) (corresponding to 25 times the number of neutrinos) to observe comparable growth with 50 individuals. This leads us to believe we likely need to increase both neutrinos and individuals to see meaningful growth in our GA. Further tests are required to see how consistent these results are. Another test, though unrealistic, we ran 1000 individuals with the root(25), this test was optimized in around 10 generations, this could be seen as a maximum performance standard.

Here is a list of things we agreed to do before starting a new PUEO run.  Feel free to add notes here as things are done and we will revisit the list next week.

Change range of the height variable to 0.75-1.75m

Start with all bad individuals so that we can be sure and see growth - all at 0.75m?

Make sure we are using the ratios in the GA that we think we are

If we are using reproduction, are we combining results when individuals are resimulated to reduce errors?  I made a program to accomplish this and created a pull request on the PUEO Github (Dylan)

Investigate whether there are obvious places to speed up PUEOSim

Consolidate the GA to one branch

Understand what is happening with cross-pol, make sure we're not gettting Veff=0 when using XF-generated cross-pol response

Edit: Have the xmacros use the realized gain instead of the idealized gain, which allows for the design to build in matching

Here is some backlogged information as well as recent updates to our progress on AREA and its optimizations:

4/13/2023

We concluded our initial test of the AREA optimization loop. While analyzing our results, we noticed that most of our runs never reached our benchmarks for performance (chi-squared scores of .25 and .1) and that those that did contain abnormally high amounts of mutation. From previous runs using the GA for the GENETIS loop, we find that mutation and immigration (Note: AREA does not use immigration) usually play a smaller role in overall growth and are simply there to promote diversity. Crossover on the other hand is what handles the bulk of the growth done by the GA. Given our results ran counter to that, we decided to look further into the matter.

Upon looking at the fitness score files, we could see that mutation-created individuals did not perform well in the best run types, but the crossover ones did contain the bulk of good scores. This lines up with expectations. However, closer inspection revealed that these crossover-derived individuals had extremely similar fitness scores, which could indicate elitism in selection methods leading to a lack of diversity in solutions. We then looked at a run that used only crossover with one selection type (roulette), and found every individual to have identical (and poor) scores that were the same up to around the 4th decimal place. This is a strong case that the selection methods used are too elitist to grow a population properly. 

We dove into the AREA GA to take a look at the functions doing the selection methods, and discovered several issues potentially causing excess elitism in the GA. It seems that the roulette selection does not behave the way we have used in our other GAs. There appears to be no weighting of individuals; rather, a threshold fitness score is collected and only individuals with a fitness score above the threshold are selected. This causes elitism, as only the most fit individuals are likely to pass this threshold. Tournament was found to be similarly troubled, with a bracket size of ⅓ of the population. This is again very elitist, as it is very large, making it very likely that only the best individuals will be selected. Before we try to further optimize the AREA GA, these selection methods should be addressed and fixed.

For roulette, we propose implementing a similar weighting scheme for individuals as is used in the PAEA GAs that gives preference to the most fit individuals, but still allows others to be selected to encourage diversity of solutions.

For crossover, we propose decreasing the bracket size. This would allow a more diverse range of individuals to be selected through “winning” smaller brackets that would have a lower probability of including top performing individuals.

4/20/23

We (Bryan and Ryan) met to work on improving the selection method functions in the AREA algorithm. 

For roulette selection, we added weighting by fitness score, as is done in the other GENETIS GAs, in an effort to prevent it from disproportionately selecting the most fit individuals only to become parents. As a first test, we ran an abbreviated version of the test-loop code, printing the fitness scores of the parents selected by the roulette method. The fix appears to have passed this initial test, as the fitness scores showed more variety and in the case of roulette crossover two unique parents seemed to be selected.

For tournament selection, the only change necessary appeared to be decreasing the bracket size of the tournament(s), which in turn encourages other individuals besides the most fit to be selected as parents, therefore increasing diversity of solutions. We ran out of time to test this change fully, so we will pick up here in our next meeting.

Looking further ahead, we will plan to work on refactoring the AREA algorithm to use the same functions as the other GENETIS GAs wherever possible. In the meantime, this working version can still be used for feature development so that progress is not halted.

5/11/2023

Upon looking at the results of the recent optimizations, our best runs are still taking over 40 generations to reach an optimized score. Given the speed of AraSim, this is still too slow to be considered optimal. Comparing this test loop run to the ones run on by the broader GENETIS GA, our best runs for the AREA GA are worse than the worst runs from those optimizations. As such, we have decided to pause optimizing the AREA GA in favor of adding it to the broader GENETIS one. As of today, we have finished a rough version of the generating function necessary to create new individuals. We have also started progress on the constraining functions and scaling functions that will be necessary to generate the spherical harmonics used to find the gain and phase. Once these functions are complete, we should be able to transplant this GA back into our test loop for AREA and resume optimizations.
 

In this post, I'm going to give step by step instructions on everything you need to do to run the loop. See Julie's thesis for the most recent iteration of this manual. Also refer to entry 123 for a list of some errors we have encountered historically in the loop and how to resolve them, though some fixes may be outdated.

Running the loop

Getting started

Assuming you have followed the instructions from the onboarding tutorials, you should be ready to access OSC through the interactive website. Instead of using ssh to access OSC, the loop requires that you go to this website: https://ondemand.osc.edu/ . Login with your usual OSC credentials. Once there, you will need to request a Lightweight Desktop by clicking on "Interactive Apps" at the top of the page. See the screenshots attached for how to do this. Old documentation may refer to the Lightweight Desktop as a VDI.

You can request up to 24 hours for a Lightweight Desktop, and it should begin almost immediately. After you request it, the website will take you directly to the waiting page. You can navigate to this page yourself by clicking "My Interactive Sessions" at the top of the ondemand page. When the job begins, you will see a blue buttong that says "Launch Lightweight Desktop". Click it and you should have a new tab open that brings you to a remote desktop environment on OSC that you interact with through your browser. 

New runs

If you're starting a new run for the first time (that is, beginning from generation 0), you will need to edit the script that runs the loop. Here is the path:  /fs/ess/PAS1960/HornEvolutionOSC/GENETIS_PUEO/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop/Loop_Scripts/Asym_XF_Loop.sh . At the top of the script is a list of variables that can be changed. See the first screenshot attached here.
For a new run, you must change the variable <RunName>, which is the first variable in the list. It should be in quotes and the convention is to start the name off with the date of the run, using YYYY/MM/DD format--this will always keep everything in chronological order when listing directories with <ls>. Other variables are explained in the comments next to them in Asym_XF_Loop.sh and you should change them depending on the type of run you are conducting. 

To begin the loop, type in the command ./Loop_Scripts/Asym_XF_Loop.sh from the Evolutionary_Loop directory.

When the loop begins, you will be asked to press any key. After you hit a key, the loop will run Part A, which executes the GA using the <start> mode. It will then automatically move on to Part B, which creates the antennas in XFdtd. XF will ask you if you want to specify a location for the XF file to be saved. Click <No>. The loop will automatically save the XF directory into the run directory when it finishes with the first part of part B. In principle (that is, barring errors that stall the loop or interruptions due to the VDI job ending), you will never need to manually enter anything to the loop for it to run beyond this point. 

However, if it is your first time using XF, you will need to set the max GHz setting upon startup to 2 instead of the default 10. 

Continuing Runs

After starting a new run, the loop will run automatically. When your Lightweight Desktop job ends, you'll need to get another one repeat the steps above. You will not be prompted for any input. 

Stepping Back

At times, you may want to continue the loop from an earlier point or interrupt the loop and step back. Usually you'll only want to do this within a generation, but it's possible that you will encounter an error or bug at the end of a generation and wish to return back to that point to remedy it before continuing forward in the next generation. This is a very tricky and delicate process that requires a lot of attention to detail. In the future, we'd like to make an option for the loop to do this automatically when given input at the beginning, but implementing that will take a long time and will be very sophisticated.

The state of the loop is recorded in the savestate file located here: /fs/ess/PAS1960/HornEvolutionOSC/GENETIS_PUEO/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop/saveStates . The file will be named <Run_Name>.savestate.txt . In it, you will see three numbers. The first one represents the current generation of the loop. The second one represents the state of the loop, that is, which part in the generation it is currently on. The third one represents the individual it is processing. In practice, the third number is rarely relevant, and is almost always 1, though it can be used for Part B, where the loop is processing individuals sequentially, so if it was interrupted on individual eight (for example) you can change the number to be 8 so that it will pick up where it left off without needing to redo the first seven individuals.

If you want to step back to an earlier state in a generation, you'll need to interrupt the loop and revert the state by changing the second number in the savestate file. This is NOT all you will have to do, however. Because of the way files are edited and moved around during states, you will need to manually revert files to be in the format or location needed by the earlier state. 

For example, you may need to revert from state 2 (when XF is creating the antenna geometries) to state 1 (when the GA generates new individuals). In this case, you'll need to edit the savestate file to change the second number from a 2 to a 1. Then, you'll need to change the generationDNA.csv file in Generation_Data to use the values from the previous generation, since the GA will have overwritten that file and uses it to determine what individuals are to be selected from. To do this, you'll need to locate the <gen>_generationDNA.csv file from the previous generation in the <RunName> directory in Run_Outputs. This is should be found here:  /fs/ess/PAS1960/HornEvolutionOSC/GENETIS_PUEO/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop/Run_Outputs/<RunName>/Generation_Data . 

The above is a relatively simple example of how to step back in the loop. Stepping back from a late state to an early state will be more complicated, though if you're willing to sacrifice run time you can pretty easily follow the above to just restart a generation. Stepping back to a previous generation will be explained in a future update to this post.

Here are the run details for the latest PUEO run we've started. There are still some bugs that need to be worked out, which are detailed on the github. Here is the directory where the data is stored: /fs/ess/PAS1960/HornEvolutionOSC/GENETIS_PUEO/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop/Run_Outputs/2023_05_08

This run uses 50 individuals per generation with 400k neutrinos per individual at an energy exponent of 19. Only the inner side length and outer sidelength are being evolved (which evolves the height since the opening angle is fixed). 

Issues With Part E (Dylan):

The Plots were not automatically created for the first generation. After a quick look through, I found that the fitness score and v effective csv files were being moved before they were copied, so the files didn’t end up in the correct place. Additionally, the errorbars.csv file was not being created, so I made a temporary fix to create a file of 0 error bars to allow for plotting the values while I work on getting the real error values. 

Update: I have modified Will’s script for reading the root file outputs, so it can now read them to output a CSV file with the effective volume along with its positive and negative errors. This script does the same and more as the fitnessfunctionPUEO.py, so I will replace it in the loop when there is downtime. Additionally, this script means we don’t need to output a veff.csv file from pueoSim. I will now work on creating a less “janky” version of my pueoSim change for reading in the gain files, so I can get Will to officially add it to pueoSim and we can pull any updates in the future.

Update, Implementing the new script into Part E:

The rootAnalysis.py script is very picky with environment variables. So, first I had to load and then unload a version of python to make sure the default version is loaded. Then, I moved the sourcing of set_plotting_env.sh to later in the script to avoid having an alternate version of root sourced while running (this would cause errors). Then I source the environment for pueosim (path hardcoded, change when loop isn’t running) and loop through the rootAnalysis.py program for each individual in the generation. This will output a ${gen}_fitnessScores.csv and a ${gen}_errorBars.csv in the run directory for plotting script to use. Because it outputs these automatically, I commented out the old methods we used to create these files and removed the temporary error bar fix as well.

Another Update:

We paused the loop. I changed the hardcoded parts, and we pushed the new updates to the github. I will make comments on the changes later today. 

Wednesday:

It seems like the loop is still not creating the csv files or plots for generation 1 or 2. Alex gave me the error message. The python script rootAnalysis.py had an issue with the include Geoid.h line. This was caused by sourcing the wrong things while trying to get the script working. I found that I only changed my local pueoSim environment to include the fftw installation to the $PATH and $LD_LIBRARY_PATH variables. So, I changed /fs/ess/PAS1960/buildingPueoSim/set_env.sh as well. Also, I found that the include statements in rootAnalysis.py were missing the PAS1960 portion of the path. After fixing these, and adding a missing ‘/’ to the output file path, I manually ran the rootAnalysis.py program for the 2023_05_08 run, mimicking the portion of the bash script that runs it in Part_E. It outputted the correct files in the correct format. I then ran the FScorePlot script. The median line for the ViolinPlot seemed wrong, and the FScorePlot was too scrunched with a y start of 0; I will need to change these.

Plotting Update (Dylan):

Ok, plots still aren’t being made, and I’m not running the loop, so I don’t have access to the error messages, so I’ll go through everything one by one. For Veff_Plots, this isn’t being made as there are currently no vEffectives.csv. This isn’t a big deal, as it’s just a copy of FScorePlot, but I added the creation of vEffectives.csv to rootAnalysis.py anyway. Next, I noticed that runData.csv, which is required for a couple of the plots was not being made. I looked, and the version of gensData.py had the old values for the skiprows when reading in the fitnessScores and generationDNA files, something that was overlooked when we merged the version I was developing with the version Alex was developing on. I can’t access the directory or the file, I think this is why it was overlooked in the merge. I created a temporary PUEO version in Antenna_Performance_Metric (I’m also fine with it just living here permanently). 

Also, the csv files are now in the GenerationData/ directory in the run directory? This affects every plotting script and they will not work without changes. I don’t know why this was changed or if this change is permanent?

Next, ok apparently, VariablePlots isn’t even a file in the loop? We really need to use GitHub better. 

I moved my local version of the program into the loop and it ran fine manually.

I am forsaking the Generation_Data directory, so I changed the rainbowPlotter and dataConverter scripts to read in a location to match all the other plotting scripts. They both also ran fine manually. 

Note: The rainbowPlotter script should be able to create a gradient from the minimum fitness score to the maximum fitness score. (currently, the range is hardcoded and this can cause basically a mono-color graph.)

FScorePlot should be good

The bash script was inputting 5 variables into colorplots when the pueo version only takes in 4 (this is also something that was already fixed before the merge.) It also requires the vEffective.csv files. After copying over the fitnessScore.csv files to vEffective.csv files, it ran fine. (The vEffective.csv files will be made automatically by rootAnalysis.py in the future)

I've started a run of the PUEO loop to get some data for evolving the antenna width and height. We are running with 24 individuals, each with 400k neutrinos at an energy exponent of 19. Here is the output directory: /fs/ess/PAS1960/HornEvolutionOSC/GENETIS_PUEO/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop/Run_Outputs/2023_05_01_Test_5

It's very important to note that the cross-pol data being used still comes from the data already stored in pueosim. This is because our extremely low cross polarizations were causing us to have no effective volumes. There's likely some issue in what values from XF we are choosing as our crosspols.

We have the PUEO loop in working order, though there are a few errors in accuracy and efficiency. We started a preliminary run with few individuals and neutrrinos as a large scale test that may still give us some useful data. 

For this run, we are only evolving the inner side length (10cm to 25 cm) and height (10 cm to 50 cm). The walls are slanted at 45 degrees, so the outer length is completely defined by those two. The ridges are kept at 6 cm wide and 4 cm from the walls at the base. The curvature is set to 0,1 and the ridges are set to have the same height as the walls. 

At the moment, we are using 24 individuals and 300,000 neutrinos. I'll update this ELOG post with more details from the run as they emerge.

The reviewer asked us to run the loop again but for just symmetric antenna designs. We have completed that run and have results ready. We still need to resimulate the top five individuals, which I'll update this post with later. I attached the violin plot (though it should be fixed to not let the legend overlap the data). See this previous entry for the details of the run: http://radiorm.physics.ohio-state.edu/elog/GENETIS/198

Plots we want to have for PUEO:

FScorePlot2D

Fitness_Scores_RG

VariablePlot (PUEO equivalent of LRT plot)

Veff_Plot (currently the veff is equivalent to the fitness score for PUEO)

Veffectives_RG

Rainbow_Plot 

Violin Plot

avg_freq

gain_vs_freq

polar_plotter

physics of results

Next steps:

Get the Error values out of the pueoSim root output (talking to Will)

Use output uan files from the new two-run XF simulation to test the frequency plots. (/fs/ess/PAS1960/HornEvolutionOSC/GENETIS_PUEO/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop/Run_Outputs/2023_04_21_Test_6)

Implement the physics of results plots into the loop (Talk to Bryan / Dennis)

Get the average Veff/Fitness score from the default gain pattern in pueoSim to compare our results to.