Dropbox link: https://www.dropbox.com/home/GP_Antennas/Updates/DailyFitnessScoreImages/Machtay_20200803_Master_Symmetric_Database_Real

Instructions for persisting AraSim errors:

Even if we still have the problem where AraSim jobs fail and the fix doesn't catch them, you can still run the loop as long as you manually step back in the loop. Here's what you have to do:

1. Exit the loop (ctrl-c).

2. Clear the following directories in the RunName directory: AraSimFlags, AraSimConfirmed, (gen)_AraSim_Errors, (gen)_AraSim_Outputs. 

3. Revert the save state (second number goes to 5). This let's you resubmit the AraSim job.

4. Run the loop again (make sure you're still in an interactive job).

Here's a power point I showed today at the GENETIS meeting for the two most recent runs we did. The first run was symmetric and we ended it at 15 generations. The second one is ongoing and is shown up through generation 9 on the power point.

I just wanted to give an update on what I've been doing today for GENETIS. I went ahead and recorded myself running the loop. It should be useful for new people we're onboarding but I also went through a whole generation, which I think is interesting for everyone to understand the timescale of each part of the loop. I'm making a small edit (I have no clue how to edit videos haha) so once I'm done I'll try to send it via email and post it on slack. I also started a long run Amy requested using antennas which are asymmetric in length and opening angle and 50 individuals per generation. Since it has so many individuals it will take a long time to go through each generation. I also had to decrease the number of AraSim jobs (thus increasing the number of neutrinos per job) because otherwise we'd be submitting too many jobs and have a lot of them blocked, holding us back from running. If anyone has problems with me using up too many jobs let me know and I can try to decrease the numer of jobs per individual futher (right now I have 10 jobs per individual with 3000 neutrinos per job, which gives us 500 AraSim jobs per generation).

AraSim EField project Error:
 

 *** Break *** segmentation violation

===========================================================
There was a crash.
This is the entire stack trace of all threads:
===========================================================
#0  0x00002b845aeb445c in waitpid () from /lib64/libc.so.6
#1  0x00002b845ae31f52 in do_system () from /lib64/libc.so.6
#2  0x00002b84562bebf4 in TUnixSystem::StackTrace() () from /cvmfs/ara.opensciencegrid.org/trunk/centos7/root_build/lib/libCore.so
#3  0x00002b84562c13ea in TUnixSystem::DispatchSignals(ESignals) () from /cvmfs/ara.opensciencegrid.org/trunk/centos7/root_build/lib/libCore.so
#4  <signal handler called>
#5  0x000000000044fd70 in void std::vector<double, std::allocator<double> >::emplace_back<double>(double&&) ()
#6  0x00000000004c3c0e in Report::Connect_Interaction_Detector(Event*, Detector*, RaySolver*, Signal*, IceModel*, Settings*, Trigger*, int) ()
#7  0x0000000000436eb3 in main ()
===========================================================

The lines below might hint at the cause of the crash.
You may get help by asking at the ROOT forum http://root.cern.ch/forum
Only if you are really convinced it is a bug in ROOT then please submit a
report at http://root.cern.ch/bugs Please post the ENTIRE stack trace
from above as an attachment in addition to anything else
that might help us fixing this issue.
===========================================================
#5  0x000000000044fd70 in void std::vector<double, std::allocator<double> >::emplace_back<double>(double&&) ()
#6  0x00000000004c3c0e in Report::Connect_Interaction_Detector(Event*, Detector*, RaySolver*, Signal*, IceModel*, Settings*, Trigger*, int) ()
#7  0x0000000000436eb3 in main ()
===========================================================

Attached is a .txt file you can find in the Evolutionary_Loop directory as Loop_Error_List.txt. It's a list of the current errors we sometimes experience in the loop, along with how to fix them if you encounter them while running. If people encounter errors that aren't in the list, let everyone know in the #genstudents chat on slack and update the file with the error message and when it was encountered (what state the loop was in) and the possible cause and solution if you know it. 

Today I removed some of the run directories which had very little or no data or weren't worth keeping around. There are still a few that I think can be removed, but I'm keeping them until we can get a consensus that they can definitely be removed. Below I listed the names and descriptions of the runs that I think we should definitely preserve going forward. In general, the more data contained in the run directory, the more important it is to keep around. 

OSC managed to figure out our issue with running XF from an interactive job on Slurm. Previously, we were losing our connection to x11 forwarding. The solution is to use sinteractive instead of srun to obtain an interactive job. Here's the syntax:

sinteractive -A PAS0654 -t <time> -N 1 -n 8 -p serial

The -p serial flag denotes the type of partition to request. It's important to specify this, as otherwise it will default to the debug node, which has a limit of 1 hour.

We're going to begin a new run. The title is Machtay_2021_1_15_NPOP50_Asym . It is using the latest version of the GA Ryan has been working on (Latest_GA_Asym.cpp). We're using 6% reproduciton, 72% crossover, and 22% mutation. We are using 80% roulette selection. This correlates to the three numbers passed into the GA being 3 36 8 (since we're using 50 individuals).

Here's the video I made a few months ago demonstrating how to run the loop. There might be some changes that have been made since then, but they'll generally be minor so it should still be representative of what running the loop looks like. It might need to be shared with you through the Microsoft drive service OSU gives us in order to view it, so either message me on Slack or email me at machtay.1@osu.edu and I'll share the link through there.

https://drive.google.com/file/d/1yL_eH_w2hXNt7J7YOwS5xl3ZHwGyl5JR/view

Because of issues we were having with XF when running at the command line, we've shifted to running through a Pitzer VDI (see here: https://ondemand.osc.edu/pun/sys/dashboard/batch_connect/sys/bc_desktop/vdi/session_contexts/new).

The VDI will give you a mirror of a desktop running on OSC's side, so things like XF will run much faster. You can open a terminal by clicking the black box at the bottom of the screen and then run the loop like normal (./Asym_XF_Loop.sh). This has the added benefit of making it possible to exit out of your browser, close your laptop, or turn off your computer without interrupting the run, since the process is running on OSC's side rather than piping to your computer.

We are currently running the loop using the improved parameters. The parameters are as follows:

50 individuals
80/20 Roulette/Tournament
6% Reproduction, 72% Crossover, 22% Mutation

Also note that individual 5 of generation 1 had its fitness score altered because one of its AraSim jobs didn't finish in time (and we would have had to wait 5 hours for it to rerun). To remedy this, I replaced the effective volume for the job with the average of the effective volumes from the other jobs for individual 5 (essentially meaning that it ran with 270k neutrinos instead of 300k). There is a .txt file in the directory titled Run_Notes.txt containing this message (and which will contain additional messages on similar things should they arise).

Run directory: /fs/project/PAS0654/BiconeEvolutionOSC/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop/Run_Outputs/Improved_Parameter_Run_2021_03_09

We began a new run using the same parameters as the previous one (see the last ELOG entry). The previous one ran for 8 generations. The difference between this run and the last one is that we have added in the polarity fix for AraSim Brian and Jorge gave us (copying the correct Report.cc and Report.h files into our AraSim version). Here are the run parameters:

Run Paramaters:
50 individuals
80/20 Roulette/Tournament
6% Reproduction, 72% Crossover, 22% Mutation
Run for 10 generations, then implement Jorge and Brian's fix for polarization in AraSim for the next run