We began a new run using the same parameters as the previous one (see the last ELOG entry). The previous one ran for 8 generations. The difference between this run and the last one is that we have added in the polarity fix for AraSim Brian and Jorge gave us (copying the correct Report.cc and Report.h files into our AraSim version). Here are the run parameters:

Run Paramaters:
50 individuals
80/20 Roulette/Tournament
6% Reproduction, 72% Crossover, 22% Mutation
Run for 10 generations, then implement Jorge and Brian's fix for polarization in AraSim for the next run

We are currently running the loop using the improved parameters. The parameters are as follows:

50 individuals
80/20 Roulette/Tournament
6% Reproduction, 72% Crossover, 22% Mutation

Also note that individual 5 of generation 1 had its fitness score altered because one of its AraSim jobs didn't finish in time (and we would have had to wait 5 hours for it to rerun). To remedy this, I replaced the effective volume for the job with the average of the effective volumes from the other jobs for individual 5 (essentially meaning that it ran with 270k neutrinos instead of 300k). There is a .txt file in the directory titled Run_Notes.txt containing this message (and which will contain additional messages on similar things should they arise).

Run directory: /fs/project/PAS0654/BiconeEvolutionOSC/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop/Run_Outputs/Improved_Parameter_Run_2021_03_09

Because of issues we were having with XF when running at the command line, we've shifted to running through a Pitzer VDI (see here: https://ondemand.osc.edu/pun/sys/dashboard/batch_connect/sys/bc_desktop/vdi/session_contexts/new).

The VDI will give you a mirror of a desktop running on OSC's side, so things like XF will run much faster. You can open a terminal by clicking the black box at the bottom of the screen and then run the loop like normal (./Asym_XF_Loop.sh). This has the added benefit of making it possible to exit out of your browser, close your laptop, or turn off your computer without interrupting the run, since the process is running on OSC's side rather than piping to your computer.

Here's the video I made a few months ago demonstrating how to run the loop. There might be some changes that have been made since then, but they'll generally be minor so it should still be representative of what running the loop looks like. It might need to be shared with you through the Microsoft drive service OSU gives us in order to view it, so either message me on Slack or email me at machtay.1@osu.edu and I'll share the link through there.

https://drive.google.com/file/d/1yL_eH_w2hXNt7J7YOwS5xl3ZHwGyl5JR/view

OSC managed to figure out our issue with running XF from an interactive job on Slurm. Previously, we were losing our connection to x11 forwarding. The solution is to use sinteractive instead of srun to obtain an interactive job. Here's the syntax:

sinteractive -A PAS0654 -t <time> -N 1 -n 8 -p serial

The -p serial flag denotes the type of partition to request. It's important to specify this, as otherwise it will default to the debug node, which has a limit of 1 hour.

We're going to begin a new run. The title is Machtay_2021_1_15_NPOP50_Asym . It is using the latest version of the GA Ryan has been working on (Latest_GA_Asym.cpp). We're using 6% reproduciton, 72% crossover, and 22% mutation. We are using 80% roulette selection. This correlates to the three numbers passed into the GA being 3 36 8 (since we're using 50 individuals).

Today I removed some of the run directories which had very little or no data or weren't worth keeping around. There are still a few that I think can be removed, but I'm keeping them until we can get a consensus that they can definitely be removed. Below I listed the names and descriptions of the runs that I think we should definitely preserve going forward. In general, the more data contained in the run directory, the more important it is to keep around. 

Attached is a .txt file you can find in the Evolutionary_Loop directory as Loop_Error_List.txt. It's a list of the current errors we sometimes experience in the loop, along with how to fix them if you encounter them while running. If people encounter errors that aren't in the list, let everyone know in the #genstudents chat on slack and update the file with the error message and when it was encountered (what state the loop was in) and the possible cause and solution if you know it. 

AraSim EField project Error:
 

 *** Break *** segmentation violation

===========================================================
There was a crash.
This is the entire stack trace of all threads:
===========================================================
#0  0x00002b845aeb445c in waitpid () from /lib64/libc.so.6
#1  0x00002b845ae31f52 in do_system () from /lib64/libc.so.6
#2  0x00002b84562bebf4 in TUnixSystem::StackTrace() () from /cvmfs/ara.opensciencegrid.org/trunk/centos7/root_build/lib/libCore.so
#3  0x00002b84562c13ea in TUnixSystem::DispatchSignals(ESignals) () from /cvmfs/ara.opensciencegrid.org/trunk/centos7/root_build/lib/libCore.so
#4  <signal handler called>
#5  0x000000000044fd70 in void std::vector<double, std::allocator<double> >::emplace_back<double>(double&&) ()
#6  0x00000000004c3c0e in Report::Connect_Interaction_Detector(Event*, Detector*, RaySolver*, Signal*, IceModel*, Settings*, Trigger*, int) ()
#7  0x0000000000436eb3 in main ()
===========================================================

The lines below might hint at the cause of the crash.
You may get help by asking at the ROOT forum http://root.cern.ch/forum
Only if you are really convinced it is a bug in ROOT then please submit a
report at http://root.cern.ch/bugs Please post the ENTIRE stack trace
from above as an attachment in addition to anything else
that might help us fixing this issue.
===========================================================
#5  0x000000000044fd70 in void std::vector<double, std::allocator<double> >::emplace_back<double>(double&&) ()
#6  0x00000000004c3c0e in Report::Connect_Interaction_Detector(Event*, Detector*, RaySolver*, Signal*, IceModel*, Settings*, Trigger*, int) ()
#7  0x0000000000436eb3 in main ()
===========================================================

I hopped on with Alex M to catch him up with what has been done and changed within AraSim so far. I looked into using pointers to access arrays outside of their declared scope and it seemed reletively easy to set up. I set up a pointer corresponding to the arrays V_forfft, T_forfft, and vmmhz_filter and set them at the end of stage one in order to be used in stage two. Also made sure to delete the pointers at the end of stage 2 (Very Important). After this and finding the correct scope to put the deletes in, we complied and didn't get an error in Report.cc. Once it compiled I signed off to study for exam and Alex M will now take over and start to run tests to make sure that with the same seed this gives the same results as a base copy of AraSim. After this the next step would be to start implementing a way to save all the variables from stage one and reading them into stage 2.