I've been working on readying the loop for the transition to slurm. OSC is switching from its current job managing software (torque/moab) to slurm, which uses some different commands but is functionally similar. Since we have places where jobs are called in the script, I've been changing them (with a backup copy of those files that we know works on torque). 

Right now I'm testing all of the changes I've made on the space OSC has set aside for playing with slurm. I have changes in all of the places I think they belong, but I need to make sure there aren't typos/different commands from what I've used. 

I started working on the slides for our meeting with Wolfgang this Friday. I'll share it via one drive/email/slack tomorrow for comments from everyone.

Over the weekend I finished the asymmetric length run. I still have a couple of fixes to make on plotting scripts, so I'll update those plots tomorrow after making the fixes. I also started a new run today using both an asymmetric length and an asymmetric opening angle. The plots aren't automatically uploading to dropbox anymore because we hit our limit, so I'll have to add them manually. You can see the asymmetric run on dropbox under  Machtay_2020_08_27_Asymmetric_Length_Real_Run(SECOND REAL RUN).

I worked on fixing the AraSim issue today. I think I have it resolved. Essentially, I made a second set of flags that the loop will look for and use the first set of flags to tell the loop which error files to look at to look for the phrase "segmentation fault." If it sees that phrase, it resubmits the job. Otherwise, it makes the second flag. Once all of the flags have been made, the loop can move on. I tested it by manually putting the phrase inside one of the error files to make sure it would catch it and resubmit the job. 

I started a new run (partly to further test the fix, but primarily to get real data). It has the fixes we discussed last week at the GENETIS meeting (ex: mutations shouldn't repeat anymore).

I'm going to be out tomorrow and Thursday so someone needs to pick up my run. I'll make sure I open permissions tonight so that anyone can run it tomorrow. If you have questions you can slack me, but I probably won't be able to respond til super late.

If there are issues with AraSim jobs failing without resubmitting, see the instructions below.

Alex M

Alex P

Eliot

Leo

Evelyn

Ryan

Ben

Ethan

Continued running the loop up to gen 15. I also adjusted the LRTPlot script so that it can slightly spread the points along the generation axis so that you can see overlapping points, but I still need to implement this in the loop. 

Helped Ryan design the bash script for running through paperclip many times. It's working and he ran it for a while to get ~1000 outputs checking the performance of different mutation sizes. Next he'll parse through them to find what sigma appears best.

Tested the database setting for the symmetric loop. It ran well and gave correct results, so all that's left to test is the asymmetric loop (going to start that momentarily). 

Julie and I helped Ben and Ethan set up a git repository for AREA and gave a brief review of using git.

Leo, Julie, and I merged the two dev branches (Antenna_Database and asymBicone) onto master. I think I resolved all of the conflicts and successfully merged. I'm going to start testing by running each setting (original, database, and asymmetric) through generation 1 to make sure that they give the correct results as the previous test runs and that they have all of the correct files they need to run correctly. Once I'm done with this I'll implement the changes I outlined last week (on the 21st I think) and begin a real run.

I helped Ryan with the paperclip algorithm. We found the error in the roulette algorithm and it now gives results comparable to tournament mode. There is a bug that needs to be fixed (Ryan changed a variable to be taken as an input, but it's giving different results when we input the same values as it had before and neither of us can tell why) but otherwise the script looks much better than last week with the roulette fix. 

Leo and I continued working on the roulette algorithm for the asymmetric bicone to try to get the evolved generations to match the original and database ones. We went through the mutation part of the algorithm and made sure we were rrunning all of the random number sequences in the correct order (for example, we need to make sure that if we define variables A and B from a random number generator, that in both cases we define A first, then B). We also had to make sure that we were using these random sequences the same number of times (for example, we don't want to define A once in the original/database versions and twice in the asymmetric version when the asymmetric version is set to evolve symmetric antennas--this puts the random number generator "off sequence" relative to how it is in the original/database version). We tested this with the gen 0 DNA and effective volumes (which matched with the results from the original/database runs) to evolve generation 1 and got it to match. I'm currently running the loop to test this over multiple generations.

Julie and I met with Ben to talk about AREA. Yesterday, Ben was able to generate the 0th generation of individuals and then run AraSim from a job to get effective volumes. Today, we focused on editing the scripts to work for OSC job submissions (he wrote the one he used yesterday; we modified the main script today and will continue to do so). We edited it so that we could submit the main script as a job which then submits other jobs for the arasim portion and waits for those jobs to finish before continuing on (since it needs those effective volumes for its fitness score). We're going to keep incrementally fixing the main script so that it can run over multiple generations as a job submission without errors.

I have a doctor's appointment in the morning, but maybe Leo can continue my current run. Once it's done, optimally we would work with Julie to get everything merged so that we can start a real run (we also have to implement the changes I outlined in the ELOG post a couple days ago before we do a real run/merge--most of those are pretty simple).

Julie and I will take a look at the paperclip script Ryan's been writing to see if we can help him figure out why the roulette algorithm is giving such worse fitness scores than the tournament algorithm (~ fact of 2 worse).