Paperclip Antenna (Ryan and Evelyn working on it):

• https://github.com/gp-antennas/Sph._Harmonics_And_Paperclip

Asymmetric Bicone (Leo and Eliot working on it):

• https://github.com/JulieRolla/GENETISBicone

AREA (Ben working on it):

• Github? We need to ask Steph where their past work exists.
• See AREA thesis attached

Symmetric Bicone(Alex M. and Alex P. working on it):

• https://github.com/JulieRolla/GENETISBicone

d

Today's Summer 2020 daily update:

Today's Summer 2020 daily update:

Learn how to use the final part of the loop

Today's Summer 2020 daily update:

As a note, today OSC was down so productivity was more limited

Today's Summer 2020 daily update:

As a note, today OSC was down so productivity was more limited

Today's Summer 2020 daily update:

Julie: Since I  won't be doing GENETIS work every day necessarily, I'll add myself down here with a blurb when I am working with the others. I will also be continuing my daily updates on Dropbox for non-GENETIS specifics. 

We met today for a long time and pushed to catch up. The following is what we have done:

(1) Worked for a bit on our current tasks. They will work on these current tasks before they start on their summer projects. 

• Antenna database (Julie, Alex P., Alex M.)
• Worked on a paper outline (Julie, Alex M.)
• Worked on wall time variable (Eliot, Evelyn, Leo)

       Note I also have to: (1) Fix merge conflict and make sure students are good about only editing the dev branch moving forward. (2) Get branches and GitHub set up for both the asymmetric bicone and the non-linear bicone. 

(2) Went over the details of how the loop works as a group.

• Went over how each piece works.
• Went over how to run it. 
• Went over best ways to test new edits. 

(3) Went over summer tasks.

• Eliot: Add parameters to bicone (ie make bicone asymmetric)
• Julie & Alex M.: Paper, and helping others
• Alex P.: Frequency database for AraSim (to speed up AraSim)
• Evelyn & Leo: Make non-linear bicone (wiggly sides) 

The students have been meeting to work out the summer schedule. We are dividing into to groups:

(1) Paper writing: Julie, Alex M., Alex. P

• We need Amy to start an overleaf draft (she has the premium version that allows you to share with more than 1 person)

  • https://www.overleaf.com/5612932179jzykrkvztqnb

• In tomorrow's collaboration meeting (Now Fridays at 3pm) we are going to request discussion on an outline

(2) Adding wall time variable structure: Eliot, Evelyn, Leo (Alex P. will help oversee)

• In order to prepare group (2) Alex M. has been going over the edits he started and is discussing his code. We also are taking some time to catch them up on the current code -- ie how it runs, what all of the pieces involved do -- so that they can know how to test the code after edits are made, and where each essential piece may lie. We will be going over this for the next few days in good detail. 
• Group (2) will then be coming up with a scheduled time each day that they will be coworking on zoom. This will work the same as if they were sitting in an office together. They'll announce this time to everyone once it is set.

Alex P, Alex M, and I are going to push to rapidly finish this antenna database so that the others can create a new branch to work on the wall time variable with. This will leave us very busy for the next few days. 

Here's the presentation we gave for the April APS meeting today. Atttached is the power point of the presentation and a pdf version. 

Note that Pitzer usage over the weekend was much better! Maybe weekends are our best time to run? Today (Monday) has been a little bit worse. 

Over the past few days the following items have been accomplished:

• We got the jobs to stop canceling on OSC.
  • Issue was the destination command for the error and output files for the GPU jobs inside the GPU job script; we fixed it by taking it out.
    • This issue was causing jobs to fail. They are no longer failing, and the loop is working again.
• Set up a way to be able to automatically upload both the L, R, Theta plots and the Fitness Score plot to dropbox as we work.
  • Set up an email that automatically uploads to dropbox under the folder GP_Antennas/Updates/DailyFitnessScoreImages when you send an attachment with that email.
  • We added this command into the loop:
    mail -s "Subject" dropbox.2dwp1o@zapiermail.com < FScorePlot2D.png
    • It automatically has it upload an image when the loop finishes a generation. It's currently so that the file will be named whatever the subject of the email is so we can control that by whatever is after the -s command. It's currently set to not overwrite.
  • Still need to add the Veff plot to dropbox. 
• Made gitpush.sh and gitpushforce.sh
  • Each time one of us pushes changes to GitHub, it makes a permssion lock on the directories. From now on, after looking at what files are changed (using git status. Remember to make sure to add any data files missed to the .gitignore)users should do:
    • git add .
    • git commit -m "add message" or git commit --amend --no-edit
    • Then run: ./gitpush.sh if you made a full new commit, or use ./gitpushforce.sh if you amended this edit to the previous commit. They both exist in: /fs/project/PAS0654/BiconeEvolutionOSC
      • These files run the git push or git push -f commands and then immediately run a chmod to change permissions after. 
• Alex is starting the APS slides!
  • He should be sharing them shortly. I told him to look at Suren's from last year for guidance. 
• We have started working on adding the feature that pulls non-unique (ie repeat parameter) individuals from an antenna database. 
  • It's currently being developed on the Antenna_Database branch in git. 
  • Alex has started writing c++ code that reads in the csv file and appends unique items to it. 
  • Comments from Amy we need to remember: 
    • Doing "==" on doubles is bad!
    • We want this "likeliness" to be between 1 and 10% (start with 10%). 
      • Remember: is this difference between L stored and new L a fraction of the wavelength?

Meeting Attendance: Julie, Alex M., Alex P, Eliot, Leo, Evelyn, Mitchell, Ryan. 

Today is the deadline for confirming our attendance at the virtual APS meeting. Alex M. will be confirming and we will be orienting ourselves so that we can have some results. Here is our to-do list for today:
 

1. Create a method for automatically adding our plots to dropbox each day. 
   • The way we are going to do it is by adding the command: mail -s "Test Subject" someemailaddress@gmail.com < testScript
     • We will create a random email to use for this that will only be receiving emails from this command. 
     • We are trying to find and set up a client that as soon as this random email address receives an email, it uploads it to the Dropbox here. 
   • This is still being worked on. 
2. Start developing a database of old antenna runs and parameters, and pull data from that for any repeat dimensions. 
   • This should speed up some of our run time. 
   • We want to create a database of old antenna dimensions and their associated UAN files. That way, if we repeat an antenna geometry very closely, instead of re-submitting a job to produce the gain patterns, we can pull from one that has already been created.
   • This should limit:
     • The number of jobs we have to submit.
     • The number of GPU hours/minutes we have to request and wait for.
   • These should probably still be run through AraSim, however, since the energy could change from run to run.
3. Keep running and collecting data. 
   • Do we want to run and have data for 20ish generations with 10 individuals for APS?
     • We have already made some progress on this so we could likely make more generations for APS if we keep going. 
   • Do we want to instead switch to doing something like 20 individuals even if we get fewer generations completed?

Notes apropos developing our database for antenna info. This is how we are going to do it, and what we need to develop in our code:

1. Add all info to csv files.
   • The csv files will have columns: length, radius, theta, uan file directory and name (ie /file/location/database#/).
   •  # will be the row that this matches the L, R, Theta within the csv file
   • Inside /file/location/database#/ will be all the uan files for each frequency for these dimensions. 
     • We need an "if/else" statement in the bash script that tells the code to look in this csv file (read it in). 
       • IF the produced L, R, Theta match any of those that currently exist, save the 4th column (file location) to a variable (say we call it ExistingUan=/file/location/database#/). 
         • Then cd ExistingUan
         • Make a loop that:
           • does CP /file/location/database#/gen_individual_freq.uan BiconeEvolutionOSC/BiconeEvolution/current_antenna_evolution/XF_Loop/Evolutionary_Loop/Run_Outputs/$RunName/ (ie to the file name and location that XFintoAra.py reads. This will make sure that it has the data as if we actually completed the XF simulations for the antennas that had repeat sizes from our past runs (I mean past by runs we may have EVER done -- not past generations). 
             • We need to copy ALL of the frequency uan files for that individual into Run_Outputs/$RunName 
       • Else fill another line in that csv file with the L, R, Theta of this individual so we can keep it for later. 
         • mkdir the new directory /file/location/database#/
         • Add the name of this new directory (ie database#) to the 4th column on the csv file next to the correlating L, R, Theta.  
         • Then submit the XF job for this individual. 
         • Once it finished the XFSolver job and produces the .uan files (for each frequency), do a loop that:
           • cp individual_freq.uan /file/location/database#/gen_individual_freq.uan
             • Note that I wrote to cp individual_freq.uan instead of gen_individual_freq.uan. Note that XF produces it as individual_freq.uan, and it is changed to gen_individual_freq.uan when we move it into the Run_Outputs/$RunName directory.
             • So you can do it either (1) after it moves it to Run_Outputs/$RunName with the name gen_individual_freq.uan, or (2) before it moves the file with the name individual_freq.uan. 
2. We need to make a new directory that holds all of these database# directories and their uan files, and the csv file the "if/else" statement checks. 
   • We need to add this new directory to the .gitignore!

This is a general update. Most of our recent meetings have been to teach the students to use Git properly, or to answer questions on the training modules. 

Update

1. Git Update: I recently cleaned up our Git and GitHub so that we have a better way to develop this code. Once I cleaned it up the goal is to educate all of the students to properly use Git so we develop code correctly moving forward. The repository had become quite a mess. The students were constantly developing on a branch, and never merging it back to the main branch. This means that we never had a working copy that we could run on while edits were being made in parallel. I attempted to merge the edits to the main branch with the hopes of getting this functioning so we could evolve while working on the bottleneck function; however, it looks like someone with poor knowledge of Git edited the main branch and we had an abundance of merge conflicts. 

What I've done now is reset the repository so that all merge conflicts are resolved and a working copy exists on the main branch. I've also added a .gitignore that will ignore all data files. Git doesn't like these because (1) it hates when tracked files disappear -- ie when we delete trial run data, if someone accidentally pushed it then Git complains-- and (2) it means it we constantly are adding more untracked files (every single new file we make is untracked and then gets tracked, like the flag files). So, I've had it only track useful code. 

The goal once I train the other on Git is to move the place of the data files and make it it's own git repository (once we start doing fewer trial runs so we won't lose important data). If the others learn to check the status of files before pushing them, they can make sure they don't push useless trial data sets, and either (1) delete them, or (2) add them to the .gitignore.

As for now, we successfully have the working, run-able copy on the main branch now. When any more revisions are made (e.g. the talk of adding the wall time variable), we can branch and edit on that so we still run. 

2. Evolution/bottleneck update: We got the loop working to the point that it can run all the way through a generation without fatal problems. The only potential problem we can see right now is if we need to go backward in the savestate. We're working on making that not a problem--the solution is to better organize the output files and data, which we were interested in anyway. Right now we are running with 10 individuals in each generation. Alex got through 2 generations without a problem, so he'll keep running but now we're trying to make some of these more minor fixes so we may have to stop the run. Alex does have plots (they don't look great, but at least they're being made correctly without error). At the moment we are swimming in GPUs. We haven't had much trouble at all this week with getting them, so that's why we were able to get 2 generations done between around midnight last night and 2ish today; not sure how long that will persist, but Pitzer has had relatively little usage whenever we've looked lately.

Comments on the plots: We're discussing this now, we don't think the individuals should really be color coded/separate on the legend because it is not like they follow one individual across multiple generations; however, we do need to label the dotted line. Another thing we were considering is decreasing the penalty factor for when the cones are too big for the hole. If we get something with a much much better Veff but it's only slightly larger than the hole, it might be worth it to help that one stick around a little bit more.

3. Wall time Variable Update: Also, last week Carl and Alex talked about the variable wall time again. We decided the best idea is to start by implementing a variable wall time based on the cylindrical volume that he mentioned before, but in the future, we might also be able to check the time of each GPU job and then plot it against the dimensions of the cone/cylindrical volume to get a better estimate of the wall time for each job