I worked on fixing the AraSim issue today. I think I have it resolved. Essentially, I made a second set of flags that the loop will look for and use the first set of flags to tell the loop which error files to look at to look for the phrase "segmentation fault." If it sees that phrase, it resubmits the job. Otherwise, it makes the second flag. Once all of the flags have been made, the loop can move on. I tested it by manually putting the phrase inside one of the error files to make sure it would catch it and resubmit the job. 

I started a new run (partly to further test the fix, but primarily to get real data). It has the fixes we discussed last week at the GENETIS meeting (ex: mutations shouldn't repeat anymore).

I'm going to be out tomorrow and Thursday so someone needs to pick up my run. I'll make sure I open permissions tonight so that anyone can run it tomorrow. If you have questions you can slack me, but I probably won't be able to respond til super late.

If there are issues with AraSim jobs failing without resubmitting, see the instructions below.