Updates and Results
Talks and Posters
Advice
Ideas
Important Figures
Write-Ups
Outreach
How-To
Funding Opportunities
GENETIS
GENETIS
Draft saved at 00:00:00
Fields marked with
*
are required
Entry time:
Tue Feb 3 15:11:18 2026
Author
*
:
Subject
*
:
<p style="text-align:center"><span style="font-size:20px"><u><strong>GENETIS Loop Work: Fixing Consol Errors By Julie Rolla</strong></u></span></p> <p><strong><span style="font-size:14px">1. Python programs are not running: </span></strong><span style="font-size:14px">I think either (1) the user isn't module loading python, or (2) they are loading the wrong version of Python (remember there's a discrepancy for Python2, and Python3). </span></p> <p><span style="font-size:14px"><em>Solution:</em> I added a line "</span>module load python/3.6-conda5.2"<span style="font-size:14px"> </span><span style="font-size:14px">in our araenv.sh script </span><span style="font-size:14px">to address the issue. All of the users are sourcing the araenv.sh script in their .bashrc when they login. It's in the directory: </span>/fs/project/PAS0654/BiconeEvolutionOSC<span style="font-size:14px">. I also added a line in the XF_Loop.sh script (at the top in the "Initialization of Variables" section) that sources araenv.sh just in case the user hasn't put that in their .bashrc. The line reads: </span>source /fs/project/PAS0654/BiconeEvolutionOSC/araenv.sh</p> <p><em><span style="font-size:14px">Error message fixed:</span></em></p> <blockquote> <pre> <span style="font-size:10px"> 0 Data successfully read. Data successfully written. Fitness scores successfully written. Fitness function concluded. rm: cannot remove 'runData.csv': No such file or directory Unable to parse the pattern ^CTraceback (most recent call last): File "LRPlot.py", line 99, in <module> plotLR(generations, lengths, radii, g.numGens, g.destination) File "LRPlot.py", line 45, in plotLR plt.show() File "/usr/lib64/python2.7/site-packages/matplotlib/pyplot.py", line 143, in show _show(*args, **kw) File "/usr/lib64/python2.7/site-packages/matplotlib/backend_bases.py", line 108, in __call__ self.mainloop() File "/usr/lib64/python2.7/site-packages/matplotlib/backends/backend_gtk.py", line 83, in mainloop gtk.main() KeyboardInterrupt</span> </pre> </blockquote> <p> </p> <p><span style="font-size:14px"><strong>2. rm runData.csv error: </strong>The following error was printed to consol</span></p> <blockquote> <pre> <span style="font-size:10px">rm: cannot remove 'runData.csv': No such file or directory</span> </pre> </blockquote> <p> </p> <p>I found these line in section (E) of the loop.</p> <blockquote> <pre> <span style="font-size:10px">cd "$WorkingDir" rm runData.csv python gensData.py 0 cd Antenna_Performance_Metric python LRPlot.py "$WorkingDir" "$WorkingDir"/Run_Outputs/$RunName 1 $NPOP cd .. # Note: gensData.py floats around in the main dir until it is moved to </span> </pre> </blockquote> <p> I think this error is fine. This is how it works: (Step 1) removes runData.csv, if it exists. (Step 2) Runs gensData.py, which creates a new runData.csv. We wanted to delete it before the .py program was run at the start of the run, because gensData.py appends data to runData.csv, if it already exists. (alternatively if runData.csv doesn't exists it creates it-- which is what we want in this case). That's also why we don't delete it later in the "looping part" of XF_Scripts.sh (part G). We want it to append at this point, so we only delete it in the very beginning to make sure it doesn't append to an old run.</p> <p> </p> <p><span style="font-size:14px"><strong>3. cp: cannot stat 'Antenna_Performance_Metric/1.uan': No such file or directory (similarly for 2.uan, 3.uan...etc)</strong>: This error exists just as the fitness score is being calculated -- ie part E. Here is the print:</span></p> <blockquote> <pre> <span style="font-size:10px">cp: cannot stat 'Antenna_Performance_Metric/1.uan': No such file or directory cp: cannot stat 'Antenna_Performance_Metric/2.uan': No such file or directory Congrats on getting a fitness score! File "FScorePlot.py", line 1 -*- coding: utf-8 -*- ^ IndentationError: unexpected indent</span></pre> </blockquote> <p> </p> <p>Note that we have a few errors here. Right now I am directly addressing the error with the "cp" command. It looks like these files are actually being saved as gen_individual#.uan. For example 0_1.uan. The current line in part E says:</p> <p> </p> <blockquote> <p><span style="font-size:10px">for i in `seq 1 $NPOP`</span></p> <p><span style="font-size:10px">do</span></p> <p><span style="font-size:10px"> #Remove if plotting software doesnt need </span></p> <p><span style="font-size:10px"> #cp data/$i.uan ${i}uan.csv </span></p> <p><span style="font-size:10px"> cp Antenna_Performance_Metric/$i.uan "$WorkingDir"/Run_Outputs/$RunName/${i}.uan</span></p> <p><span style="font-size:10px">done</span></p> </blockquote> <p> </p> <p><span style="font-size:12px">However, this is not how these files are being saved. I corrected the first iteration of the loop (ie Part E for generation 0) by editing the line to say:</span></p> <p> </p> <blockquote> <p><span style="font-size:10px">for i in `seq 1 $NPOP`</span></p> <p><span style="font-size:10px">do</span></p> <p><span style="font-size:10px"> #Remove if plotting software doesnt need </span></p> <p><span style="font-size:10px"> #cp data/$i.uan ${i}uan.csv </span></p> <p><span style="font-size:10px"> cp Antenna_Performance_Metric/0_$i.uan "$WorkingDir"/Run_Outputs/$RunName/0_${i}.uan</span></p> <p><span style="font-size:10px">done</span></p> </blockquote> <p> </p> <p>And for the looping part of XF_Loop.sh (ie part G) it has been edited to say:</p> <p> </p> <blockquote> <p><span style="font-size:10px">for i in `seq 1 $NPOP`</span></p> <p><span style="font-size:10px"> do</span></p> <p><span style="font-size:10px"> # Gens data used to create a .csv file for the uan file for gain plotting </span></p> <p><span style="font-size:10px"> # cp Antenna_Performance_Metric/$i.uan ${i}uan.csv </span></p> <p><span style="font-size:10px"> cp Antenna_Performance_Metric/${gen}_${i}.uan "$WorkingDir"/Run_Outputs/$RunName/${gen}_${i}.uan</span></p> <p><span style="font-size:10px"> done</span></p> </blockquote> <p> </p> <p><span style="font-size:14px"><strong>4. Made Number of Neutrinos thrown (NNU) in AraSim's setup.txt file a variable: </strong>I've added the following line in part D -- as well as the looping section in part G. </span></p> <p>These lines replace the number of neutrinos thrown in our setup.txt AraSim file with what ever number you assigned NNT at the top of XF_Loop.sh in the "variables" section. "setup_dummy.txt" is a copy of setup.txt that has NNU=num_nnu instead of a number. NNU is the number of neutrinos thrown (as seen in setup.txt). This "sed" line finds every instance of num_nnu in setup_dummy.txt and replaces it with $NNT (the number you assigned NNT in the variable section at the top of XF_Loop.sh). It then pipes this information into setup.txt (and overwrites the last setup.txt file allowing the number of neutrinos thrown to be as variable changed at the top of this script instead of manually changing it in setup.txt each time. Command works the following way: sed "s/oldword/newwordReplacingOldword/" path/to/filewiththisword.txt > path/to/fileWeAreOverwriting.txt. Both setup.txt and setup_dummy.txt can be found in the directory: /fs/project/PAS0654/BiconeEvolutionOSC/AraSim/</p> <blockquote> <p><span style="font-size:10px">cd "$AraSimExec"</span></p> <p><span style="font-size:10px">for i in `seq 1 $NPOP`</span></p> <p><span style="font-size:10px">do</span></p> <p><span style="font-size:10px">#This next line replaces the number of neutrinos thrown in our setup.txt AraSim file with what ever number you assigned NNT at the top of this program. setup_dummy.txt is a copy of setup.txt that has \</span></p> <p><span style="font-size:10px">NNU=num_nnu (NNU is the number of neutrinos thrown. This line finds every instance of num_nnu in setup_dummy.txt and replaces it with $NNT (the number you assigned NNT above). It then pipes this infor\</span></p> <p><span style="font-size:10px">mation into setup.txt (and overwrites the last setup.txt file allowing the number of neutrinos thrown to be as variable changed at the top of this script instead of manually changing it in setup.txt e\</span></p> <p><span style="font-size:10px">ach time. Command works the following way: sed "s/oldword/newwordReplacingOldword/" path/to/filewiththisword.txt > path/to/fileWeAreOverwriting.txt </span></p> <p><span style="font-size:10px"> sed "s/num_nnu/$NNT/" /fs/project/PAS0654/BiconeEvolutionOSC/AraSim/setup_dummy.txt > /fs/project/PAS0654/BiconeEvolutionOSC/AraSim/setup.txt</span></p> <p><span style="font-size:10px"> #We will want to call a job here to do what this AraSim call is doing so it can run in parallel </span></p> <p><span style="font-size:10px"> cd $WorkingDir</span></p> <p><span style="font-size:10px"> qsub -v num=$i AraSimCall.sh</span></p> <p><span style="font-size:10px"> rm outputs/*.root</span></p> <p><span style="font-size:10px">done</span></p> </blockquote> <p> </p> <p><span style="font-size:14px"><strong>5. rm cannot remove 'simulation_PEC.xmacro': </strong>No such file or directory error appearing in generations after the first (ie not seen in gen 0, but seen in gen 1+).</span></p> <blockquote> <pre> <span style="font-size:10px">Statistics initialized. Tournament parents selected. Tournament complete. rm: cannot remove 'simulation_PEC.xmacro': No such file or directory</span></pre> </blockquote> <p>I found that it tries to remove it twice. See here:</p> <blockquote> <p><span style="font-size:10px">######## Loop (G) ###################################################################################### </span><span style="font-size:10px"> </span></p> <p><span style="font-size:10px"># 1. Does steps A-F for each generation </span><span style="font-size:10px"> </span></p> <p><span style="font-size:10px">########################################################################################################### </span></p> <p><span style="font-size:10px">for gen in `seq 1 $TotalGens`</span></p> <p><span style="font-size:10px">do</span></p> <p><span style="font-size:10px"># used for fixed number of generations </span></p> <p><span style="font-size:10px">#gen=0; while [ `cat highfive.txt` -eq 0 ]; do (( gen++ )) </span></p> <p><span style="font-size:10px"># use for runs until convergence </span></p> <p><span style="font-size:10px">#should the line below be double indented? </span></p> <p><span style="font-size:10px"> read -p "Starting generation ${gen}. Press any key to continue... " -n1 -s</span></p> <p><span style="font-size:10px">#Part A </span></p> <p><span style="font-size:10px"> <span style="background-color:#FFFF00"> cd $XmacrosDir </span></span></p> <p><span style="font-size:10px"><span style="background-color:#FFFF00"> rm simulation_PEC.xmacro </span></span></p> <p><span style="font-size:10px"> cd "$WorkingDir"</span></p> <p><span style="font-size:10px"> ./roulette_algorithm.exe cont $NPOP</span></p> <p><span style="font-size:10px"> cp generationDNA.csv Run_Outputs/$RunName/${gen}_generationDNA.csv</span></p> <p><span style="font-size:10px"> ##chmod -R 777 $WorkingDir </span></p> <p><span style="font-size:10px">#we can make a function that does this with the flag $gen </span></p> <p><span style="font-size:10px">#doing this in part A would mean setting $gen=0 </span></p> <p><span style="font-size:10px">#Part B </span></p> <p><span style="font-size:10px"> # First, remove the old .xmacro files </span></p> <p><span style="font-size:10px"> #when do that, we end up making the files only readable; we should just overwrite them </span></p> <p><span style="font-size:10px"> #alternatively, we can just set them as rwe when the script makes them </span></p> <p><span style="font-size:10px"> cd $XmacrosDir</span></p> <p><span style="font-size:10px"> #I'm commenting out the next two lines. They are written to below (around lines 154-170) </span></p> <p><span style="font-size:10px"> #If we keep them this way then the files have restricted permissions </span></p> <p><span style="font-size:10px"> <span style="background-color:#FFFF00">rm output.xmacro</span></span></p> <p><span style="font-size:10px"><span style="background-color:#FFFF00"> rm simulation_PEC.xmacro</span></span></p> </blockquote> <p> </p> <p>I have commented the first removal of simulation_PEC.xmacro. If there are any issues with it running, we can go ahead and comment it back in; however, I suspect this is not a necessary line to have added in. A run has not been performed since these edits. So, once it has been tested, we can delete the comments out:</p> <p> </p> <blockquote> <p><span style="font-size:10px">#Part A </span></p> <p><span style="font-size:10px"> #I think these next two lines can be deleted. Its repeated again just below...11/30/19 comment by Julie #cd $XmacrosDir #rm simulation_PEC.xmacro </span></p> <p><span style="font-size:10px"> cd "$WorkingDir"</span></p> <p><span style="font-size:10px"> ./roulette_algorithm.exe cont $NPOP</span></p> <p><span style="font-size:10px"> cp generationDNA.csv Run_Outputs/$RunName/${gen}_generationDNA.csv</span></p> </blockquote> <p><strong><span style="font-size:14px">6. Python Error from FScorePlot.py: The error can be seen in the line below.</span></strong><span style="font-size:10px"><strong> </strong> </span></p> <blockquote> <p><span style="font-size:10px">Congrats on getting a fitness score!</span></p> <p><span style="font-size:10px"> File "FScorePlot.py", line 1</span></p> <p><span style="font-size:10px"> -*- coding: utf-8 -*-</span></p> <p><span style="font-size:10px"> ^</span></p> <p><span style="font-size:10px">IndentationError: unexpected indent</span></p> <p><span style="font-size:10px">Congrats on getting some nice plots!</span></p> </blockquote> <p> </p> <p><span style="font-size:12px">I've commented out this strange line at the start of this python program: </span><span style="font-size:10px"> -*- coding: utf-8 -*-. </span><span style="font-size:12px">This fixed the immediate error, but threw another one when I tried to run it outside of the loop with the command </span></p> <p> </p> <blockquote> <p><span style="font-size:10px">python FScorePlot.py /fs/project/PAS0654/BiconeEvolutionOSC/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop/Run_Outputs/Julie_11_30/ /fs/project/PAS0654/BiconeEvolutionOSC/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop/Run_Outputs/Julie_11_30/ 2</span></p> </blockquote> <p> </p> <p><span style="font-size:12px">The next error it threw can be seen below.</span><span style="font-size:10px"> </span></p> <p><span style="font-size:10px"> </span></p> <blockquote> <p><span style="font-size:10px">Traceback (most recent call last):</span></p> <p><span style="font-size:10px"> File "FScorePlot.py", line 79, in <module></span></p> <p><span style="font-size:10px"> fScores.append(fScorei[2:])</span></p> <p><span style="font-size:10px">AttributeError: 'numpy.ndarray' object has no attribute 'append'</span></p> </blockquote> <p> <strong> This is Alex P. I edited the append() and changed it to np.append(fScores, fScorei[2:]) on line 79 in FScorePlot.py</strong></p> <p> </p> <p><span style="font-size:14px"><strong>7. rm: cannot remove 'outputs/*.root': No such file or directory: </strong></span>This error comes up right after the AraSim job is submitted in part D. </p> <blockquote> <pre> <span style="font-size:10px"><a href="http://1011007.pitzer-batch.ten.osc.edu/" rel="noopener noreferrer" target="_blank">1011007.pitzer-batch.ten.osc.edu</a> rm: cannot remove 'outputs/*.root': No such file or directory Waiting for AraSim jobs to finish...</span></pre> </blockquote> <p> </p> <p>It's definitely coming from the lines:</p> <p> </p> <blockquote> <pre> <span style="font-size:10px">for i in `seq 1 $NPOP` do #We will want to call a job here to do what this AraSim call is doing so it can run in parallel cd $WorkingDir qsub -v num=$i AraSimCall.sh rm outputs/*.rootdone</span></pre> </blockquote> <p>We are currently in $WorkingDir at this point-- which is: WorkingDir= /fs/project/PAS0654/BiconeEvolutionOSC/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop. There is not directory called "outputs" in this location. This is why we are getting an error.<strong>Can someone explain what .root files we are trying to remove, and why? I cannot proceed unless someone makes this clear to me, as I do not know what files we are looking to remove (and where they exist/when they are created).</strong></p> <p> </p> <p>Immediately after that, I get the following error:</p> <blockquote> <pre> <span style="font-size:10px">Waiting for AraSim jobs to finish... rm: remove write-protected regular file 'AraSimFlags/1.txt'? y rm: remove write-protected regular file 'AraSimFlags/2.txt'? y mv: cannot stat '*.root': No such file or directory</span></pre> </blockquote> <p> </p> <p>It looks like it is coming from the following line:</p> <blockquote> <pre> <span style="font-size:10px">cd "$WorkingDir"/Antenna_Performance_Metric mv *.root "$WorkingDir/Run_Outputs/$RunName/RootFilesGen0/"</span></pre> </blockquote> <p>However-- again-- it looks like it's trying to move .root files from the directory: /fs/project/PAS0654/BiconeEvolutionOSC/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop/Antenna_Performance_Metric and put them into: /fs/project/PAS0654/BiconeEvolutionOSC/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop/Run_Outputs/$RunName/RootFilesGen0/</p> <p>It can't move these, because they do not exist where it is trying to take them from. <strong>Similarly, the question in bold above would help me with this a lot. From what I can see, we are only utilizing an .txt file output from Arasim. Does Arasim output this to .txt, or are we somehow converting from a .root to a .txt?</strong> </p> <p><span style="font-size:12px"><span style="background-color:#FFA07A"> </span><strong><span style="background-color:#FFA07A">This error has<u> not yet </u>been corrected!!!!</span></strong></span><span style="font-size:10px"><span style="background-color:#FFA07A"> </span> </span></p> <p> </p> <p><em>Cade's comment here:</em></p> <p><em>The .root files live in BiconeEvolutionOSC/AraSim/outputs/ folder. Thus, we were looking at the wrong directory. They were being removed because it is a giant file (GB size) and are created everytime AraSim runs. It just overwrites itself everytime though so it doesnt add up too much. We can have them deleted later if needed. It's our choice on whether we take it out completely, or redirect to delete the .root files in the correct directory.</em></p> <p> </p> <p><span style="font-size:14px"><strong>8. New errors found after running with these corrections: <span style="background-color:#FF8C00">These need to be fixed!</span></strong></span></p> <blockquote> <p><span style="font-size:10px">cp: cannot stat ‘Antenna_Performance_Metric/0_1.uan’: No such file or directory</span></p> <p><span style="font-size:10px">cp: cannot stat ‘Antenna_Performance_Metric/0_2.uan’: No such file or directory</span></p> <p><span style="font-size:10px">Congrats on getting a fitness score!</span></p> <p><span style="font-size:10px">Traceback (most recent call last):</span></p> <p><span style="font-size:10px"> File "FScorePlot.py", line 86, in <module></span></p> <p><span style="font-size:10px"> Gen = np.zeros(fScores.shape[0]*fScores.shape[1])</span></p> </blockquote> <p>I found the cp error. It looks like I was incorrect in #3 about this cp command for the .uan files. I moved all of the currently existing .uan files into a new directory in</p> <blockquote> <p> /fs/project/PAS0654/BiconeEvolutionOSC/BiconeEvolution/current_antenna_evo_build/XF_Loop/Evolutionary_Loop/Antenna_Performance_Metric/UAN</p> </blockquote> <p>With all of them out of the Antenna_Performance_Metric directory, I did a run from scratch to see what it creates. Here's the outcome: So on generation 1, for individual one its going</p> <p>1_1.uan<br /> 1_2.uan<br /> 1_3.uan<br /> <br /> ie-- > individual_frequencyStep.uan<br /> <br /> So once it hits all 60 frequencies for individual 1, it does all 60 for individual 2 by saying:<br /> 2_1.uan<br /> 2_2.uan<br /> 2_3.uan</p> <p>...etc.This isn't great because it never actually cares about generation number. In section E (so generation 0) we currently have it as:</p> <blockquote> <pre> <span style="font-size:10px">for i in `seq 1 $NPOP` do #Remove if plotting software doesnt need #cp data/$i.uan ${i}uan.csv cp Antenna_Performance_Metric/0_$i.uan "$WorkingDir"/Run_Outputs/$RunName/0_${i}.uan done</span></pre> </blockquote> <p>and in section G of the looping part (gens 1+) we have:</p> <blockquote> <pre> <span style="font-size:10px"> for i in `seq 1 $NPOP` do # Gens data used to create a .csv file for the uan file for gain plotting # cp Antenna_Performance_Metric/$i.uan ${i}uan.csv cp Antenna_Performance_Metric/${gen}_${i}.uan "$WorkingDir"/Run_Outputs/$RunName/${gen}_${i}.uan done</span></pre> </blockquote> <p>This doesn't seem like it's being output from XF the way we thought it is. I think the easiest way to do this is to edit the xmacros to have it output like:</p> <blockquote> <pre> <span style="font-size:10px">generation_individual_frequency.uan</span></pre> </blockquote> <p>and thus</p> <blockquote> <pre> <span style="font-size:10px">cp Antenna_Performance_Metric/${gen}_${i}_${frequency}.uan</span></pre> </blockquote> <p> If so, we'd have to make a new variable in the bash script for $frequency to go from 1-60, and have it loop moving all of those. I will be looking into this with Cade tomorrow 12/3/19 (since he is the xmacro script expert). Stay tuned!</p> <p> </p> <p> </p> <p>Phew....thanks for sticking with me on this long update.</p>
Encoding
:
HTML
ELCode
plain
Suppress Email notification
Attachment 1:
Drop attachments here...
Draft saved at 00:00:00
ELOG V3.1.5-fc6679b