ELOG How-To

Updates and Results Talks and Posters Advice Ideas Important Figures Write-Ups Outreach How-To Funding Opportunities GENETIS

Place to document instructions for how to do things

Fields marked with * are required
Entry time:	Fri Nov 9 00:44:09 2018
Author*:
Subject:
Project:
<p>Brian had to move ~7 TB of data from the IceCube data warehouse to OSC.</p> <p>To do this, he used the gridftp software. The advantage is that griftp is optimized for large file transfers, and will manage data transfer better than something like scp or rsync.</p> <p>Note that this utilizes the gridftp software installed on OSC, but doesn't formally use the globus end point described here: <a href="https://www.osc.edu/resources/getting_started/howto/howto_transfer_files_using_globus_connect">https://www.osc.edu/resources/getting_started/howto/howto_transfer_files_using_globus_connect</a>. This is because IceCube doesn't have a formal globus endpoint to connect too. The formal globus endpoint would have been even easier if it was available, but oh well...</p> <p><u>Setup goes as follows:</u></p> <ol> <li>Follow the IceCube instructions for getting an OSG certificate <ol> <li>Go through CILogon (https://cilogon.org/) and generate and download a certificate</li> </ol> </li> <li>Install your certificate on the IceCube machines <ol> <li>Move your certificate to the following place on IceCube: <code>./globus/usercred.p12</code></li> <li>Change the permissions on this certificate: <code>chmod 600 ./globus/usercred.p12</code></li> <li>Get your "subject" to this key: <code>openssl pkcs12 -in .globus/usercred.p12 -nokeys \| grep subject</code></li> <li>Copy the subject line into your IceCube LDAP account <ol> <li>Select "Edit your profile"</li> <li>Enter IceCube credentials</li> <li>Paste the subject into the "x509 Subject DN" box</li> </ol> </li> </ol> </li> <li>Install your certificates on the OSC machines <ol> <li>Follow the same instructions as for IceCube to install the globus credentials, but you don't need to do the IceCube LDAP part</li> </ol> </li> </ol> <p><u>How to actually make a transfer:</u></p> <ol> <li>Initialize a proxy certificate on OSC: grid-proxy-init -bits 1024</li> <li>Use globus-url-copy to move a file, for example: <code>globus-url-copy -r gsiftp://gridftp.icecube.wisc.edu/data/wipac/ARA/2016/unblinded/L1/ARA03/ 2016/ &</code> <ol> <li>I'm using the command "globus-url-copy"</li> <li>"-r" says to transfer recursively</li> <li>"gsiftp://gridftp.icecube.wisc.edu/data/wipac/ARA/2016/ublinded/L1/ARA03/" is the entire directory I'm trying to copy</li> <li>"2016/" is the directory I'm copying them to</li> <li>"&" says do this in the background once launched</li> </ol> </li> <li>Note that it's kind of syntatically picky: <ol> <li>To copy a directory, the source path name must end in "/"</li> <li>To copy a directory, the destination path name must also end in "/"</li> </ol> </li> </ol> <p> </p> <p> </p> Encoding: HTML ELCode plain Suppress Email notification Resubmit as new entry

Attachment 1:
Drop attachments here...

ELOG V3.1.5-fc6679b