婉兮清扬

Pegasus / Montage workflow on Amazon Web Services

发表时间：2014-09-16 20:06:25

---

I took some notes while going through Mats Rynge's tutorial on "Pegasus / Montage workflow on Amazon Web Services". The tutorial if officially available at the following URL, but the content is far from complete. I managed to finished this tutorial, and thought that my experience might be valuable for someone else out there in the dark.

https://confluence.pegasus.isi.edu/display/pegasus/2013+-+Montage+workflow+using+Amazon+Web+Services

Step 1. Launch an EC2 instance

In the Oregon region, launch an EC2 instance with AMI ami-f4e47cc4. It is recommend that you use the same security group for all EC2 instance to be launched in the Condor cluster. Also, in the security group, all communication between all EC2 instances using the same security group.

SSH into the instance using the following command:

ssh -i keypair.pem montage@IP

ssh-keygen

cd .ssh

cat id_rsa.pub >> authorized_keys

cd ~

Step 2. Configure Condor

Edit /etc/condor/config.d/20_security.conf，update ALLOW_WRITE and ALLOW_READ to the IP range of your VPC. For example, if the IP range of your VPC is 172.31.0.0/16, then you can set "ALLOW_WRITE = 172.31.*" and "ALLOW_READ = 172.31.*". Then you need to restart Condor with the following command:

sudo service condor restart

Step 3. Create a Montage workflow

mkdir workfow

cd workflow

mDAG 2mass j M17 0.5 0.5 0.0002777778 . file://$PWD file://$PWD/inputs

generate-montage-replica-catalog 

Step 4. Pegasus Related

cd ~/etc

cp ../workflow/replica.catalog .

cp ../workflow/dag.xml .

Step 5. Update site.xml with the following content

    

        

            

        

        

            

        

        /home/montage/.ssh/id_rsa

    

    

        

            

        

        condor

        vanilla

        /opt/montage/v3.3

    

Step 6. Plan the workflow

pegasus-plan --conf pegasus.conf --dax dag.xml

Step 7. Run the workflow

pegasus-run  /home/montage/etc/montage/pegasus/montage/run0001

Step 8. Monitor the workflow

pegasus-status -l /home/montage/etc/montage/pegasus/montage/run0001

condor_status

condor_q

tail montage/pegasus/montage/run0001/jobstate.log

The last command is very nice in that you can see what is currently being run. Please note that you need to replace the path with the actual path given to you by Pegasus.

That's all. After spending weeks searching over the Internet, everything now seems to be simple.

上一篇

下一篇