Pegasus / Montage workflow on Amazon Web Services

By , 2014年9月16日 8:06 下午

I took some notes while going through Mats Rynge’s tutorial on “Pegasus / Montage workflow on Amazon Web Services“. The tutorial if officially available at the following URL, but the content is far from complete. I managed to finished this tutorial, and thought that my experience might be valuable for someone else out there in the dark.

https://confluence.pegasus.isi.edu/display/pegasus/2013+-+Montage+workflow+using+Amazon+Web+Services

Step 1. Launch an EC2 instance

In the Oregon region, launch an EC2 instance with AMI ami-f4e47cc4. It is recommend that you use the same security group for all EC2 instance to be launched in the Condor cluster. Also, in the security group, all communication between all EC2 instances using the same security group.

SSH into the instance using the following command:

ssh -i keypair.pem montage@IP
ssh-keygen
cd .ssh
cat id_rsa.pub >> authorized_keys
cd ~

Step 2. Configure Condor

Edit /etc/condor/config.d/20_security.conf,update ALLOW_WRITE and ALLOW_READ to the IP range of your VPC. For example, if the IP range of your VPC is 172.31.0.0/16, then you can set “ALLOW_WRITE = 172.31.*” and “ALLOW_READ = 172.31.*”. Then you need to restart Condor with the following command:

sudo service condor restart

Step 3. Create a Montage workflow

mkdir workfow
cd workflow
mDAG 2mass j M17 0.5 0.5 0.0002777778 . file://$PWD file://$PWD/inputs
generate-montage-replica-catalog 

Step 4. Pegasus Related

cd ~/etc
cp ../workflow/replica.catalog .
cp ../workflow/dag.xml .

Step 5. Update site.xml with the following content



    
        
            
        
        
            
        
        /home/montage/.ssh/id_rsa
    
    
        
            
        
        condor
        vanilla
        /opt/montage/v3.3
    

Step 6. Plan the workflow

pegasus-plan --conf pegasus.conf --dax dag.xml

Step 7. Run the workflow

pegasus-run  /home/montage/etc/montage/pegasus/montage/run0001

Step 8. Monitor the workflow

pegasus-status -l /home/montage/etc/montage/pegasus/montage/run0001
condor_status
condor_q
tail montage/pegasus/montage/run0001/jobstate.log

The last command is very nice in that you can see what is currently being run. Please note that you need to replace the path with the actual path given to you by Pegasus.

That’s all. After spending weeks searching over the Internet, everything now seems to be simple.

2 Responses to “Pegasus / Montage workflow on Amazon Web Services”

  1. dk说道:

    Hi, Qingye. I successfully installed condor and pegasus, and finished the command “mDAG 2mass j M17 0.5 0.5 0.0002777778 . file://$PWD file://$PWD/inputs”. But, I don’t quite understand the step “generate-montage-replica-catalog”, since I found it is not a command. So, I want to ask how to generate-montage-replica-catalog.

  2. dk说道:

    I copied pegasusrc from some template files and successfully run the montage job, but after completing 18 tasks, the job finally failed. Do you know what’s the problem? Do I need to keep connecting to the Internet while runing the job?

Leave a Reply

Panorama Theme by Themocracy