Reducing the size of a volume in Galaxy Cloud

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

Reducing the size of a volume in Galaxy Cloud

clarkd3

Hi,

I wish to remove most of the mnt/galaxyIndices/genomes files so I can have a smaller volume that contains those files and save money.

I am new to this system.  I have set up a Galaxy Cloudman account and used the tools on the web interface, and I can ssh the server using Ubuntu.  So far, so good.

I tried the first approach here twice https://wiki.galaxyproject.org/CloudMan/CustomizeGalaxyCloud but the system hangs (>24h).  I can’t follow the “manual” instructions on this page and I’m not sure it would result in a smaller volume anyway.

 

So, I was trying to follow Enis’ steps from this thread to reduce the size of the galaxyIndices volume:

http://dev.list.galaxyproject.org/Reducing-costs-in-Cloud-Galaxy-td4483448.html

 

His instructions are:

“1. Start a new Galaxy cluster (ie, one you can easily delete later)

2. ssh into the master instance and delete whatever genomes you don't need/want (these are all located under /mnt/galaxyIndices)

3. Create a new EBS volume of size that'll fit whatever's left on the original volume, attach it and mount it

4. Copy over the data from the original volume to the new one while keeping the directory structure the same (rsync is probably the best tool for this)

5. Unmount & detach the new volume; create a snapshot from it

6. For the cluster you want to keep around (while it is terminated), edit persistent_data.yaml in it's bucket on S3 and replace the existing snap ID for the galaxyIndices with the snapshot ID you got in the previous step

7. Start that cluster and you should have a file system from the new snapshot mounted.

8. Terminate & delete the cluster you created in step 1

 

Here are my questions to help me to get through step 4:

1)      Step 1: Is a “cluster” the same thing as an “instance”?

2)      Step 2: I deleted the directories for individual genomes using rm –rf .  Is that the correct approach?

3)      Step 3: Do I add the newly created EBS volume to the same instance where I deleted the genomes? Or is it added the instance I want to keep?

4)      Step 3: I can see how to attach this newly created volume using the AWS EC2 management console, but how do I mount it? (and unmount it in Step 5?)

5)      Step 4: what is the syntax for the rsync (or cp) command to copy directories/files from one volume to another volume (within the same instance, or in if they are in different instances)?

 

Any help would be gratefully received!

Denise

 

 


___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
Reply | Threaded
Open this post in threaded view
|

Re: Reducing the size of a volume in Galaxy Cloud

Enis Afgan-2
Hi Denise, 
Although the instructions you found can still be used, a couple of steps can be updated now so I'll to that now and hopefully answer your questions in the process.

Updated instructions:
1. Start a brand new Galaxy CloudMan cluster/instance (this will be only a temporary cluster needed for the duration of the following steps)
2. ssh into the master instance and delete whatever genomes you don't need/want (these are all located under /mnt/galaxyIndices; using 'rm -rf <dir name>' is the command to use)
3. From CloudMan's Admin page, add a new file system of type 'new volume' and desired size (ie, size of data on /mnt/galaxyIndices after you've deleted what you didn't want). Give this file system a name 'indices'.
4. ssh into the master instance and copy over the data from /mnt/galaxyIndices to /mnt/indices using command: rsync -avz /mnt/galaxyIndices/ /mnt/indices/. (do this as 'galaxy' user)
5. Unmount the new 'indices' file system using command: sudo umount /mnt/indices
6. From the AWS console, create a snapshot of the volume for the 'indices' file system. You can retrieve the volume ID from the file system 'details' on the CloudMan Admin page
7. For the cluster you want to keep around (while it is terminated), edit persistent_data.yaml in it's bucket on S3 (you can get the bucket name from the CloudMan Admin page - it will look like so cm-<hash>) and replace the existing snap ID for the galaxyIndices with the snapshot ID you got in the previous step
8. Start that cluster and you should have a file system from the new snapshot mounted
9. Terminate & delete the cluster you created in step 1



 

Here are my questions to help me to get through step 4:

1)      Step 1: Is a “cluster” the same thing as an “instance”?

Yes 

2)      Step 2: I deleted the directories for individual genomes using rm –rf .  Is that the correct approach?

Yes 

3)      Step 3: Do I add the newly created EBS volume to the same instance where I deleted the genomes? Or is it added the instance I want to keep?

 You add the new file system (ie, EBS volume) to the newly created cluster (and you also delete the extra genomes on that same cluster).

4)      Step 3: I can see how to attach this newly created volume using the AWS EC2 management console, but how do I mount it? (and unmount it in Step 5?)

With the updated instructions above, mounting the file system by hand is no longer necessary.  

5)      Step 4: what is the syntax for the rsync (or cp) command to copy directories/files from one volume to another volume (within the same instance, or in if they are in different instances)?

I included in the command in the instructions now.


Hope this helps. Let us know if you have any trouble or if you have any more questions,
Enis

___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/