Problem with cleanup.py

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

Problem with cleanup.py

georgeweingart
Hello,

We are incurring a problem with cleanup.py:

We have already run two cycles of cleanup.py (1-6) with a cut off of 30 days,  we see in the logs that many datasets are deleted,  but when we do df -kh  to check the file system, we see creeping up utilization of space - no relinquishing of any space.

We are running in a docker environment and our cleanup job looks as follows:
================================================================

#!/bin/bash

export GALAXY_VIRTUAL_ENV=/galaxy_venv

source $GALAXY_VIRTUAL_ENV/bin/activate

export script="/galaxy-central/scripts/cleanup_datasets/cleanup_datasets.py" 

export config="/etc/galaxy/galaxy.ini"

export logdir="/export/galaxy-central/logs"

export days=30

export num=$1

nohup $script $config -d $days -$num -r -f >> $logdir/cleanup$num.log &

================================================================


and we run it with ./run_cleanup.sh 1

or 2, 3 etc.


Any ideas ?


Thanks!

George Weingart PhD

Huttenhower Lab

Harvard School of Public Health


___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/
Reply | Threaded
Open this post in threaded view
|

Re: Problem with cleanup.py

Jelle Scholtalbers
Hi George,

on our instance I have noticed that the tmp (new_file_path) was taking up quite some space and files were only deletable after restarting the web / uwsgi process.

FYI a while ago I wrote this script which should ease the cronjob setup a bit (does not cleanup tmp though): https://github.com/galaxyproject/galaxy/blob/dev/scripts/maintenance.sh

Cheers,
Jelle

On Tue, Oct 30, 2018 at 5:04 AM George Weingart <[hidden email]> wrote:
Hello,

We are incurring a problem with cleanup.py:

We have already run two cycles of cleanup.py (1-6) with a cut off of 30 days,  we see in the logs that many datasets are deleted,  but when we do df -kh  to check the file system, we see creeping up utilization of space - no relinquishing of any space.

We are running in a docker environment and our cleanup job looks as follows:
================================================================

#!/bin/bash

export GALAXY_VIRTUAL_ENV=/galaxy_venv

source $GALAXY_VIRTUAL_ENV/bin/activate

export script="/galaxy-central/scripts/cleanup_datasets/cleanup_datasets.py" 

export config="/etc/galaxy/galaxy.ini"

export logdir="/export/galaxy-central/logs"

export days=30

export num=$1

nohup $script $config -d $days -$num -r -f >> $logdir/cleanup$num.log &

================================================================


and we run it with ./run_cleanup.sh 1

or 2, 3 etc.


Any ideas ?


Thanks!

George Weingart PhD

Huttenhower Lab

Harvard School of Public Health

___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/

___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/