Purging datasets as part of workflow?

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Purging datasets as part of workflow?

Sveinung Gundersen
Hi,

Is there an automatic way to permanently delete and purge datasets that are the result of intermittent tools in a workflow? That is, how can one automatically keep only the first and the last steps of an workflow without having to manually delete all the intermittent datasets and running the purge scripts?

Thanks,
Sveinung Gundersen

--
Sveinung Gundersen, PhD. Head engineer, ELIXIR.NO / The Genomic HyperBrowser team
Department of Informatics, University of Oslo, Boks 1072 Blindern, NO-0316 OSLO, Norway
Email: [hidden email]. Phone: +47 93 00 94 54


___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
Reply | Threaded
Open this post in threaded view
|

Re: Purging datasets as part of workflow?

Peter Cock
On Thu, Mar 13, 2014 at 6:30 PM, Sveinung Gundersen
<[hidden email]> wrote:

> Hi,
>
> Is there an automatic way to permanently delete and purge
> datasets that are the result of intermittent tools in a workflow?
> That is, how can one automatically keep only the first and
> the last steps of an workflow without having to manually
> delete all the intermittent datasets and running the purge
> scripts?
>
> Thanks,
> Sveinung Gundersen

+1

In the workflows editor we can star particular outputs to be
kept, and the rest are "deleted". I'd much prefer they be
more aggressively "permanently deleted".

Alternately, when editing a workflow there (used to be) a
delete dataset action - how about a "permanently delete"
action which would be great for some big-data workflows?

Peter
___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
Reply | Threaded
Open this post in threaded view
|

Re: Purging datasets as part of workflow?

John Chilton-4
I don't even think you can delete datasets in workflow presently - let
alone permanently delete them - only "hide" them.

https://trello.com/c/YfLGkJKe

The core team just meet and a number of high priorities for the next 9
months were identified and reworking workflow scheduling was very high
on this list. I would expect an number of key workflow modifications
to be made by early fall that would make this much easier.

https://trello.com/c/K2qLZCrg

My own opinion on the particulars of this topic are that whether
datasets are deleted or permanently deleted should be up to the
workflow runner not the workflow author. In particular, the workflow
"model" describes what datasets are to be deleted and the workflow
runner could opt to permanently delete them if their Galaxy instance
allows this. It is exactly because this is not a universally allowed
option that I think it should not be part of the workflow description
that can be shared between instances and on the tool shed.

-John

On Thu, Mar 13, 2014 at 1:38 PM, Peter Cock <[hidden email]> wrote:

> On Thu, Mar 13, 2014 at 6:30 PM, Sveinung Gundersen
> <[hidden email]> wrote:
>> Hi,
>>
>> Is there an automatic way to permanently delete and purge
>> datasets that are the result of intermittent tools in a workflow?
>> That is, how can one automatically keep only the first and
>> the last steps of an workflow without having to manually
>> delete all the intermittent datasets and running the purge
>> scripts?
>>
>> Thanks,
>> Sveinung Gundersen
>
> +1
>
> In the workflows editor we can star particular outputs to be
> kept, and the rest are "deleted". I'd much prefer they be
> more aggressively "permanently deleted".
>
> Alternately, when editing a workflow there (used to be) a
> delete dataset action - how about a "permanently delete"
> action which would be great for some big-data workflows?
>
> Peter
> ___________________________________________________________
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>   http://lists.bx.psu.edu/
>
> To search Galaxy mailing lists use the unified search at:
>   http://galaxyproject.org/search/mailinglists/
___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/