How to speed up submissions of workflows

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

How to speed up submissions of workflows

thondeboer

Hi,

 

I am routinely running hundreds of pair-end sequencing runs and find that STARTING a workflow on all of these can easily take hours since it seems each workflow preparation takes about 3 minutes, so for doing 128 fastq pairs like this it will take over 3 hours just to get all workflows running….

 

Is there any way for me to speed up this process, maybe by adding more handlers (although I am suspecting the WEB server is the bottleneck here)?

I already run 8 handlers but only one web server…

I could loadbalance the webservers through my proxy server…Will that speed up the submissions or are we stuck with this linear process?

Thanks,

 

Thon

 


___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
Reply | Threaded
Open this post in threaded view
|

Re: How to speed up submissions of workflows

Dannon Baker-2
The web process is definitely the bottleneck right now, creating more should improve things.  I'd guess it wouldn't speed up any individual submission, but it'd allow you to submit several at once.  Lastly, this is a known issue that we're hoping to resolve with a rewrite of the workflow dispatching mechanism.

-Dannon


On Fri, Jun 14, 2013 at 2:17 PM, Thon Deboer <[hidden email]> wrote:

Hi,

 

I am routinely running hundreds of pair-end sequencing runs and find that STARTING a workflow on all of these can easily take hours since it seems each workflow preparation takes about 3 minutes, so for doing 128 fastq pairs like this it will take over 3 hours just to get all workflows running….

 

Is there any way for me to speed up this process, maybe by adding more handlers (although I am suspecting the WEB server is the bottleneck here)?

I already run 8 handlers but only one web server…

I could loadbalance the webservers through my proxy server…Will that speed up the submissions or are we stuck with this linear process?

Thanks,

 

Thon

 


___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/


___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/