submit() gets blocked when running workflows via API

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

submit() gets blocked when running workflows via API

Carlos del Ojo
Hi All,

I am using the API (posting to 'http://localhost:8080/api/workflows' )
and running 8 workflows in a row.

I use submit function imported from common.

First call to submit takes few seconds to return from submit()
second takes a few more seconds, but in the fifth submit it gets hanged.

However if I start stopping all workflows via the web interface fifth submit() ends goes on with 6th 7th and 8th.

I am not sure if there is a limit of workflows running in parallel but if a run them manually through the GUI they work fine (the eight)

does anyone know why API calls get blocked? Is it maybe something to do with the Paste HTTP server used by galaxy?

Best,
C



___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
Reply | Threaded
Open this post in threaded view
|

Re: submit() gets blocked when running workflows via API

John Chilton-4
Hey Carlos,

  Have you made any progress on this, I have never heard of this
behavior but at the same time it does not surprise tremendously
either. How large is the workflow? If large - you will almost
certainly want to make sure you are using a Postgres database server
instead of sqlite. Likewise if you don't have a proxy like nginx or
Apache setup that might help.

You can checkout the following page for more information on scaling up Galaxy:
https://wiki.galaxyproject.org/Admin/Config/Performance/ProductionServer

  What version of Galaxy are you using?

Thanks for using Galaxy and good luck,
-John

On Wed, Jun 18, 2014 at 11:33 AM, Carlos del Ojo <[hidden email]> wrote:

> Hi All,
>
> I am using the API (posting to 'http://localhost:8080/api/workflows' )
> and running 8 workflows in a row.
>
> I use submit function imported from common.
>
> First call to submit takes few seconds to return from submit()
> second takes a few more seconds, but in the fifth submit it gets hanged.
>
> However if I start stopping all workflows via the web interface fifth
> submit() ends goes on with 6th 7th and 8th.
>
> I am not sure if there is a limit of workflows running in parallel but if a
> run them manually through the GUI they work fine (the eight)
>
> does anyone know why API calls get blocked? Is it maybe something to do with
> the Paste HTTP server used by galaxy?
>
> Best,
> C
>
>
>
> ___________________________________________________________
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>   http://lists.bx.psu.edu/
>
> To search Galaxy mailing lists use the unified search at:
>   http://galaxyproject.org/search/mailinglists/
___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
Reply | Threaded
Open this post in threaded view
|

Re: submit() gets blocked when running workflows via API

Gildas Le Corguillé
In reply to this post by Carlos del Ojo
Hi,

We had this behavior when we used the default SQLite.
Since we switched on MySQL, it fixed this :P

Maybe a clue for you ?

Gildas


Le 18/06/2014 18:33, Carlos del Ojo a écrit :
Hi All,

I am using the API (posting to 'http://localhost:8080/api/workflows' )
and running 8 workflows in a row.

I use submit function imported from common.

First call to submit takes few seconds to return from submit()
second takes a few more seconds, but in the fifth submit it gets hanged.

However if I start stopping all workflows via the web interface fifth submit() ends goes on with 6th 7th and 8th.

I am not sure if there is a limit of workflows running in parallel but if a run them manually through the GUI they work fine (the eight)

does anyone know why API calls get blocked? Is it maybe something to do with the Paste HTTP server used by galaxy?

Best,
C




___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/


___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
Reply | Threaded
Open this post in threaded view
|

Re: submit() gets blocked when running workflows via API

Carlos del Ojo
Hi,

Yes I found a workaround.

Initially I had problems because I was using SQLite and I had lock problems that led to ugly SQLAlchemy errors.
After changing to Postgres SQL I got rid of lock errors but I had same issue.

So I tried to create a simple script (my workflow has 14 processes)

from common import submit

submit(...
submit(...
submit(...
...
x8 times

and same issue, after the 4th or 5th submit call, the HTTP response of of the call to the api takes ages (minutes) to return (or never returns)

However, if I run separate threads each calling submit, everything works fine :S...

I know it does not make much sense but before implementing threads I did first experiments in serial and it was failing
so I did not even try in parallel... but it works.

I've been doing some debugging and API seems to work fine, and I think this is some issue with Paste HTTP server
but I couldn't be bothered investigating further.

Best,
C



On Mon, Jun 23, 2014 at 8:43 AM, Gildas Le Corguille <[hidden email]> wrote:
Hi,

We had this behavior when we used the default SQLite.
Since we switched on MySQL, it fixed this :P

Maybe a clue for you ?

Gildas


Le 18/06/2014 18:33, Carlos del Ojo a écrit :
Hi All,

I am using the API (posting to 'http://localhost:8080/api/workflows' )
and running 8 workflows in a row.

I use submit function imported from common.

First call to submit takes few seconds to return from submit()
second takes a few more seconds, but in the fifth submit it gets hanged.

However if I start stopping all workflows via the web interface fifth submit() ends goes on with 6th 7th and 8th.

I am not sure if there is a limit of workflows running in parallel but if a run them manually through the GUI they work fine (the eight)

does anyone know why API calls get blocked? Is it maybe something to do with the Paste HTTP server used by galaxy?

Best,
C




___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/



___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/