Error executing workflow via API

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

Error executing workflow via API

burdettn
Hi,
     I execute a number of workflows via the API which all work fine. However, the longest one I use (consists of three linked tasks), returns an error. I call the workflow in the python script using popen and has the format:

/home/galaxy/milxcloud-new/scripts/api/workflow_execute.py d2fcd3feb4c6318c496d55fa8869b67c http://barium-rbh/new/api/workflows f597429621d6eb2b hist_id=f597429621d6eb2b 4=hda=c8c00aa41dc69085

The task appears on the history panel and all tasks eventually complete in the workflow, however, the task that calls the workflow using the above popen call returns the following error:

Traceback (most recent call last):
  File "/home/galaxy/milxcloud-new/scripts/api/workflow_execute.py", line 31, in <module>
    main()
  File "/home/galaxy/milxcloud-new/scripts/api/workflow_execute.py", line 28, in main
    submit( sys.argv[1], sys.argv[2], data )
  File "/home/galaxy/milxcloud-new/scripts/api/common.py", line 117, in submit
    r = post( api_key, url, data )
  File "/home/galaxy/milxcloud-new/scripts/api/common.py", line 51, in post
    return json.loads( urllib2.urlopen( req ).read() )
  File "/usr/lib/python2.7/socket.py", line 351, in read
    data = self._sock.recv(rbufsize)
  File "/usr/lib/python2.7/httplib.py", line 541, in read
    return self._read_chunked(amt)
  File "/usr/lib/python2.7/httplib.py", line 586, in _read_chunked
    raise IncompleteRead(''.join(value))
httplib.IncompleteRead: IncompleteRead(147 bytes read)

This worked on older versions of Galaxy. I need this to work so I can "monitor" the workflow so know when all tasks are complete.

has anyone seen this error? Or got an idea how to solve it.

hg summary reports:
% hg summary
parent: 13771:7a4d321c0e38 tip
 Updated tag latest_2014.06.02 for changeset 8c30e91bc9ae
branch: stable
commit: (clean)
update: (current)

Thanks
Neil

___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
Reply | Threaded
Open this post in threaded view
|

Re: Error executing workflow via API

John Chilton-4
There are definitely limitations to the size of workflows that can be
executed right now but I feel like that problem should be getting
better not worse so this is a little confusing. Did something besides
the Galaxy version change (like proxy settings, timeouts, etc...?).

The client side error interesting but I feel like a server side error
would reveal more - is there a server side error in the Galaxy logs
for this?

-John

On Mon, Sep 8, 2014 at 9:29 PM,  <[hidden email]> wrote:

> Hi,
>      I execute a number of workflows via the API which all work fine.
> However, the longest one I use (consists of three linked tasks), returns an
> error. I call the workflow in the python script using popen and has the
> format:
>
> /home/galaxy/milxcloud-new/scripts/api/workflow_execute.py
> d2fcd3feb4c6318c496d55fa8869b67c http://barium-rbh/new/api/workflows
> f597429621d6eb2b hist_id=f597429621d6eb2b 4=hda=c8c00aa41dc69085
>
> The task appears on the history panel and all tasks eventually complete in
> the workflow, however, the task that calls the workflow using the above
> popen call returns the following error:
>
> Traceback (most recent call last):
>   File "/home/galaxy/milxcloud-new/scripts/api/workflow_execute.py", line
> 31, in <module>
>     main()
>   File "/home/galaxy/milxcloud-new/scripts/api/workflow_execute.py", line
> 28, in main
>     submit( sys.argv[1], sys.argv[2], data )
>   File "/home/galaxy/milxcloud-new/scripts/api/common.py", line 117, in
> submit
>     r = post( api_key, url, data )
>   File "/home/galaxy/milxcloud-new/scripts/api/common.py", line 51, in post
>     return json.loads( urllib2.urlopen( req ).read() )
>   File "/usr/lib/python2.7/socket.py", line 351, in read
>     data = self._sock.recv(rbufsize)
>   File "/usr/lib/python2.7/httplib.py", line 541, in read
>     return self._read_chunked(amt)
>   File "/usr/lib/python2.7/httplib.py", line 586, in _read_chunked
>     raise IncompleteRead(''.join(value))
> httplib.IncompleteRead: IncompleteRead(147 bytes read)
>
> This worked on older versions of Galaxy. I need this to work so I can
> "monitor" the workflow so know when all tasks are complete.
>
> has anyone seen this error? Or got an idea how to solve it.
>
> hg summary reports:
> % hg summary
> parent: 13771:7a4d321c0e38 tip
>  Updated tag latest_2014.06.02 for changeset 8c30e91bc9ae
> branch: stable
> commit: (clean)
> update: (current)
>
> Thanks
> Neil
>
> ___________________________________________________________
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>   http://lists.bx.psu.edu/
>
> To search Galaxy mailing lists use the unified search at:
>   http://galaxyproject.org/search/mailinglists/
___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
Reply | Threaded
Open this post in threaded view
|

Re: Error executing workflow via API

burdettn
Hi John,
           Thanks for the quick response. There seems to be nothing obvious on the server side log. This is all I get:

galaxy.jobs.runners DEBUG 2014-09-09 11:43:23,850 (322) command is: python /home/galaxy/milxcloud-new/tools/cte/cte_process_input_data.py '/home/galaxy/milxcloud-new/tools/cte/cte_process_input_data.py' '/home/galaxy/milxcloud-new/database/files/000/dataset_447.dat' 'Input data (141-S-0851-MRI-T1-Screening)' '/home/galaxy/milxcloud-new/database/files/000/dataset_447_files' d2fcd3feb4c6318c496d55fa8869b67c '/home/galaxy/milxcloud-new/database/files/000/dataset_454.dat' '/home/galaxy/milxcloud-new/database/job_working_directory/000/322/dataset_454_files'
galaxy.jobs.runners.local DEBUG 2014-09-09 11:43:23,852 (322) executing job script: /home/galaxy/milxcloud-new/database/job_working_directory/000/322/galaxy_322.sh
galaxy.jobs DEBUG 2014-09-09 11:43:23,882 (322) Persisting job destination (destination id: local:///)
127.0.0.1 - - [09/Sep/2014:11:43:23 +1000] "POST /new/api/workflows?key=d2fcd3feb4c6318c496d55fa8869b67c HTTP/1.1" 200 - "-" "Python-urllib/2.7"

I still have an 'old' version of Galaxy and so can confirm that this workflow works fine on older versions of Galaxy.

With this 'latest' version of Galaxy I am using the default values stated in universe_wsgi.ini . I am happy to modify values to see if I can resolve this issue if you can suggest where /what I modify.

Also, in older versions of Galaxy after executing the popen command I get a response almost immediately (and I do with other workflows I use). However, in this particular case I don't get a response for 5 minutes.

Thanks
Neil
________________________________________
From: John Chilton [[hidden email]]
Sent: Tuesday, September 09, 2014 11:37 AM
To: Burdett, Neil (DP&S, Herston - RBWH)
Cc: [hidden email]
Subject: Re: [galaxy-dev] Error executing workflow via API

There are definitely limitations to the size of workflows that can be
executed right now but I feel like that problem should be getting
better not worse so this is a little confusing. Did something besides
the Galaxy version change (like proxy settings, timeouts, etc...?).

The client side error interesting but I feel like a server side error
would reveal more - is there a server side error in the Galaxy logs
for this?

-John

On Mon, Sep 8, 2014 at 9:29 PM,  <[hidden email]> wrote:

> Hi,
>      I execute a number of workflows via the API which all work fine.
> However, the longest one I use (consists of three linked tasks), returns an
> error. I call the workflow in the python script using popen and has the
> format:
>
> /home/galaxy/milxcloud-new/scripts/api/workflow_execute.py
> d2fcd3feb4c6318c496d55fa8869b67c http://barium-rbh/new/api/workflows
> f597429621d6eb2b hist_id=f597429621d6eb2b 4=hda=c8c00aa41dc69085
>
> The task appears on the history panel and all tasks eventually complete in
> the workflow, however, the task that calls the workflow using the above
> popen call returns the following error:
>
> Traceback (most recent call last):
>   File "/home/galaxy/milxcloud-new/scripts/api/workflow_execute.py", line
> 31, in <module>
>     main()
>   File "/home/galaxy/milxcloud-new/scripts/api/workflow_execute.py", line
> 28, in main
>     submit( sys.argv[1], sys.argv[2], data )
>   File "/home/galaxy/milxcloud-new/scripts/api/common.py", line 117, in
> submit
>     r = post( api_key, url, data )
>   File "/home/galaxy/milxcloud-new/scripts/api/common.py", line 51, in post
>     return json.loads( urllib2.urlopen( req ).read() )
>   File "/usr/lib/python2.7/socket.py", line 351, in read
>     data = self._sock.recv(rbufsize)
>   File "/usr/lib/python2.7/httplib.py", line 541, in read
>     return self._read_chunked(amt)
>   File "/usr/lib/python2.7/httplib.py", line 586, in _read_chunked
>     raise IncompleteRead(''.join(value))
> httplib.IncompleteRead: IncompleteRead(147 bytes read)
>
> This worked on older versions of Galaxy. I need this to work so I can
> "monitor" the workflow so know when all tasks are complete.
>
> has anyone seen this error? Or got an idea how to solve it.
>
> hg summary reports:
> % hg summary
> parent: 13771:7a4d321c0e38 tip
>  Updated tag latest_2014.06.02 for changeset 8c30e91bc9ae
> branch: stable
> commit: (clean)
> update: (current)
>
> Thanks
> Neil
>
> ___________________________________________________________
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>   http://lists.bx.psu.edu/
>
> To search Galaxy mailing lists use the unified search at:
>   http://galaxyproject.org/search/mailinglists/

___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
Reply | Threaded
Open this post in threaded view
|

Re: Error executing workflow via API

burdettn
Hi,
    It seems all my other workflows that work have a txt output feeding into a an input accepting a text input. This particular workflow (that's failing) is outputting a html output to a html input ? Could this be the reason for the failure? has anyone else got html outputs to inputs on their workflows ? Or maybe on the wrong track again? The workflows are the same on this version of Galaxy as they are on the older version that works.

Everything else remains the same i.e. same desktop etc...

Neil
________________________________________
From: Burdett, Neil (DP&S, Herston - RBWH)
Sent: Tuesday, September 09, 2014 11:55 AM
To: John Chilton
Cc: [hidden email]
Subject: RE: [galaxy-dev] Error executing workflow via API

Hi John,
           Thanks for the quick response. There seems to be nothing obvious on the server side log. This is all I get:

galaxy.jobs.runners DEBUG 2014-09-09 11:43:23,850 (322) command is: python /home/galaxy/milxcloud-new/tools/cte/cte_process_input_data.py '/home/galaxy/milxcloud-new/tools/cte/cte_process_input_data.py' '/home/galaxy/milxcloud-new/database/files/000/dataset_447.dat' 'Input data (141-S-0851-MRI-T1-Screening)' '/home/galaxy/milxcloud-new/database/files/000/dataset_447_files' d2fcd3feb4c6318c496d55fa8869b67c '/home/galaxy/milxcloud-new/database/files/000/dataset_454.dat' '/home/galaxy/milxcloud-new/database/job_working_directory/000/322/dataset_454_files'
galaxy.jobs.runners.local DEBUG 2014-09-09 11:43:23,852 (322) executing job script: /home/galaxy/milxcloud-new/database/job_working_directory/000/322/galaxy_322.sh
galaxy.jobs DEBUG 2014-09-09 11:43:23,882 (322) Persisting job destination (destination id: local:///)
127.0.0.1 - - [09/Sep/2014:11:43:23 +1000] "POST /new/api/workflows?key=d2fcd3feb4c6318c496d55fa8869b67c HTTP/1.1" 200 - "-" "Python-urllib/2.7"

I still have an 'old' version of Galaxy and so can confirm that this workflow works fine on older versions of Galaxy.

With this 'latest' version of Galaxy I am using the default values stated in universe_wsgi.ini . I am happy to modify values to see if I can resolve this issue if you can suggest where /what I modify.

Also, in older versions of Galaxy after executing the popen command I get a response almost immediately (and I do with other workflows I use). However, in this particular case I don't get a response for 5 minutes.

Thanks
Neil
________________________________________
From: John Chilton [[hidden email]]
Sent: Tuesday, September 09, 2014 11:37 AM
To: Burdett, Neil (DP&S, Herston - RBWH)
Cc: [hidden email]
Subject: Re: [galaxy-dev] Error executing workflow via API

There are definitely limitations to the size of workflows that can be
executed right now but I feel like that problem should be getting
better not worse so this is a little confusing. Did something besides
the Galaxy version change (like proxy settings, timeouts, etc...?).

The client side error interesting but I feel like a server side error
would reveal more - is there a server side error in the Galaxy logs
for this?

-John

On Mon, Sep 8, 2014 at 9:29 PM,  <[hidden email]> wrote:

> Hi,
>      I execute a number of workflows via the API which all work fine.
> However, the longest one I use (consists of three linked tasks), returns an
> error. I call the workflow in the python script using popen and has the
> format:
>
> /home/galaxy/milxcloud-new/scripts/api/workflow_execute.py
> d2fcd3feb4c6318c496d55fa8869b67c http://barium-rbh/new/api/workflows
> f597429621d6eb2b hist_id=f597429621d6eb2b 4=hda=c8c00aa41dc69085
>
> The task appears on the history panel and all tasks eventually complete in
> the workflow, however, the task that calls the workflow using the above
> popen call returns the following error:
>
> Traceback (most recent call last):
>   File "/home/galaxy/milxcloud-new/scripts/api/workflow_execute.py", line
> 31, in <module>
>     main()
>   File "/home/galaxy/milxcloud-new/scripts/api/workflow_execute.py", line
> 28, in main
>     submit( sys.argv[1], sys.argv[2], data )
>   File "/home/galaxy/milxcloud-new/scripts/api/common.py", line 117, in
> submit
>     r = post( api_key, url, data )
>   File "/home/galaxy/milxcloud-new/scripts/api/common.py", line 51, in post
>     return json.loads( urllib2.urlopen( req ).read() )
>   File "/usr/lib/python2.7/socket.py", line 351, in read
>     data = self._sock.recv(rbufsize)
>   File "/usr/lib/python2.7/httplib.py", line 541, in read
>     return self._read_chunked(amt)
>   File "/usr/lib/python2.7/httplib.py", line 586, in _read_chunked
>     raise IncompleteRead(''.join(value))
> httplib.IncompleteRead: IncompleteRead(147 bytes read)
>
> This worked on older versions of Galaxy. I need this to work so I can
> "monitor" the workflow so know when all tasks are complete.
>
> has anyone seen this error? Or got an idea how to solve it.
>
> hg summary reports:
> % hg summary
> parent: 13771:7a4d321c0e38 tip
>  Updated tag latest_2014.06.02 for changeset 8c30e91bc9ae
> branch: stable
> commit: (clean)
> update: (current)
>
> Thanks
> Neil
>
> ___________________________________________________________
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>   http://lists.bx.psu.edu/
>
> To search Galaxy mailing lists use the unified search at:
>   http://galaxyproject.org/search/mailinglists/

___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/