broken after upgrade to 17.09 : uploading files via the API

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

broken after upgrade to 17.09 : uploading files via the API

Hans-Rudolf Hotz
Dear All



If I execute the following command:

python import_with_dbkey.py *** file.bam hg19sub

the bam files will be added to the user's history



the code (import_with_dbkey.py) is as follows:

import sys

sys.path.append('***/bioblend-0.10.0-py2.7.egg')
sys.path.append('***/requests_toolbelt-0.7.0-py2.7.egg')

from bioblend.galaxy import GalaxyInstance
from bioblend.galaxy.tools import ToolClient

url = "http://galaxy-dev.fmi.ch"
key = "***"

history_id = sys.argv[1]
file_path = sys.argv[2]
dbkey= sys.argv[3]

gi = GalaxyInstance(url=url, key=key)

toolClient = ToolClient(gi)

uploadedFile = toolClient.upload_file(file_path, history_id, dbkey=dbkey)



Now, if I use the virtual environment from Galaxy, I get:

(.venv)-bash-4.2$ python import_with_dbkey.py *** file.bam hg19sub
Traceback (most recent call last):
   File "/***/galaxy/tools/fmi__api_helpers/import_with_dbkey.py", line
40, in <module>
     uploadedFile = toolClient.upload_file(file_path, history_id,
dbkey=dbkey)
   File
"/***/galaxy/.venv/lib/python2.7/site-packages/bioblend/galaxy/tools/__init__.py",
line 148, in upload_file
     return self._tool_post(payload, files_attached=True)
   File
"/***/galaxy/.venv/lib/python2.7/site-packages/bioblend/galaxy/tools/__init__.py",
line 218, in _tool_post
     return Client._post(self, payload, files_attached=files_attached)
   File
"/***/galaxy/.venv/lib/python2.7/site-packages/bioblend/galaxy/client.py",
line 171, in _post
     files_attached=files_attached)
   File
"/***/galaxy/.venv/lib/python2.7/site-packages/bioblend/galaxyclient.py", line
131, in make_post_request
     r.status_code, body=r.text)
bioblend.galaxy.client.ConnectionError: Unexpected response from galaxy:
400: <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>400 Bad Request</title>
</head><body>
<h1>Bad Request</h1>
<p>Your browser sent a request that this server could not understand.<br />
</p>
</body></html>

(.venv)-bash-4.2$


...and the file does not get uploaded.


I will continue investigating the difference between the python modules
used, but maybe someone has a clue and could help me


Thank you very much

Hans-Rudolf

___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/
Reply | Threaded
Open this post in threaded view
|

Re: broken after upgrade to 17.09 : uploading files via the API

John Chilton-4
Thanks for the report,

I suspect the breaking change would be this -
https://github.com/galaxyproject/galaxy/pull/4563. I added a bunch of
tests for these API changes that I though would ensure backward
compatibility but perhaps there is some sort of breakage here. I
opened a PR in response to this thread against bioblend to ensure
setting the dbkey still works
(https://github.com/galaxyproject/bioblend/pull/252) and it seems to
work for me.

Does that dbkey not properly exist for that user? Maybe there is a
change in behavior for non-existent dbkeys? Or is the dbkey a custom
build for that user? I could write some tests for that if that is the
case.

-John

On Mon, Nov 20, 2017 at 7:38 AM, Hans-Rudolf Hotz <[hidden email]> wrote:

> Dear All
>
>
>
> If I execute the following command:
>
> python import_with_dbkey.py *** file.bam hg19sub
>
> the bam files will be added to the user's history
>
>
>
> the code (import_with_dbkey.py) is as follows:
>
> import sys
>
> sys.path.append('***/bioblend-0.10.0-py2.7.egg')
> sys.path.append('***/requests_toolbelt-0.7.0-py2.7.egg')
>
> from bioblend.galaxy import GalaxyInstance
> from bioblend.galaxy.tools import ToolClient
>
> url = "http://galaxy-dev.fmi.ch"
> key = "***"
>
> history_id = sys.argv[1]
> file_path = sys.argv[2]
> dbkey= sys.argv[3]
>
> gi = GalaxyInstance(url=url, key=key)
>
> toolClient = ToolClient(gi)
>
> uploadedFile = toolClient.upload_file(file_path, history_id, dbkey=dbkey)
>
>
>
> Now, if I use the virtual environment from Galaxy, I get:
>
> (.venv)-bash-4.2$ python import_with_dbkey.py *** file.bam hg19sub
> Traceback (most recent call last):
>   File "/***/galaxy/tools/fmi__api_helpers/import_with_dbkey.py", line 40,
> in <module>
>     uploadedFile = toolClient.upload_file(file_path, history_id,
> dbkey=dbkey)
>   File
> "/***/galaxy/.venv/lib/python2.7/site-packages/bioblend/galaxy/tools/__init__.py",
> line 148, in upload_file
>     return self._tool_post(payload, files_attached=True)
>   File
> "/***/galaxy/.venv/lib/python2.7/site-packages/bioblend/galaxy/tools/__init__.py",
> line 218, in _tool_post
>     return Client._post(self, payload, files_attached=files_attached)
>   File
> "/***/galaxy/.venv/lib/python2.7/site-packages/bioblend/galaxy/client.py",
> line 171, in _post
>     files_attached=files_attached)
>   File
> "/***/galaxy/.venv/lib/python2.7/site-packages/bioblend/galaxyclient.py",
> line 131, in make_post_request
>     r.status_code, body=r.text)
> bioblend.galaxy.client.ConnectionError: Unexpected response from galaxy:
> 400: <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
> <html><head>
> <title>400 Bad Request</title>
> </head><body>
> <h1>Bad Request</h1>
> <p>Your browser sent a request that this server could not understand.<br />
> </p>
> </body></html>
>
> (.venv)-bash-4.2$
>
>
> ...and the file does not get uploaded.
>
>
> I will continue investigating the difference between the python modules
> used, but maybe someone has a clue and could help me
>
>
> Thank you very much
>
> Hans-Rudolf
>
> ___________________________________________________________
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>  https://lists.galaxyproject.org/
>
> To search Galaxy mailing lists use the unified search at:
>  http://galaxyproject.org/search/
___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/
Reply | Threaded
Open this post in threaded view
|

Re: broken after upgrade to 17.09 : uploading files via the API

Hans-Rudolf Hotz
Dear John

Thank you very much for your quick response. Unfortunately, this doesn't
work for me.

Yes, the dbkey exists. It is not custom build for this user, and other
datasets already exist in this history with this dbkey.



The culprit is somewhere in the virtual env:


(.venv)-bash-4.2$ which python
/***/galaxy/.venv/bin/python
(.venv)-bash-4.2$
(.venv)-bash-4.2$ python import_with_dbkey.py *** file.bam hg19sub

=> does not work

...but:

(.venv)-bash-4.2$ /usr/bin/python import_with_dbkey.py *** file.bam hg19sub

=> does work


I can reproduce this without the virtual env (the other way round)


-bash-4.2$ which python
/usr/bin/python
-bash-4.2$
-bash-4.2$ python import_with_dbkey.py *** file.bam hg19sub

=> does work

-bash-4.2$ /***/galaxy/.venv/bin/python import_with_dbkey.py ***
file.bam hg19sub

=> does not work


and if I use the old directory (used with release 17.01)

-bash-4.2$ /***/galaxy_20171120/.venv/bin/python import_with_dbkey.py
*** file.bam hg19sub

=> it does work as well



Hans-Rudolf



On 11/20/2017 02:02 PM, John Chilton wrote:

> Thanks for the report,
>
> I suspect the breaking change would be this -
> https://github.com/galaxyproject/galaxy/pull/4563. I added a bunch of
> tests for these API changes that I though would ensure backward
> compatibility but perhaps there is some sort of breakage here. I
> opened a PR in response to this thread against bioblend to ensure
> setting the dbkey still works
> (https://github.com/galaxyproject/bioblend/pull/252) and it seems to
> work for me.
>
> Does that dbkey not properly exist for that user? Maybe there is a
> change in behavior for non-existent dbkeys? Or is the dbkey a custom
> build for that user? I could write some tests for that if that is the
> case.
>
> -John
>
> On Mon, Nov 20, 2017 at 7:38 AM, Hans-Rudolf Hotz <[hidden email]> wrote:
>> Dear All
>>
>>
>>
>> If I execute the following command:
>>
>> python import_with_dbkey.py *** file.bam hg19sub
>>
>> the bam files will be added to the user's history
>>
>>
>>
>> the code (import_with_dbkey.py) is as follows:
>>
>> import sys
>>
>> sys.path.append('***/bioblend-0.10.0-py2.7.egg')
>> sys.path.append('***/requests_toolbelt-0.7.0-py2.7.egg')
>>
>> from bioblend.galaxy import GalaxyInstance
>> from bioblend.galaxy.tools import ToolClient
>>
>> url = "http://galaxy-dev.fmi.ch"
>> key = "***"
>>
>> history_id = sys.argv[1]
>> file_path = sys.argv[2]
>> dbkey= sys.argv[3]
>>
>> gi = GalaxyInstance(url=url, key=key)
>>
>> toolClient = ToolClient(gi)
>>
>> uploadedFile = toolClient.upload_file(file_path, history_id, dbkey=dbkey)
>>
>>
>>
>> Now, if I use the virtual environment from Galaxy, I get:
>>
>> (.venv)-bash-4.2$ python import_with_dbkey.py *** file.bam hg19sub
>> Traceback (most recent call last):
>>    File "/***/galaxy/tools/fmi__api_helpers/import_with_dbkey.py", line 40,
>> in <module>
>>      uploadedFile = toolClient.upload_file(file_path, history_id,
>> dbkey=dbkey)
>>    File
>> "/***/galaxy/.venv/lib/python2.7/site-packages/bioblend/galaxy/tools/__init__.py",
>> line 148, in upload_file
>>      return self._tool_post(payload, files_attached=True)
>>    File
>> "/***/galaxy/.venv/lib/python2.7/site-packages/bioblend/galaxy/tools/__init__.py",
>> line 218, in _tool_post
>>      return Client._post(self, payload, files_attached=files_attached)
>>    File
>> "/***/galaxy/.venv/lib/python2.7/site-packages/bioblend/galaxy/client.py",
>> line 171, in _post
>>      files_attached=files_attached)
>>    File
>> "/***/galaxy/.venv/lib/python2.7/site-packages/bioblend/galaxyclient.py",
>> line 131, in make_post_request
>>      r.status_code, body=r.text)
>> bioblend.galaxy.client.ConnectionError: Unexpected response from galaxy:
>> 400: <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
>> <html><head>
>> <title>400 Bad Request</title>
>> </head><body>
>> <h1>Bad Request</h1>
>> <p>Your browser sent a request that this server could not understand.<br />
>> </p>
>> </body></html>
>>
>> (.venv)-bash-4.2$
>>
>>
>> ...and the file does not get uploaded.
>>
>>
>> I will continue investigating the difference between the python modules
>> used, but maybe someone has a clue and could help me
>>
>>
>> Thank you very much
>>
>> Hans-Rudolf
>>
>> ___________________________________________________________
>> Please keep all replies on the list by using "reply all"
>> in your mail client.  To manage your subscriptions to this
>> and other Galaxy lists, please use the interface at:
>>   https://lists.galaxyproject.org/
>>
>> To search Galaxy mailing lists use the unified search at:
>>   http://galaxyproject.org/search/
___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/