Re: Multi File upload api

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

Re: Multi File upload api

alexbk66

Hi John,

 

Can you please have a look at Neil’s question.

 

Thank you,

 

-Alex

 

From: Burdett, Neil (ICT Centre, Herston - RBWH)
Sent: Thursday, 30 May 2013 4:30 PM
To: Khassapov, Alex (CSIRO IM&T, Clayton)
Subject: Multi File upload api

 

Hi Alex,

          The file “galaxy-dist/scripts/api/example_watch_folder.py” allows us to watch a folder then upload files that arrive in a specified input folder into the database and history.

 

Can you ask your friend (who implemented the multi file upload tool), what changes are necessary to this file so we can upload multiple files as we do from the GUI. I assume he would know quite quickly what to do and hopefully quite simple....

 

Thanks

Neil


___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
Reply | Threaded
Open this post in threaded view
|

Re: Multi File upload api

John Chilton
Alex, if I am not mistaken you guys are using a fairly old version of
the multiple file dataset code. I think on the server side you would
need to apply the following three changesets to update the library
backend to support the creation of multifile datasets (I guess if you
are doing path pasting only the first changeset is really needed):

https://bitbucket.org/msiappdev/galaxy-extras/diff/lib/galaxy/webapps/galaxy/controllers/library_common.py?diff2=25e137130d32&at=stable
https://bitbucket.org/msiappdev/galaxy-extras/diff/lib/galaxy/webapps/galaxy/controllers/library_common.py?diff2=a15ce4c00716&at=stable
https://bitbucket.org/msiappdev/galaxy-extras/diff/lib/galaxy/webapps/galaxy/controllers/library_common.py?diff2=d2ae2265bc96&at=stable

If your version of Galaxy is likewise pretty old you will need pull
request 123 as first
(https://bitbucket.org/galaxy/galaxy-central/pull-request/123/refactor-library_commonpy-toward-more/diff).

Once those are in place updating example_watch_folder.py should be
pretty doable (Hagai Cohen had it working at some point):

1) You will need to replace:

data['upload_option'] = 'upload_paths'

with

data['upload_option'] = 'upload_paths_multifile'

2) You will need to specify a name (e.g. add something like:
data['NAME'] = os.path.basename(fullpath))

3) Multiple file datasets do not support data['ext'] = 'auto', so you
will need to specify an explicit format there (the script will
hopefully be able to determine this from the inputs).

The most updated version of the multiple file dataset stuff
https://bitbucket.org/msiappdev/galaxy-extras/ paired with the most
updated version of blend4j (https://github.com/jmchilton/blend4j) are
compatible and if you do enough digging can provide insights on how to
create multiple file datasets via the API (library path pastes,
library server directory uploads, and history uploads).

Hope this helps,
-John

On Thu, May 30, 2013 at 1:40 AM,  <[hidden email]> wrote:

> Hi John,
>
>
>
> Can you please have a look at Neil’s question.
>
>
>
> Thank you,
>
>
>
> -Alex
>
>
>
> From: Burdett, Neil (ICT Centre, Herston - RBWH)
> Sent: Thursday, 30 May 2013 4:30 PM
> To: Khassapov, Alex (CSIRO IM&T, Clayton)
> Subject: Multi File upload api
>
>
>
> Hi Alex,
>
>           The file “galaxy-dist/scripts/api/example_watch_folder.py” allows
> us to watch a folder then upload files that arrive in a specified input
> folder into the database and history.
>
>
>
> Can you ask your friend (who implemented the multi file upload tool), what
> changes are necessary to this file so we can upload multiple files as we do
> from the GUI. I assume he would know quite quickly what to do and hopefully
> quite simple....
>
>
>
> Thanks
>
> Neil

___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
Reply | Threaded
Open this post in threaded view
|

Re: Multi File upload api

alexbk66
Thanks John,

We do have pretty old code, so I applied your patches.

One minor problem, in multi_upload tool "Dataset Name" is ignored, just picks the first file name. Do we need to update upoald.py or multi_upload.xml?

-Alex

-----Original Message-----
From: [hidden email] [mailto:[hidden email]] On Behalf Of John Chilton
Sent: Friday, 31 May 2013 5:50 AM
To: Khassapov, Alex (CSIRO IM&T, Clayton)
Cc: Burdett, Neil (ICT Centre, Herston - RBWH); <[hidden email]>
Subject: Re: Multi File upload api

Alex, if I am not mistaken you guys are using a fairly old version of the multiple file dataset code. I think on the server side you would need to apply the following three changesets to update the library backend to support the creation of multifile datasets (I guess if you are doing path pasting only the first changeset is really needed):

https://bitbucket.org/msiappdev/galaxy-extras/diff/lib/galaxy/webapps/galaxy/controllers/library_common.py?diff2=25e137130d32&at=stable
https://bitbucket.org/msiappdev/galaxy-extras/diff/lib/galaxy/webapps/galaxy/controllers/library_common.py?diff2=a15ce4c00716&at=stable
https://bitbucket.org/msiappdev/galaxy-extras/diff/lib/galaxy/webapps/galaxy/controllers/library_common.py?diff2=d2ae2265bc96&at=stable

If your version of Galaxy is likewise pretty old you will need pull request 123 as first (https://bitbucket.org/galaxy/galaxy-central/pull-request/123/refactor-library_commonpy-toward-more/diff).

Once those are in place updating example_watch_folder.py should be pretty doable (Hagai Cohen had it working at some point):

1) You will need to replace:

data['upload_option'] = 'upload_paths'

with

data['upload_option'] = 'upload_paths_multifile'

2) You will need to specify a name (e.g. add something like:
data['NAME'] = os.path.basename(fullpath))

3) Multiple file datasets do not support data['ext'] = 'auto', so you will need to specify an explicit format there (the script will hopefully be able to determine this from the inputs).

The most updated version of the multiple file dataset stuff https://bitbucket.org/msiappdev/galaxy-extras/ paired with the most updated version of blend4j (https://github.com/jmchilton/blend4j) are compatible and if you do enough digging can provide insights on how to create multiple file datasets via the API (library path pastes, library server directory uploads, and history uploads).

Hope this helps,
-John

On Thu, May 30, 2013 at 1:40 AM,  <[hidden email]> wrote:

> Hi John,
>
>
>
> Can you please have a look at Neil's question.
>
>
>
> Thank you,
>
>
>
> -Alex
>
>
>
> From: Burdett, Neil (ICT Centre, Herston - RBWH)
> Sent: Thursday, 30 May 2013 4:30 PM
> To: Khassapov, Alex (CSIRO IM&T, Clayton)
> Subject: Multi File upload api
>
>
>
> Hi Alex,
>
>           The file "galaxy-dist/scripts/api/example_watch_folder.py"
> allows us to watch a folder then upload files that arrive in a
> specified input folder into the database and history.
>
>
>
> Can you ask your friend (who implemented the multi file upload tool),
> what changes are necessary to this file so we can upload multiple
> files as we do from the GUI. I assume he would know quite quickly what
> to do and hopefully quite simple....
>
>
>
> Thanks
>
> Neil

___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
Reply | Threaded
Open this post in threaded view
|

Re: Multi File upload api

Hakeem Almabrazi
Hi guys,

I am sure I will be looking into this shortly but now I want to get the one file case working first :).  Do you guys have any idea why the example_watch_folder.pl will copy the file but not its contents?  The file it copies into the library folder is empty.  How can I go about resolving this issue?

I appreciate any help.

Regards,


-----Original Message-----
From: [hidden email] [mailto:[hidden email]] On Behalf Of [hidden email]
Sent: Friday, May 31, 2013 1:22 AM
To: [hidden email]
Cc: [hidden email]; [hidden email]
Subject: Re: [galaxy-dev] Multi File upload api

Thanks John,

We do have pretty old code, so I applied your patches.

One minor problem, in multi_upload tool "Dataset Name" is ignored, just picks the first file name. Do we need to update upoald.py or multi_upload.xml?

-Alex

-----Original Message-----
From: [hidden email] [mailto:[hidden email]] On Behalf Of John Chilton
Sent: Friday, 31 May 2013 5:50 AM
To: Khassapov, Alex (CSIRO IM&T, Clayton)
Cc: Burdett, Neil (ICT Centre, Herston - RBWH); <[hidden email]>
Subject: Re: Multi File upload api

Alex, if I am not mistaken you guys are using a fairly old version of the multiple file dataset code. I think on the server side you would need to apply the following three changesets to update the library backend to support the creation of multifile datasets (I guess if you are doing path pasting only the first changeset is really needed):

https://bitbucket.org/msiappdev/galaxy-extras/diff/lib/galaxy/webapps/galaxy/controllers/library_common.py?diff2=25e137130d32&at=stable
https://bitbucket.org/msiappdev/galaxy-extras/diff/lib/galaxy/webapps/galaxy/controllers/library_common.py?diff2=a15ce4c00716&at=stable
https://bitbucket.org/msiappdev/galaxy-extras/diff/lib/galaxy/webapps/galaxy/controllers/library_common.py?diff2=d2ae2265bc96&at=stable

If your version of Galaxy is likewise pretty old you will need pull request 123 as first (https://bitbucket.org/galaxy/galaxy-central/pull-request/123/refactor-library_commonpy-toward-more/diff).

Once those are in place updating example_watch_folder.py should be pretty doable (Hagai Cohen had it working at some point):

1) You will need to replace:

data['upload_option'] = 'upload_paths'

with

data['upload_option'] = 'upload_paths_multifile'

2) You will need to specify a name (e.g. add something like:
data['NAME'] = os.path.basename(fullpath))

3) Multiple file datasets do not support data['ext'] = 'auto', so you will need to specify an explicit format there (the script will hopefully be able to determine this from the inputs).

The most updated version of the multiple file dataset stuff https://bitbucket.org/msiappdev/galaxy-extras/ paired with the most updated version of blend4j (https://github.com/jmchilton/blend4j) are compatible and if you do enough digging can provide insights on how to create multiple file datasets via the API (library path pastes, library server directory uploads, and history uploads).

Hope this helps,
-John

On Thu, May 30, 2013 at 1:40 AM,  <[hidden email]> wrote:

> Hi John,
>
>
>
> Can you please have a look at Neil's question.
>
>
>
> Thank you,
>
>
>
> -Alex
>
>
>
> From: Burdett, Neil (ICT Centre, Herston - RBWH)
> Sent: Thursday, 30 May 2013 4:30 PM
> To: Khassapov, Alex (CSIRO IM&T, Clayton)
> Subject: Multi File upload api
>
>
>
> Hi Alex,
>
>           The file "galaxy-dist/scripts/api/example_watch_folder.py"
> allows us to watch a folder then upload files that arrive in a
> specified input folder into the database and history.
>
>
>
> Can you ask your friend (who implemented the multi file upload tool),
> what changes are necessary to this file so we can upload multiple
> files as we do from the GUI. I assume he would know quite quickly what
> to do and hopefully quite simple....
>
>
>
> Thanks
>
> Neil

___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
Reply | Threaded
Open this post in threaded view
|

Re: Multi File upload api

John Chilton
In reply to this post by alexbk66
This changeset is where I implemented collecting and assigning names
for multiple file datasets:

https://bitbucket.org/msiappdev/galaxy-extras/commits/bfb07a8c8b7b4c404339cd1efe50f9b2becb3b62

-John


On Fri, May 31, 2013 at 1:21 AM,  <[hidden email]> wrote:

> Thanks John,
>
> We do have pretty old code, so I applied your patches.
>
> One minor problem, in multi_upload tool "Dataset Name" is ignored, just picks the first file name. Do we need to update upoald.py or multi_upload.xml?
>
> -Alex
>
> -----Original Message-----
> From: [hidden email] [mailto:[hidden email]] On Behalf Of John Chilton
> Sent: Friday, 31 May 2013 5:50 AM
> To: Khassapov, Alex (CSIRO IM&T, Clayton)
> Cc: Burdett, Neil (ICT Centre, Herston - RBWH); <[hidden email]>
> Subject: Re: Multi File upload api
>
> Alex, if I am not mistaken you guys are using a fairly old version of the multiple file dataset code. I think on the server side you would need to apply the following three changesets to update the library backend to support the creation of multifile datasets (I guess if you are doing path pasting only the first changeset is really needed):
>
> https://bitbucket.org/msiappdev/galaxy-extras/diff/lib/galaxy/webapps/galaxy/controllers/library_common.py?diff2=25e137130d32&at=stable
> https://bitbucket.org/msiappdev/galaxy-extras/diff/lib/galaxy/webapps/galaxy/controllers/library_common.py?diff2=a15ce4c00716&at=stable
> https://bitbucket.org/msiappdev/galaxy-extras/diff/lib/galaxy/webapps/galaxy/controllers/library_common.py?diff2=d2ae2265bc96&at=stable
>
> If your version of Galaxy is likewise pretty old you will need pull request 123 as first (https://bitbucket.org/galaxy/galaxy-central/pull-request/123/refactor-library_commonpy-toward-more/diff).
>
> Once those are in place updating example_watch_folder.py should be pretty doable (Hagai Cohen had it working at some point):
>
> 1) You will need to replace:
>
> data['upload_option'] = 'upload_paths'
>
> with
>
> data['upload_option'] = 'upload_paths_multifile'
>
> 2) You will need to specify a name (e.g. add something like:
> data['NAME'] = os.path.basename(fullpath))
>
> 3) Multiple file datasets do not support data['ext'] = 'auto', so you will need to specify an explicit format there (the script will hopefully be able to determine this from the inputs).
>
> The most updated version of the multiple file dataset stuff https://bitbucket.org/msiappdev/galaxy-extras/ paired with the most updated version of blend4j (https://github.com/jmchilton/blend4j) are compatible and if you do enough digging can provide insights on how to create multiple file datasets via the API (library path pastes, library server directory uploads, and history uploads).
>
> Hope this helps,
> -John
>
> On Thu, May 30, 2013 at 1:40 AM,  <[hidden email]> wrote:
>> Hi John,
>>
>>
>>
>> Can you please have a look at Neil's question.
>>
>>
>>
>> Thank you,
>>
>>
>>
>> -Alex
>>
>>
>>
>> From: Burdett, Neil (ICT Centre, Herston - RBWH)
>> Sent: Thursday, 30 May 2013 4:30 PM
>> To: Khassapov, Alex (CSIRO IM&T, Clayton)
>> Subject: Multi File upload api
>>
>>
>>
>> Hi Alex,
>>
>>           The file "galaxy-dist/scripts/api/example_watch_folder.py"
>> allows us to watch a folder then upload files that arrive in a
>> specified input folder into the database and history.
>>
>>
>>
>> Can you ask your friend (who implemented the multi file upload tool),
>> what changes are necessary to this file so we can upload multiple
>> files as we do from the GUI. I assume he would know quite quickly what
>> to do and hopefully quite simple....
>>
>>
>>
>> Thanks
>>
>> Neil

___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/