Copying files to FTP upload directory

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

Copying files to FTP upload directory

Keith Suderman
Hello,

I am trying to (ab)use the FTP file upload functionality to get data from an external service into Galaxy.  From my reading of the docs I simply need to configure the ftp_upload_dir and ftp_upload_site in my galaxy.ini (haven't migrated to yaml yet) and Galaxy will expect to find subdirectories named after the user's account (email) in the ftp_upload_dir.

I have configured the ftp_upload_dir (/home/galaxy/database/ftp in our case) and the "Choose FTP File" button is now available in the Upload dialog, however regardless of where I put files, or what I name them, Galaxy always reports that "Your FTP directory does not contain any files."  For example, if I have the user account [hidden email] I would expect to be able to place files in /home/galaxy/database/ftp/[hidden email] and they will be available in the Upload dialog for the [hidden email] user.  However, this is not working; am I missing something?

Thanks,
Keith


___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/
Reply | Threaded
Open this post in threaded view
|

Re: [External] Copying files to FTP upload directory

Keith Suderman
I have been creating directories for the users, but putting files in (for example) /[hidden email] does not make them available for the [hidden email] user in Galaxy.

Also for the list admins... I never did see my message appear on the list and the message Cicada replied to appears to be from [hidden email].  Any ideas why my message(s) appear to be bouncing.  I am pretty sure I am sending from the correct address.

Cheers,
Keith

On Jan 25, 2019, at 4:36 PM, Dennis, H. E. Cicada Brokaw <[hidden email]> wrote:

I think you have to create the user's directories under /home/galaxy/database/ftp. I don't think Galaxy will create them if they are not there.
Cicada

________________________________________
From: galaxy-dev <[hidden email]> on behalf of Keith Suderman <[hidden email]>
Sent: Friday, January 25, 2019 4:01 PM
To: Galaxy Dev List
Subject: [External] [galaxy-dev] Copying files to FTP upload directory

Hello,

I am trying to (ab)use the FTP file upload functionality to get data from an external service into Galaxy.  From my reading of the docs I simply need to configure the ftp_upload_dir and ftp_upload_site in my galaxy.ini (haven't migrated to yaml yet) and Galaxy will expect to find subdirectories named after the user's account (email) in the ftp_upload_dir.

I have configured the ftp_upload_dir (/home/galaxy/database/ftp in our case) and the "Choose FTP File" button is now available in the Upload dialog, however regardless of where I put files, or what I name them, Galaxy always reports that "Your FTP directory does not contain any files."  For example, if I have the user account [hidden email] I would expect to be able to place files in /[hidden email] and they will be available in the Upload dialog for the [hidden email] user.  However, this is not working; am I missing something?

Thanks,
Keith


___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
 https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
 http://galaxyproject.org/search/


___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/
Reply | Threaded
Open this post in threaded view
|

Re: [External] Copying files to FTP upload directory

Martin Čech-2
Hi Keith,

your message is not bouncing, it shows here: http://dev.list.galaxyproject.org/

Your setup seems correct to me, does Galaxy have permission to read files in the given folders?

Also Galaxy will auto-create folders if set the `user_library_import_dir` to be the same as ftp and use `#user_library_import_dir_auto_creation: false`

M.

On Fri, Jan 25, 2019 at 4:58 PM Keith Suderman <[hidden email]> wrote:
I have been creating directories for the users, but putting files in (for example) /[hidden email] does not make them available for the [hidden email] user in Galaxy.

Also for the list admins... I never did see my message appear on the list and the message Cicada replied to appears to be from [hidden email].  Any ideas why my message(s) appear to be bouncing.  I am pretty sure I am sending from the correct address.

Cheers,
Keith

On Jan 25, 2019, at 4:36 PM, Dennis, H. E. Cicada Brokaw <[hidden email]> wrote:

I think you have to create the user's directories under /home/galaxy/database/ftp. I don't think Galaxy will create them if they are not there.
Cicada

________________________________________
From: galaxy-dev <[hidden email]> on behalf of Keith Suderman <[hidden email]>
Sent: Friday, January 25, 2019 4:01 PM
To: Galaxy Dev List
Subject: [External] [galaxy-dev] Copying files to FTP upload directory


Hello,

I am trying to (ab)use the FTP file upload functionality to get data from an external service into Galaxy.  From my reading of the docs I simply need to configure the ftp_upload_dir and ftp_upload_site in my galaxy.ini (haven't migrated to yaml yet) and Galaxy will expect to find subdirectories named after the user's account (email) in the ftp_upload_dir.

I have configured the ftp_upload_dir (/home/galaxy/database/ftp in our case) and the "Choose FTP File" button is now available in the Upload dialog, however regardless of where I put files, or what I name them, Galaxy always reports that "Your FTP directory does not contain any files."  For example, if I have the user account [hidden email] I would expect to be able to place files in /[hidden email] and they will be available in the Upload dialog for the [hidden email] user.  However, this is not working; am I missing something?

Thanks,
Keith


___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
 https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
 http://galaxyproject.org/search/
___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/

___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/
Reply | Threaded
Open this post in threaded view
|

Re: [External] Copying files to FTP upload directory

Keith Suderman
Hi Martin,

Yes, Galaxy has permission to read the files.  I have only tested on my own workstation, so Galaxy runs as me, all the files/directories are owned by me, etc. but I will try on our dev server and see if it makes any difference.  Would using an admin account be a problem?  

And just to clarify, I am assuming I can do the following to get a file available in Galaxy (where the ftp_upload_dir is /home/galaxy/database/ftp):

$> mkdir -p /[hidden email].edu
$> echo "Hello world." > /[hidden email]/Hello.txt
$> chown -R galaxy:galaxy /home/galaxy/database/ftp

Thanks,
Keith

On Jan 25, 2019, at 5:07 PM, Martin Čech <[hidden email]> wrote:

Hi Keith,

your message is not bouncing, it shows here: http://dev.list.galaxyproject.org/

Your setup seems correct to me, does Galaxy have permission to read files in the given folders?

Also Galaxy will auto-create folders if set the `user_library_import_dir` to be the same as ftp and use `#user_library_import_dir_auto_creation: false`

M.

On Fri, Jan 25, 2019 at 4:58 PM Keith Suderman <[hidden email]> wrote:
I have been creating directories for the users, but putting files in (for example) /[hidden email] does not make them available for the [hidden email] user in Galaxy.

Also for the list admins... I never did see my message appear on the list and the message Cicada replied to appears to be from [hidden email].  Any ideas why my message(s) appear to be bouncing.  I am pretty sure I am sending from the correct address.

Cheers,
Keith

On Jan 25, 2019, at 4:36 PM, Dennis, H. E. Cicada Brokaw <[hidden email]> wrote:

I think you have to create the user's directories under /home/galaxy/database/ftp. I don't think Galaxy will create them if they are not there.
Cicada

________________________________________
From: galaxy-dev <[hidden email]> on behalf of Keith Suderman <[hidden email]>
Sent: Friday, January 25, 2019 4:01 PM
To: Galaxy Dev List
Subject: [External] [galaxy-dev] Copying files to FTP upload directory


Hello,

I am trying to (ab)use the FTP file upload functionality to get data from an external service into Galaxy.  From my reading of the docs I simply need to configure the ftp_upload_dir and ftp_upload_site in my galaxy.ini (haven't migrated to yaml yet) and Galaxy will expect to find subdirectories named after the user's account (email) in the ftp_upload_dir.

I have configured the ftp_upload_dir (/home/galaxy/database/ftp in our case) and the "Choose FTP File" button is now available in the Upload dialog, however regardless of where I put files, or what I name them, Galaxy always reports that "Your FTP directory does not contain any files."  For example, if I have the user account [hidden email] I would expect to be able to place files in /[hidden email] and they will be available in the Upload dialog for the [hidden email] user.  However, this is not working; am I missing something?

Thanks,
Keith


___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
 https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
 http://galaxyproject.org/search/
___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/


___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/
Reply | Threaded
Open this post in threaded view
|

Re: [External] Copying files to FTP upload directory [resolved]

Keith Suderman
It turns out I had fat-fingered the key name in the config and had specified ftp_upoload_dir, once I noticed the typo and changed it to ftp_upload_dir everything works as expected.  What threw me was that the "Choose FTP file" button appeared in the upload dialog even though the ftp_upload_dir was not specified.  Looking at the log files I see 403 errors for GET /api/remote_files and when I try that URL in a browser I get some JSON with the error message, "The configuration of this Galaxy instance does not allow upload from FTP directories."

I am not familiar enough with the Javascript client, but it looks like the logic used to determine whether or not to show the "Choose FTP file" button only checks if ftp_upload_site has been specified and not if ftp_upload_dir has been defined.

- Keith

On Jan 25, 2019, at 7:29 PM, Keith Suderman <[hidden email]> wrote:

Hi Martin,

Yes, Galaxy has permission to read the files.  I have only tested on my own workstation, so Galaxy runs as me, all the files/directories are owned by me, etc. but I will try on our dev server and see if it makes any difference.  Would using an admin account be a problem?  

And just to clarify, I am assuming I can do the following to get a file available in Galaxy (where the ftp_upload_dir is /home/galaxy/database/ftp):

$> mkdir -p /[hidden email].edu
$> echo "Hello world." > /[hidden email]/Hello.txt
$> chown -R galaxy:galaxy /home/galaxy/database/ftp

Thanks,
Keith

On Jan 25, 2019, at 5:07 PM, Martin Čech <[hidden email]> wrote:

Hi Keith,

your message is not bouncing, it shows here: http://dev.list.galaxyproject.org/

Your setup seems correct to me, does Galaxy have permission to read files in the given folders?

Also Galaxy will auto-create folders if set the `user_library_import_dir` to be the same as ftp and use `#user_library_import_dir_auto_creation: false`

M.

On Fri, Jan 25, 2019 at 4:58 PM Keith Suderman <[hidden email]> wrote:
I have been creating directories for the users, but putting files in (for example) /[hidden email] does not make them available for the [hidden email] user in Galaxy.

Also for the list admins... I never did see my message appear on the list and the message Cicada replied to appears to be from [hidden email].  Any ideas why my message(s) appear to be bouncing.  I am pretty sure I am sending from the correct address.

Cheers,
Keith

On Jan 25, 2019, at 4:36 PM, Dennis, H. E. Cicada Brokaw <[hidden email]> wrote:

I think you have to create the user's directories under /home/galaxy/database/ftp. I don't think Galaxy will create them if they are not there.
Cicada

________________________________________
From: galaxy-dev <[hidden email]> on behalf of Keith Suderman <[hidden email]>
Sent: Friday, January 25, 2019 4:01 PM
To: Galaxy Dev List
Subject: [External] [galaxy-dev] Copying files to FTP upload directory


Hello,

I am trying to (ab)use the FTP file upload functionality to get data from an external service into Galaxy.  From my reading of the docs I simply need to configure the ftp_upload_dir and ftp_upload_site in my galaxy.ini (haven't migrated to yaml yet) and Galaxy will expect to find subdirectories named after the user's account (email) in the ftp_upload_dir.

I have configured the ftp_upload_dir (/home/galaxy/database/ftp in our case) and the "Choose FTP File" button is now available in the Upload dialog, however regardless of where I put files, or what I name them, Galaxy always reports that "Your FTP directory does not contain any files."  For example, if I have the user account [hidden email] I would expect to be able to place files in /[hidden email] and they will be available in the Upload dialog for the [hidden email] user.  However, this is not working; am I missing something?

Thanks,
Keith


___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
 https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
 http://galaxyproject.org/search/
___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/



___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/