Question using sftp to upload file to galaxy

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

Question using sftp to upload file to galaxy

Rui Wang
Hey Folks,

I'm looking at the instructions of using ftp with proftpd. There is a section talking about extending it to use sftp. However, the sample config isn't comprehensive. I'm wondering if anyone has a working config for reference?

What's the setting of user and group? It says it should match the one in the SQLNamedQuery, what does it mean exactly? I start proftpd as root, but start galaxy as bioinfoadmin(normal user with sudo).

Just fyi, my proftpd config module and config file are pasted below. I'm working it out on a trial and error fashion, please feel free to point out if anything is wrong!

Cheers,
Rui

modules:
$ sbin/proftpd -l
Compiled-in modules:
  mod_core.c
  mod_xfer.c
  mod_rlimit.c
  mod_auth_unix.c
  mod_auth.c
  mod_ls.c
  mod_log.c
  mod_site.c
  mod_delay.c
  mod_facts.c
  mod_sql.c
  mod_sql_postgres.c
  mod_sql_passwd.c
  mod_sftp.c
  mod_cap.c

etc/proftpd.conf

ServerType                    standalone
  # You must put this in a virtual host if you want it to listen on its own port. VHost != Apache Vhost.
  <VirtualHost 10.3.17.42>
    Port 2222
    SFTPEngine on
    AuthOrder mod_auth_unix.c mod_sql.c # If you don't do this you will get weird disconnects
    SFTPHostKey /etc/ssh/ssh_host_rsa_key
    RequireValidShell no
    MaxLoginAttempts 6
    ServerName                      "Galaxy SFTP"
    DefaultServer                       on
    Umask                           077
    User                             bioinfoadmin
    Group                           bioinfoadmin
    UseFtpUsers off
    DefaultRoot                     ~
    AllowOverwrite                  on
    AllowStoreRestart               on
    SQLEngine                       on
    SQLGroupInfo                    sftp_groups name id members

# Do not authenticate against real (system) users
<IfModule mod_auth_pam.c>
AuthPAM                         off
</IfModule>

# Common SQL authentication options
SQLPasswordEngine               on
SQLBackend                      postgres
SQLConnectInfo                  galaxy@...:5432 bioinfoadmin dbpwd
SQLAuthenticate                 users

# Configuration that handles PBKDF2 encryption
# Set up mod_sql to authenticate against the Galaxy database
SQLAuthTypes                    PBKDF2
SQLPasswordPBKDF2               SHA256 10000 24
SQLPasswordEncoding             base64
SQLPasswordUserSalt             sql:/GetUserSalt

# Define a custom query for lookup that returns a passwd-like entry. Replace 512s with the UID and GID of the user running the Galaxy server
SQLUserInfo                     custom:/LookupGalaxyUser
SQLNamedQuery                   LookupGalaxyUser SELECT "email, (CASE WHEN substring(password from 1 for 6) = 'PBKDF2' THEN substring(password from 38 for 69) ELSE password END) AS password2,512,512,'/media/galaxy/galaxy/database/ftp/%U','/bin/bash' FROM galaxy_user WHERE email='%U'"

# Define custom query to fetch the password salt
SQLNamedQuery                   GetUserSalt SELECT "(CASE WHEN SUBSTRING (password from 1 for 6) = 'PBKDF2' THEN SUBSTRING (password from 21 for 16) END) AS salt FROM galaxy_user WHERE email='%U'"
  </VirtualHost>

# Don't use IPv6 support by default.
UseIPv6                         off
MaxInstances                    30

# To cause every FTP user to be "jailed" (chrooted) into their home
# directory, uncomment this line.
# Bar use of SITE CHMOD by default
<Limit SITE_CHMOD>
  DenyAll
</Limit>

# Bar use of RETR (download) since this is not a public file drop
<Limit RETR>
  DenyAll
</Limit>
~


___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/
Reply | Threaded
Open this post in threaded view
|

Re: Question using sftp to upload file to galaxy

Rui Wang
Hey Folks,

I tried a few times with different configurations, but none worked. Did anyone have the successful experience that could share? :-)

Cheers,
Rui

On Sat, Jan 19, 2019 at 1:43 PM Rui Wang <[hidden email]> wrote:
Hey Folks,

I'm looking at the instructions of using ftp with proftpd. There is a section talking about extending it to use sftp. However, the sample config isn't comprehensive. I'm wondering if anyone has a working config for reference?

What's the setting of user and group? It says it should match the one in the SQLNamedQuery, what does it mean exactly? I start proftpd as root, but start galaxy as bioinfoadmin(normal user with sudo).

Just fyi, my proftpd config module and config file are pasted below. I'm working it out on a trial and error fashion, please feel free to point out if anything is wrong!

Cheers,
Rui

modules:
$ sbin/proftpd -l
Compiled-in modules:
  mod_core.c
  mod_xfer.c
  mod_rlimit.c
  mod_auth_unix.c
  mod_auth.c
  mod_ls.c
  mod_log.c
  mod_site.c
  mod_delay.c
  mod_facts.c
  mod_sql.c
  mod_sql_postgres.c
  mod_sql_passwd.c
  mod_sftp.c
  mod_cap.c

etc/proftpd.conf

ServerType                    standalone
  # You must put this in a virtual host if you want it to listen on its own port. VHost != Apache Vhost.
  <VirtualHost 10.3.17.42>
    Port 2222
    SFTPEngine on
    AuthOrder mod_auth_unix.c mod_sql.c # If you don't do this you will get weird disconnects
    SFTPHostKey /etc/ssh/ssh_host_rsa_key
    RequireValidShell no
    MaxLoginAttempts 6
    ServerName                      "Galaxy SFTP"
    DefaultServer                       on
    Umask                           077
    User                             bioinfoadmin
    Group                           bioinfoadmin
    UseFtpUsers off
    DefaultRoot                     ~
    AllowOverwrite                  on
    AllowStoreRestart               on
    SQLEngine                       on
    SQLGroupInfo                    sftp_groups name id members

# Do not authenticate against real (system) users
<IfModule mod_auth_pam.c>
AuthPAM                         off
</IfModule>

# Common SQL authentication options
SQLPasswordEngine               on
SQLBackend                      postgres
SQLConnectInfo                  galaxy@...:5432 bioinfoadmin dbpwd
SQLAuthenticate                 users

# Configuration that handles PBKDF2 encryption
# Set up mod_sql to authenticate against the Galaxy database
SQLAuthTypes                    PBKDF2
SQLPasswordPBKDF2               SHA256 10000 24
SQLPasswordEncoding             base64
SQLPasswordUserSalt             sql:/GetUserSalt

# Define a custom query for lookup that returns a passwd-like entry. Replace 512s with the UID and GID of the user running the Galaxy server
SQLUserInfo                     custom:/LookupGalaxyUser
SQLNamedQuery                   LookupGalaxyUser SELECT "email, (CASE WHEN substring(password from 1 for 6) = 'PBKDF2' THEN substring(password from 38 for 69) ELSE password END) AS password2,512,512,'/media/galaxy/galaxy/database/ftp/%U','/bin/bash' FROM galaxy_user WHERE email='%U'"

# Define custom query to fetch the password salt
SQLNamedQuery                   GetUserSalt SELECT "(CASE WHEN SUBSTRING (password from 1 for 6) = 'PBKDF2' THEN SUBSTRING (password from 21 for 16) END) AS salt FROM galaxy_user WHERE email='%U'"
  </VirtualHost>

# Don't use IPv6 support by default.
UseIPv6                         off
MaxInstances                    30

# To cause every FTP user to be "jailed" (chrooted) into their home
# directory, uncomment this line.
# Bar use of SITE CHMOD by default
<Limit SITE_CHMOD>
  DenyAll
</Limit>

# Bar use of RETR (download) since this is not a public file drop
<Limit RETR>
  DenyAll
</Limit>
~


___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/
Reply | Threaded
Open this post in threaded view
|

Re: Question using sftp to upload file to galaxy

Marius van den Beek
Hi Rui,

there's a fairly complete explanation and example in in https://docs.galaxyproject.org/en/latest/admin/special_topics/ftp.html

Hope that helps,
Marius

On Mon, 28 Jan 2019 at 07:35, Rui Wang <[hidden email]> wrote:
Hey Folks,

I tried a few times with different configurations, but none worked. Did anyone have the successful experience that could share? :-)

Cheers,
Rui

On Sat, Jan 19, 2019 at 1:43 PM Rui Wang <[hidden email]> wrote:
Hey Folks,

I'm looking at the instructions of using ftp with proftpd. There is a section talking about extending it to use sftp. However, the sample config isn't comprehensive. I'm wondering if anyone has a working config for reference?

What's the setting of user and group? It says it should match the one in the SQLNamedQuery, what does it mean exactly? I start proftpd as root, but start galaxy as bioinfoadmin(normal user with sudo).

Just fyi, my proftpd config module and config file are pasted below. I'm working it out on a trial and error fashion, please feel free to point out if anything is wrong!

Cheers,
Rui

modules:
$ sbin/proftpd -l
Compiled-in modules:
  mod_core.c
  mod_xfer.c
  mod_rlimit.c
  mod_auth_unix.c
  mod_auth.c
  mod_ls.c
  mod_log.c
  mod_site.c
  mod_delay.c
  mod_facts.c
  mod_sql.c
  mod_sql_postgres.c
  mod_sql_passwd.c
  mod_sftp.c
  mod_cap.c

etc/proftpd.conf

ServerType                    standalone
  # You must put this in a virtual host if you want it to listen on its own port. VHost != Apache Vhost.
  <VirtualHost 10.3.17.42>
    Port 2222
    SFTPEngine on
    AuthOrder mod_auth_unix.c mod_sql.c # If you don't do this you will get weird disconnects
    SFTPHostKey /etc/ssh/ssh_host_rsa_key
    RequireValidShell no
    MaxLoginAttempts 6
    ServerName                      "Galaxy SFTP"
    DefaultServer                       on
    Umask                           077
    User                             bioinfoadmin
    Group                           bioinfoadmin
    UseFtpUsers off
    DefaultRoot                     ~
    AllowOverwrite                  on
    AllowStoreRestart               on
    SQLEngine                       on
    SQLGroupInfo                    sftp_groups name id members

# Do not authenticate against real (system) users
<IfModule mod_auth_pam.c>
AuthPAM                         off
</IfModule>

# Common SQL authentication options
SQLPasswordEngine               on
SQLBackend                      postgres
SQLConnectInfo                  galaxy@...:5432 bioinfoadmin dbpwd
SQLAuthenticate                 users

# Configuration that handles PBKDF2 encryption
# Set up mod_sql to authenticate against the Galaxy database
SQLAuthTypes                    PBKDF2
SQLPasswordPBKDF2               SHA256 10000 24
SQLPasswordEncoding             base64
SQLPasswordUserSalt             sql:/GetUserSalt

# Define a custom query for lookup that returns a passwd-like entry. Replace 512s with the UID and GID of the user running the Galaxy server
SQLUserInfo                     custom:/LookupGalaxyUser
SQLNamedQuery                   LookupGalaxyUser SELECT "email, (CASE WHEN substring(password from 1 for 6) = 'PBKDF2' THEN substring(password from 38 for 69) ELSE password END) AS password2,512,512,'/media/galaxy/galaxy/database/ftp/%U','/bin/bash' FROM galaxy_user WHERE email='%U'"

# Define custom query to fetch the password salt
SQLNamedQuery                   GetUserSalt SELECT "(CASE WHEN SUBSTRING (password from 1 for 6) = 'PBKDF2' THEN SUBSTRING (password from 21 for 16) END) AS salt FROM galaxy_user WHERE email='%U'"
  </VirtualHost>

# Don't use IPv6 support by default.
UseIPv6                         off
MaxInstances                    30

# To cause every FTP user to be "jailed" (chrooted) into their home
# directory, uncomment this line.
# Bar use of SITE CHMOD by default
<Limit SITE_CHMOD>
  DenyAll
</Limit>

# Bar use of RETR (download) since this is not a public file drop
<Limit RETR>
  DenyAll
</Limit>
~

___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/

___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/
Reply | Threaded
Open this post in threaded view
|

Re: Question using sftp to upload file to galaxy

Rui Wang
Hi Marius,

Thanks for the note. The link you pasted is how I came up with the config in the original question. However it doesn't work...it kept saying my password is incorrect. :-(

$ sftp  -oKexAlgorithms=diffie-hellman-group14-sha1 -oPort=2222 bioinfoadmin@localhost
bioinfoadmin@localhost's password:
Permission denied, please try again.
bioinfoadmin@localhost's password:
Permission denied, please try again.
bioinfoadmin@localhost's password:

Not sure why this would happen. :-( Have you seen this before?

Cheers,
Rui

On Sun, Jan 27, 2019 at 10:49 PM Marius van den Beek <[hidden email]> wrote:
Hi Rui,

there's a fairly complete explanation and example in in https://docs.galaxyproject.org/en/latest/admin/special_topics/ftp.html

Hope that helps,
Marius

On Mon, 28 Jan 2019 at 07:35, Rui Wang <[hidden email]> wrote:
Hey Folks,

I tried a few times with different configurations, but none worked. Did anyone have the successful experience that could share? :-)

Cheers,
Rui

On Sat, Jan 19, 2019 at 1:43 PM Rui Wang <[hidden email]> wrote:
Hey Folks,

I'm looking at the instructions of using ftp with proftpd. There is a section talking about extending it to use sftp. However, the sample config isn't comprehensive. I'm wondering if anyone has a working config for reference?

What's the setting of user and group? It says it should match the one in the SQLNamedQuery, what does it mean exactly? I start proftpd as root, but start galaxy as bioinfoadmin(normal user with sudo).

Just fyi, my proftpd config module and config file are pasted below. I'm working it out on a trial and error fashion, please feel free to point out if anything is wrong!

Cheers,
Rui

modules:
$ sbin/proftpd -l
Compiled-in modules:
  mod_core.c
  mod_xfer.c
  mod_rlimit.c
  mod_auth_unix.c
  mod_auth.c
  mod_ls.c
  mod_log.c
  mod_site.c
  mod_delay.c
  mod_facts.c
  mod_sql.c
  mod_sql_postgres.c
  mod_sql_passwd.c
  mod_sftp.c
  mod_cap.c

etc/proftpd.conf

ServerType                    standalone
  # You must put this in a virtual host if you want it to listen on its own port. VHost != Apache Vhost.
  <VirtualHost 10.3.17.42>
    Port 2222
    SFTPEngine on
    AuthOrder mod_auth_unix.c mod_sql.c # If you don't do this you will get weird disconnects
    SFTPHostKey /etc/ssh/ssh_host_rsa_key
    RequireValidShell no
    MaxLoginAttempts 6
    ServerName                      "Galaxy SFTP"
    DefaultServer                       on
    Umask                           077
    User                             bioinfoadmin
    Group                           bioinfoadmin
    UseFtpUsers off
    DefaultRoot                     ~
    AllowOverwrite                  on
    AllowStoreRestart               on
    SQLEngine                       on
    SQLGroupInfo                    sftp_groups name id members

# Do not authenticate against real (system) users
<IfModule mod_auth_pam.c>
AuthPAM                         off
</IfModule>

# Common SQL authentication options
SQLPasswordEngine               on
SQLBackend                      postgres
SQLConnectInfo                  galaxy@...:5432 bioinfoadmin dbpwd
SQLAuthenticate                 users

# Configuration that handles PBKDF2 encryption
# Set up mod_sql to authenticate against the Galaxy database
SQLAuthTypes                    PBKDF2
SQLPasswordPBKDF2               SHA256 10000 24
SQLPasswordEncoding             base64
SQLPasswordUserSalt             sql:/GetUserSalt

# Define a custom query for lookup that returns a passwd-like entry. Replace 512s with the UID and GID of the user running the Galaxy server
SQLUserInfo                     custom:/LookupGalaxyUser
SQLNamedQuery                   LookupGalaxyUser SELECT "email, (CASE WHEN substring(password from 1 for 6) = 'PBKDF2' THEN substring(password from 38 for 69) ELSE password END) AS password2,512,512,'/media/galaxy/galaxy/database/ftp/%U','/bin/bash' FROM galaxy_user WHERE email='%U'"

# Define custom query to fetch the password salt
SQLNamedQuery                   GetUserSalt SELECT "(CASE WHEN SUBSTRING (password from 1 for 6) = 'PBKDF2' THEN SUBSTRING (password from 21 for 16) END) AS salt FROM galaxy_user WHERE email='%U'"
  </VirtualHost>

# Don't use IPv6 support by default.
UseIPv6                         off
MaxInstances                    30

# To cause every FTP user to be "jailed" (chrooted) into their home
# directory, uncomment this line.
# Bar use of SITE CHMOD by default
<Limit SITE_CHMOD>
  DenyAll
</Limit>

# Bar use of RETR (download) since this is not a public file drop
<Limit RETR>
  DenyAll
</Limit>
~

___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/

___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/
Reply | Threaded
Open this post in threaded view
|

Re: Question using sftp to upload file to galaxy

Marius van den Beek
Are you logging in with the email adress and password registered for your Galaxy user acccount ?
The authentication happens against the Galaxy database.
Also keep an eye on your proftpd logs in case that wasn't the issue.

On Fri, 1 Feb 2019 at 19:25, Rui Wang <[hidden email]> wrote:
Hi Marius,

Thanks for the note. The link you pasted is how I came up with the config in the original question. However it doesn't work...it kept saying my password is incorrect. :-(

$ sftp  -oKexAlgorithms=diffie-hellman-group14-sha1 -oPort=2222 bioinfoadmin@localhost
bioinfoadmin@localhost's password:
Permission denied, please try again.
bioinfoadmin@localhost's password:
Permission denied, please try again.
bioinfoadmin@localhost's password:

Not sure why this would happen. :-( Have you seen this before?

Cheers,
Rui

On Sun, Jan 27, 2019 at 10:49 PM Marius van den Beek <[hidden email]> wrote:
Hi Rui,

there's a fairly complete explanation and example in in https://docs.galaxyproject.org/en/latest/admin/special_topics/ftp.html

Hope that helps,
Marius

On Mon, 28 Jan 2019 at 07:35, Rui Wang <[hidden email]> wrote:
Hey Folks,

I tried a few times with different configurations, but none worked. Did anyone have the successful experience that could share? :-)

Cheers,
Rui

On Sat, Jan 19, 2019 at 1:43 PM Rui Wang <[hidden email]> wrote:
Hey Folks,

I'm looking at the instructions of using ftp with proftpd. There is a section talking about extending it to use sftp. However, the sample config isn't comprehensive. I'm wondering if anyone has a working config for reference?

What's the setting of user and group? It says it should match the one in the SQLNamedQuery, what does it mean exactly? I start proftpd as root, but start galaxy as bioinfoadmin(normal user with sudo).

Just fyi, my proftpd config module and config file are pasted below. I'm working it out on a trial and error fashion, please feel free to point out if anything is wrong!

Cheers,
Rui

modules:
$ sbin/proftpd -l
Compiled-in modules:
  mod_core.c
  mod_xfer.c
  mod_rlimit.c
  mod_auth_unix.c
  mod_auth.c
  mod_ls.c
  mod_log.c
  mod_site.c
  mod_delay.c
  mod_facts.c
  mod_sql.c
  mod_sql_postgres.c
  mod_sql_passwd.c
  mod_sftp.c
  mod_cap.c

etc/proftpd.conf

ServerType                    standalone
  # You must put this in a virtual host if you want it to listen on its own port. VHost != Apache Vhost.
  <VirtualHost 10.3.17.42>
    Port 2222
    SFTPEngine on
    AuthOrder mod_auth_unix.c mod_sql.c # If you don't do this you will get weird disconnects
    SFTPHostKey /etc/ssh/ssh_host_rsa_key
    RequireValidShell no
    MaxLoginAttempts 6
    ServerName                      "Galaxy SFTP"
    DefaultServer                       on
    Umask                           077
    User                             bioinfoadmin
    Group                           bioinfoadmin
    UseFtpUsers off
    DefaultRoot                     ~
    AllowOverwrite                  on
    AllowStoreRestart               on
    SQLEngine                       on
    SQLGroupInfo                    sftp_groups name id members

# Do not authenticate against real (system) users
<IfModule mod_auth_pam.c>
AuthPAM                         off
</IfModule>

# Common SQL authentication options
SQLPasswordEngine               on
SQLBackend                      postgres
SQLConnectInfo                  galaxy@...:5432 bioinfoadmin dbpwd
SQLAuthenticate                 users

# Configuration that handles PBKDF2 encryption
# Set up mod_sql to authenticate against the Galaxy database
SQLAuthTypes                    PBKDF2
SQLPasswordPBKDF2               SHA256 10000 24
SQLPasswordEncoding             base64
SQLPasswordUserSalt             sql:/GetUserSalt

# Define a custom query for lookup that returns a passwd-like entry. Replace 512s with the UID and GID of the user running the Galaxy server
SQLUserInfo                     custom:/LookupGalaxyUser
SQLNamedQuery                   LookupGalaxyUser SELECT "email, (CASE WHEN substring(password from 1 for 6) = 'PBKDF2' THEN substring(password from 38 for 69) ELSE password END) AS password2,512,512,'/media/galaxy/galaxy/database/ftp/%U','/bin/bash' FROM galaxy_user WHERE email='%U'"

# Define custom query to fetch the password salt
SQLNamedQuery                   GetUserSalt SELECT "(CASE WHEN SUBSTRING (password from 1 for 6) = 'PBKDF2' THEN SUBSTRING (password from 21 for 16) END) AS salt FROM galaxy_user WHERE email='%U'"
  </VirtualHost>

# Don't use IPv6 support by default.
UseIPv6                         off
MaxInstances                    30

# To cause every FTP user to be "jailed" (chrooted) into their home
# directory, uncomment this line.
# Bar use of SITE CHMOD by default
<Limit SITE_CHMOD>
  DenyAll
</Limit>

# Bar use of RETR (download) since this is not a public file drop
<Limit RETR>
  DenyAll
</Limit>
~

___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/

___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/