installation is not finishing

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

installation is not finishing

Jochen
Hi,

I wanted to install two tools but my installation is stuck in Installing
tool dependencies. So I found out that the xmls are cloned but the
dependencies are not getting installed.

It looks like anything related to python is not getting installed?

anyone needs some log file, please tell me.

Cheers Jochen


--
ETH Zurich
*Jochen Bick*
Animal Physiology
Institute of Agricultural Sciences
Postal address: Universitätstrasse 2 / LFW B 58.1
8092 Zurich, Switzerland
Office: Eschikon 27
8315 Lindau, Switzerland

Phone +41 52 354 92 06
[hidden email] <mailto:[hidden email]>
www.ap.ethz.ch

___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/
Reply | Threaded
Open this post in threaded view
|

Re: installation is not finishing

Jochen
It's fixed a restart and uninstall of the half installed tool helped.

Cheers Jochen

On 23.11.18 09:32, Jochen Bick wrote:

> Hi,
>
> I wanted to install two tools but my installation is stuck in
> Installing tool dependencies. So I found out that the xmls are cloned
> but the dependencies are not getting installed.
>
> It looks like anything related to python is not getting installed?
>
> anyone needs some log file, please tell me.
>
> Cheers Jochen
>
>
--
ETH Zurich
*Jochen Bick*
Animal Physiology
Institute of Agricultural Sciences
Postal address: Universitätstrasse 2 / LFW B 58.1
8092 Zurich, Switzerland
Office: Eschikon 27
8315 Lindau, Switzerland

Phone +41 52 354 92 06
[hidden email] <mailto:[hidden email]>
www.ap.ethz.ch

___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/
Reply | Threaded
Open this post in threaded view
|

Using soft-links to NFS-shares as user directories

Previti
In reply to this post by Jochen
Dear all,

We implemented a system using soft-links to link user-directories to NFS shares in order to import/link datasets that are larger than 2GB (through links to data-libraries) and provide our users with longer term storage space for their analyses.
This worked fine for Galaxy version 17.09...and stopped working for Version 18.05.

An example can be seen here in the folder (the path to this folder will vary between installations): /opt/galaxy/galaxy/database/files

lrwxrwxrwx   1 galaxy  W190-htseq-bioinf      28 Sep 11 12:37 [hidden email] -> /galaxy-clients/w190_galaxy/

Setting the proper rights allows users to link files from the nfs data-share make available under /galaxy-clients (in this case the share  is called "w190_galaxy")
directly into a data-library (with the approriate rights).

Would anybody have an idea why this doesn't work anymore in v18.05?
Is it a galaxy.ini configuration issue?

Thanks for any advise you could provide!

Christopher




--
Dr. Christopher Previti
Genomics and
Proteomics Core Facility
High Throughput Sequencing (W190)
Bioinformatician

German Cancer Research Center (DKFZ)
Foundation under Public Law
Im Neuenheimer Feld 580
69120 Heidelberg
Germany
Room: B2.102 (INF580/TP3)
Phone: +49 6221 42-4661

christopher.previti@...
www.dkfz.de

Management Board: Prof. Dr. Michael Baumann, Prof. Dr. Josef Puchta
VAT-ID No.: DE143293537

Vertraulichkeitshinweis: Diese Nachricht ist ausschließlich für die Personen bestimmt, an die sie adressiert ist.
Sie kann vertrauliche und/oder nur für den/die Empfänger bestimmte Informationen enthalten. Sollten Sie nicht
der bestimmungsgemäße Empfänger sein, kontaktieren Sie bitte den Absender und löschen Sie die Mitteilung.
Jegliche unbefugte Verwendung der Informationen in dieser Nachricht ist untersagt.



    

___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/
Reply | Threaded
Open this post in threaded view
|

Re: Using soft-links to NFS-shares as user directories

Youssef  GHORBAL
Hello,

What do you mean exactly "doesn't work anymore" ? Do current links no longer appear in data-library ? Do new soft-links can no longer be made available into data-library ? Do you have any error messages ?
By the way the symlink permissions are not valuable, you need to check the target permissions (/galaxy-clients/w190_galaxy/ for instance)

Youssef Ghorbal
IIS / DSI
Institut Pasteur
-----------------------
On 26 Nov 2018, at 15:23, Previti <[hidden email]> wrote:

Dear all,

We implemented a system using soft-links to link user-directories to NFS shares in order to import/link datasets that are larger than 2GB (through links to data-libraries) and provide our users with longer term storage space for their analyses.
This worked fine for Galaxy version 17.09...and stopped working for Version 18.05.

An example can be seen here in the folder (the path to this folder will vary between installations): /opt/galaxy/galaxy/database/files

lrwxrwxrwx   1 galaxy  W190-htseq-bioinf      28 Sep 11 12:37 [hidden email] -> /galaxy-clients/w190_galaxy/

Setting the proper rights allows users to link files from the nfs data-share make available under /galaxy-clients (in this case the share  is called "w190_galaxy")
directly into a data-library (with the approriate rights).

Would anybody have an idea why this doesn't work anymore in v18.05?
Is it a galaxy.ini configuration issue?

Thanks for any advise you could provide!

Christopher




--
Dr. Christopher Previti
Genomics and
Proteomics Core Facility
High Throughput Sequencing (W190)
Bioinformatician

German Cancer Research Center (DKFZ)
Foundation under Public Law
Im Neuenheimer Feld 580
69120 Heidelberg
Germany
Room: B2.102 (INF580/TP3)
Phone: +49 6221 42-4661

christopher.previti@...
www.dkfz.de

Management Board: Prof. Dr. Michael Baumann, Prof. Dr. Josef Puchta
VAT-ID No.: DE143293537

Vertraulichkeitshinweis: Diese Nachricht ist ausschließlich für die Personen bestimmt, an die sie adressiert ist.
Sie kann vertrauliche und/oder nur für den/die Empfänger bestimmte Informationen enthalten. Sollten Sie nicht
der bestimmungsgemäße Empfänger sein, kontaktieren Sie bitte den Absender und löschen Sie die Mitteilung.
Jegliche unbefugte Verwendung der Informationen in dieser Nachricht ist untersagt.



___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
 https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
 http://galaxyproject.org/search/


___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/
Reply | Threaded
Open this post in threaded view
|

Re: Using soft-links to NFS-shares as user directories

Previti
Dear Youssef,
Thanks for answering, the target permissions have not changed and are defined via NFS4 ACLs:

A::OWNER@:rwaDxtTnNcCy
A::GROUP@:rwaDxtTnNcy
[hidden email]
[hidden email]

In this case the galaxy user (i.e. the user starting the process) has full access to the share.
In our instance the "user directories" are just symbolic links to the target folders mention above.
When I try to import data from a user directory/NFS4 share into a data library we get the following error: 

"You do not habe any files in your user directory. Use FTP to upload there." (I attached a screenshot)

We still use galaxy.ini to configure our instance, in the galaxy.yml I find the following parameter:

# For security reasons, users may not import any files that actually
  # lie outside of their `user_library_import_dir` (e.g. using symbolic
  # links). A list of directories can be allowed by setting the
  # following option (the list is comma-separated). Be aware that *any*
  # user with library import permissions can import from anywhere in
  # these directories (assuming they are able to create symlinks to
  # them).
  #user_library_import_symlink_whitelist: null

Would implementing this solve my issue?

Cheers,
Christopher



On 11/26/18 4:36 PM, Youssef GHORBAL wrote:
Hello,

What do you mean exactly "doesn't work anymore" ? Do current links no longer appear in data-library ? Do new soft-links can no longer be made available into data-library ? Do you have any error messages ?
By the way the symlink permissions are not valuable, you need to check the target permissions (/galaxy-clients/w190_galaxy/ for instance)

Youssef Ghorbal
IIS / DSI
Institut Pasteur
-----------------------
On 26 Nov 2018, at 15:23, Previti <[hidden email]> wrote:

Dear all,

We implemented a system using soft-links to link user-directories to NFS shares in order to import/link datasets that are larger than 2GB (through links to data-libraries) and provide our users with longer term storage space for their analyses.
This worked fine for Galaxy version 17.09...and stopped working for Version 18.05.

An example can be seen here in the folder (the path to this folder will vary between installations): /opt/galaxy/galaxy/database/files

lrwxrwxrwx   1 galaxy  W190-htseq-bioinf      28 Sep 11 12:37 [hidden email] -> /galaxy-clients/w190_galaxy/

Setting the proper rights allows users to link files from the nfs data-share make available under /galaxy-clients (in this case the share  is called "w190_galaxy")
directly into a data-library (with the approriate rights).

Would anybody have an idea why this doesn't work anymore in v18.05?
Is it a galaxy.ini configuration issue?

Thanks for any advise you could provide!

Christopher




--
Dr. Christopher Previti
Genomics and
Proteomics Core Facility
High Throughput Sequencing (W190)
Bioinformatician

German Cancer Research Center (DKFZ)
Foundation under Public Law
Im Neuenheimer Feld 580
69120 Heidelberg
Germany
Room: B2.102 (INF580/TP3)
Phone: +49 6221 42-4661

christopher.previti@...
www.dkfz.de

Management Board: Prof. Dr. Michael Baumann, Prof. Dr. Josef Puchta
VAT-ID No.: DE143293537

Vertraulichkeitshinweis: Diese Nachricht ist ausschließlich für die Personen bestimmt, an die sie adressiert ist.
Sie kann vertrauliche und/oder nur für den/die Empfänger bestimmte Informationen enthalten. Sollten Sie nicht
der bestimmungsgemäße Empfänger sein, kontaktieren Sie bitte den Absender und löschen Sie die Mitteilung.
Jegliche unbefugte Verwendung der Informationen in dieser Nachricht ist untersagt.


___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
 https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
 http://galaxyproject.org/search/


--
Dr. Christopher Previti
Genomics and
Proteomics Core Facility
High Throughput Sequencing (W190)
Bioinformatician

German Cancer Research Center (DKFZ)
Foundation under Public Law
Im Neuenheimer Feld 580
69120 Heidelberg
Germany
Room: B2.102 (INF580/TP3)
Phone: +49 6221 42-4661

christopher.previti@...
www.dkfz.de

Management Board: Prof. Dr. Michael Baumann, Prof. Dr. Josef Puchta
VAT-ID No.: DE143293537

Vertraulichkeitshinweis: Diese Nachricht ist ausschließlich für die Personen bestimmt, an die sie adressiert ist.
Sie kann vertrauliche und/oder nur für den/die Empfänger bestimmte Informationen enthalten. Sollten Sie nicht
der bestimmungsgemäße Empfänger sein, kontaktieren Sie bitte den Absender und löschen Sie die Mitteilung.
Jegliche unbefugte Verwendung der Informationen in dieser Nachricht ist untersagt.



    

___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/

Data_library_error.png (174K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Using soft-links to NFS-shares as user directories

Youssef  GHORBAL
Hello,

The message you are seeing "You do not have any files in your user directory. Use FTP to upload there" has nothing to do with user_library_import_symlink_whitelist
The symlink control (involving user_library_import_symlink_whitelist) is done a little bit later in the code.

The test code is rather straightforward it does an os.path.exists on the path contact of the user_library_import_dir setting and the user's email address.

Option 1: Galaxy is not resolving the correct email address for your account, I don't see why this may change after upgrade. But it's worth double checking (especially the case)
Option 2: There is an issue with permissions on the symlink target. I've checked python will return false on os.path.exists if the symlink targets a directory not accessible by the user.
For me you need to double check the permissions on your server. Maybe with a shell access as the [hidden email] user, you can try access one of the symlinks and see how it goes. You may also check that the Galaxy service is actually running under the intended identity ([hidden email])

Youssef Ghorbal
IIS / DSI
Institut Pasteur
---------------------------------
On 27 Nov 2018, at 11:43, Previti <[hidden email]> wrote:

Dear Youssef,
Thanks for answering, the target permissions have not changed and are defined via NFS4 ACLs:

A::OWNER@:rwaDxtTnNcCy
A::GROUP@:rwaDxtTnNcy
[hidden email]
[hidden email]

In this case the galaxy user (i.e. the user starting the process) has full access to the share.
In our instance the "user directories" are just symbolic links to the target folders mention above.
When I try to import data from a user directory/NFS4 share into a data library we get the following error: 

"You do not habe any files in your user directory. Use FTP to upload there." (I attached a screenshot)

We still use galaxy.ini to configure our instance, in the galaxy.yml I find the following parameter:

# For security reasons, users may not import any files that actually
  # lie outside of their `user_library_import_dir` (e.g. using symbolic
  # links). A list of directories can be allowed by setting the
  # following option (the list is comma-separated). Be aware that *any*
  # user with library import permissions can import from anywhere in
  # these directories (assuming they are able to create symlinks to
  # them).
  #user_library_import_symlink_whitelist: null

Would implementing this solve my issue?

Cheers,
Christopher



On 11/26/18 4:36 PM, Youssef GHORBAL wrote:
Hello,

What do you mean exactly "doesn't work anymore" ? Do current links no longer appear in data-library ? Do new soft-links can no longer be made available into data-library ? Do you have any error messages ?
By the way the symlink permissions are not valuable, you need to check the target permissions (/galaxy-clients/w190_galaxy/ for instance)

Youssef Ghorbal
IIS / DSI
Institut Pasteur
-----------------------
On 26 Nov 2018, at 15:23, Previti <[hidden email]> wrote:

Dear all,

We implemented a system using soft-links to link user-directories to NFS shares in order to import/link datasets that are larger than 2GB (through links to data-libraries) and provide our users with longer term storage space for their analyses.
This worked fine for Galaxy version 17.09...and stopped working for Version 18.05.

An example can be seen here in the folder (the path to this folder will vary between installations): /opt/galaxy/galaxy/database/files

lrwxrwxrwx   1 galaxy  W190-htseq-bioinf      28 Sep 11 12:37 [hidden email] -> /galaxy-clients/w190_galaxy/

Setting the proper rights allows users to link files from the nfs data-share make available under /galaxy-clients (in this case the share  is called "w190_galaxy")
directly into a data-library (with the approriate rights).

Would anybody have an idea why this doesn't work anymore in v18.05?
Is it a galaxy.ini configuration issue?

Thanks for any advise you could provide!

Christopher




--
Dr. Christopher Previti
Genomics and
Proteomics Core Facility
High Throughput Sequencing (W190)
Bioinformatician

German Cancer Research Center (DKFZ)
Foundation under Public Law
Im Neuenheimer Feld 580
69120 Heidelberg
Germany
Room: B2.102 (INF580/TP3)
Phone: +49 6221 42-4661

christopher.previti@...
www.dkfz.de

Management Board: Prof. Dr. Michael Baumann, Prof. Dr. Josef Puchta
VAT-ID No.: DE143293537

Vertraulichkeitshinweis: Diese Nachricht ist ausschließlich für die Personen bestimmt, an die sie adressiert ist.
Sie kann vertrauliche und/oder nur für den/die Empfänger bestimmte Informationen enthalten. Sollten Sie nicht
der bestimmungsgemäße Empfänger sein, kontaktieren Sie bitte den Absender und löschen Sie die Mitteilung.
Jegliche unbefugte Verwendung der Informationen in dieser Nachricht ist untersagt.


___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
 https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
 http://galaxyproject.org/search/


--
Dr. Christopher Previti
Genomics and
Proteomics Core Facility
High Throughput Sequencing (W190)
Bioinformatician

German Cancer Research Center (DKFZ)
Foundation under Public Law
Im Neuenheimer Feld 580
69120 Heidelberg
Germany
Room: B2.102 (INF580/TP3)
Phone: +49 6221 42-4661

christopher.previti@...
www.dkfz.de

Management Board: Prof. Dr. Michael Baumann, Prof. Dr. Josef Puchta
VAT-ID No.: DE143293537

Vertraulichkeitshinweis: Diese Nachricht ist ausschließlich für die Personen bestimmt, an die sie adressiert ist.
Sie kann vertrauliche und/oder nur für den/die Empfänger bestimmte Informationen enthalten. Sollten Sie nicht
der bestimmungsgemäße Empfänger sein, kontaktieren Sie bitte den Absender und löschen Sie die Mitteilung.
Jegliche unbefugte Verwendung der Informationen in dieser Nachricht ist untersagt.



<Data_library_error.png>


___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/
Reply | Threaded
Open this post in threaded view
|

Re: Using soft-links to NFS-shares as user directories

Previti
Thanks for the infos, I guess we will just have to check everything again.
I wonder, did anything change from 17.09 to 18.05 that would affect the way the system resolves the email-addresses?

We'll let everybody know if we have a solution/cause for the problem.

Cheers and thanks again,
Christopher

On 11/27/18 11:00 PM, Youssef GHORBAL wrote:
Hello,

The message you are seeing "You do not have any files in your user directory. Use FTP to upload there" has nothing to do with user_library_import_symlink_whitelist
The symlink control (involving user_library_import_symlink_whitelist) is done a little bit later in the code.

The test code is rather straightforward it does an os.path.exists on the path contact of the user_library_import_dir setting and the user's email address.

Option 1: Galaxy is not resolving the correct email address for your account, I don't see why this may change after upgrade. But it's worth double checking (especially the case)
Option 2: There is an issue with permissions on the symlink target. I've checked python will return false on os.path.exists if the symlink targets a directory not accessible by the user.
For me you need to double check the permissions on your server. Maybe with a shell access as the [hidden email] user, you can try access one of the symlinks and see how it goes. You may also check that the Galaxy service is actually running under the intended identity ([hidden email])

Youssef Ghorbal
IIS / DSI
Institut Pasteur
---------------------------------
On 27 Nov 2018, at 11:43, Previti <[hidden email]> wrote:

Dear Youssef,
Thanks for answering, the target permissions have not changed and are defined via NFS4 ACLs:

A::OWNER@:rwaDxtTnNcCy
A::GROUP@:rwaDxtTnNcy
[hidden email]
[hidden email]

In this case the galaxy user (i.e. the user starting the process) has full access to the share.
In our instance the "user directories" are just symbolic links to the target folders mention above.
When I try to import data from a user directory/NFS4 share into a data library we get the following error: 

"You do not habe any files in your user directory. Use FTP to upload there." (I attached a screenshot)

We still use galaxy.ini to configure our instance, in the galaxy.yml I find the following parameter:

# For security reasons, users may not import any files that actually
  # lie outside of their `user_library_import_dir` (e.g. using symbolic
  # links). A list of directories can be allowed by setting the
  # following option (the list is comma-separated). Be aware that *any*
  # user with library import permissions can import from anywhere in
  # these directories (assuming they are able to create symlinks to
  # them).
  #user_library_import_symlink_whitelist: null

Would implementing this solve my issue?

Cheers,
Christopher



On 11/26/18 4:36 PM, Youssef GHORBAL wrote:
Hello,

What do you mean exactly "doesn't work anymore" ? Do current links no longer appear in data-library ? Do new soft-links can no longer be made available into data-library ? Do you have any error messages ?
By the way the symlink permissions are not valuable, you need to check the target permissions (/galaxy-clients/w190_galaxy/ for instance)

Youssef Ghorbal
IIS / DSI
Institut Pasteur
-----------------------
On 26 Nov 2018, at 15:23, Previti <[hidden email]> wrote:

Dear all,

We implemented a system using soft-links to link user-directories to NFS shares in order to import/link datasets that are larger than 2GB (through links to data-libraries) and provide our users with longer term storage space for their analyses.
This worked fine for Galaxy version 17.09...and stopped working for Version 18.05.

An example can be seen here in the folder (the path to this folder will vary between installations): /opt/galaxy/galaxy/database/files

lrwxrwxrwx   1 galaxy  W190-htseq-bioinf      28 Sep 11 12:37 [hidden email] -> /galaxy-clients/w190_galaxy/

Setting the proper rights allows users to link files from the nfs data-share make available under /galaxy-clients (in this case the share  is called "w190_galaxy")
directly into a data-library (with the approriate rights).

Would anybody have an idea why this doesn't work anymore in v18.05?
Is it a galaxy.ini configuration issue?

Thanks for any advise you could provide!

Christopher




--
Dr. Christopher Previti
Genomics and
Proteomics Core Facility
High Throughput Sequencing (W190)
Bioinformatician

German Cancer Research Center (DKFZ)
Foundation under Public Law
Im Neuenheimer Feld 580
69120 Heidelberg
Germany
Room: B2.102 (INF580/TP3)
Phone: +49 6221 42-4661

christopher.previti@...
www.dkfz.de

Management Board: Prof. Dr. Michael Baumann, Prof. Dr. Josef Puchta
VAT-ID No.: DE143293537

Vertraulichkeitshinweis: Diese Nachricht ist ausschließlich für die Personen bestimmt, an die sie adressiert ist.
Sie kann vertrauliche und/oder nur für den/die Empfänger bestimmte Informationen enthalten. Sollten Sie nicht
der bestimmungsgemäße Empfänger sein, kontaktieren Sie bitte den Absender und löschen Sie die Mitteilung.
Jegliche unbefugte Verwendung der Informationen in dieser Nachricht ist untersagt.


___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
 https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
 http://galaxyproject.org/search/


--
Dr. Christopher Previti
Genomics and
Proteomics Core Facility
High Throughput Sequencing (W190)
Bioinformatician

German Cancer Research Center (DKFZ)
Foundation under Public Law
Im Neuenheimer Feld 580
69120 Heidelberg
Germany
Room: B2.102 (INF580/TP3)
Phone: +49 6221 42-4661

christopher.previti@...
www.dkfz.de

Management Board: Prof. Dr. Michael Baumann, Prof. Dr. Josef Puchta
VAT-ID No.: DE143293537

Vertraulichkeitshinweis: Diese Nachricht ist ausschließlich für die Personen bestimmt, an die sie adressiert ist.
Sie kann vertrauliche und/oder nur für den/die Empfänger bestimmte Informationen enthalten. Sollten Sie nicht
der bestimmungsgemäße Empfänger sein, kontaktieren Sie bitte den Absender und löschen Sie die Mitteilung.
Jegliche unbefugte Verwendung der Informationen in dieser Nachricht ist untersagt.


<Data_library_error.png>


--
Dr. Christopher Previti
Genomics and
Proteomics Core Facility
High Throughput Sequencing (W190)
Bioinformatician

German Cancer Research Center (DKFZ)
Foundation under Public Law
Im Neuenheimer Feld 580
69120 Heidelberg
Germany
Room: B2.102 (INF580/TP3)
Phone: +49 6221 42-4661

christopher.previti@...
www.dkfz.de

Management Board: Prof. Dr. Michael Baumann, Prof. Dr. Josef Puchta
VAT-ID No.: DE143293537

Vertraulichkeitshinweis: Diese Nachricht ist ausschließlich für die Personen bestimmt, an die sie adressiert ist.
Sie kann vertrauliche und/oder nur für den/die Empfänger bestimmte Informationen enthalten. Sollten Sie nicht
der bestimmungsgemäße Empfänger sein, kontaktieren Sie bitte den Absender und löschen Sie die Mitteilung.
Jegliche unbefugte Verwendung der Informationen in dieser Nachricht ist untersagt.



    

___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/
Reply | Threaded
Open this post in threaded view
|

Re: Using soft-links to NFS-shares as user directories

Previti
Dear all,
I got it to work again.
I just added:

user_library_import_symlink_whitelist = "symlink_target_folder"

to the galaxy.ini (we'll switch to the .yml next update).
It got recognized and the symlinks can be used again.

Best regards,
Christopher

On 11/28/18 9:25 AM, Previti wrote:
Thanks for the infos, I guess we will just have to check everything again.
I wonder, did anything change from 17.09 to 18.05 that would affect the way the system resolves the email-addresses?

We'll let everybody know if we have a solution/cause for the problem.

Cheers and thanks again,
Christopher

On 11/27/18 11:00 PM, Youssef GHORBAL wrote:
Hello,

The message you are seeing "You do not have any files in your user directory. Use FTP to upload there" has nothing to do with user_library_import_symlink_whitelist
The symlink control (involving user_library_import_symlink_whitelist) is done a little bit later in the code.

The test code is rather straightforward it does an os.path.exists on the path contact of the user_library_import_dir setting and the user's email address.

Option 1: Galaxy is not resolving the correct email address for your account, I don't see why this may change after upgrade. But it's worth double checking (especially the case)
Option 2: There is an issue with permissions on the symlink target. I've checked python will return false on os.path.exists if the symlink targets a directory not accessible by the user.
For me you need to double check the permissions on your server. Maybe with a shell access as the [hidden email] user, you can try access one of the symlinks and see how it goes. You may also check that the Galaxy service is actually running under the intended identity ([hidden email])

Youssef Ghorbal
IIS / DSI
Institut Pasteur
---------------------------------
On 27 Nov 2018, at 11:43, Previti <[hidden email]> wrote:

Dear Youssef,
Thanks for answering, the target permissions have not changed and are defined via NFS4 ACLs:

A::OWNER@:rwaDxtTnNcCy
A::GROUP@:rwaDxtTnNcy
[hidden email]
[hidden email]

In this case the galaxy user (i.e. the user starting the process) has full access to the share.
In our instance the "user directories" are just symbolic links to the target folders mention above.
When I try to import data from a user directory/NFS4 share into a data library we get the following error: 

"You do not habe any files in your user directory. Use FTP to upload there." (I attached a screenshot)

We still use galaxy.ini to configure our instance, in the galaxy.yml I find the following parameter:

# For security reasons, users may not import any files that actually
  # lie outside of their `user_library_import_dir` (e.g. using symbolic
  # links). A list of directories can be allowed by setting the
  # following option (the list is comma-separated). Be aware that *any*
  # user with library import permissions can import from anywhere in
  # these directories (assuming they are able to create symlinks to
  # them).
  #user_library_import_symlink_whitelist: null

Would implementing this solve my issue?

Cheers,
Christopher



On 11/26/18 4:36 PM, Youssef GHORBAL wrote:
Hello,

What do you mean exactly "doesn't work anymore" ? Do current links no longer appear in data-library ? Do new soft-links can no longer be made available into data-library ? Do you have any error messages ?
By the way the symlink permissions are not valuable, you need to check the target permissions (/galaxy-clients/w190_galaxy/ for instance)

Youssef Ghorbal
IIS / DSI
Institut Pasteur
-----------------------
On 26 Nov 2018, at 15:23, Previti <[hidden email]> wrote:

Dear all,

We implemented a system using soft-links to link user-directories to NFS shares in order to import/link datasets that are larger than 2GB (through links to data-libraries) and provide our users with longer term storage space for their analyses.
This worked fine for Galaxy version 17.09...and stopped working for Version 18.05.

An example can be seen here in the folder (the path to this folder will vary between installations): /opt/galaxy/galaxy/database/files

lrwxrwxrwx   1 galaxy  W190-htseq-bioinf      28 Sep 11 12:37 [hidden email] -> /galaxy-clients/w190_galaxy/

Setting the proper rights allows users to link files from the nfs data-share make available under /galaxy-clients (in this case the share  is called "w190_galaxy")
directly into a data-library (with the approriate rights).

Would anybody have an idea why this doesn't work anymore in v18.05?
Is it a galaxy.ini configuration issue?

Thanks for any advise you could provide!

Christopher




--
Dr. Christopher Previti
Genomics and
Proteomics Core Facility
High Throughput Sequencing (W190)
Bioinformatician

German Cancer Research Center (DKFZ)
Foundation under Public Law
Im Neuenheimer Feld 580
69120 Heidelberg
Germany
Room: B2.102 (INF580/TP3)
Phone: +49 6221 42-4661

christopher.previti@...
www.dkfz.de

Management Board: Prof. Dr. Michael Baumann, Prof. Dr. Josef Puchta
VAT-ID No.: DE143293537

Vertraulichkeitshinweis: Diese Nachricht ist ausschließlich für die Personen bestimmt, an die sie adressiert ist.
Sie kann vertrauliche und/oder nur für den/die Empfänger bestimmte Informationen enthalten. Sollten Sie nicht
der bestimmungsgemäße Empfänger sein, kontaktieren Sie bitte den Absender und löschen Sie die Mitteilung.
Jegliche unbefugte Verwendung der Informationen in dieser Nachricht ist untersagt.


___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
 https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
 http://galaxyproject.org/search/


--
Dr. Christopher Previti
Genomics and
Proteomics Core Facility
High Throughput Sequencing (W190)
Bioinformatician

German Cancer Research Center (DKFZ)
Foundation under Public Law
Im Neuenheimer Feld 580
69120 Heidelberg
Germany
Room: B2.102 (INF580/TP3)
Phone: +49 6221 42-4661

christopher.previti@...
www.dkfz.de

Management Board: Prof. Dr. Michael Baumann, Prof. Dr. Josef Puchta
VAT-ID No.: DE143293537

Vertraulichkeitshinweis: Diese Nachricht ist ausschließlich für die Personen bestimmt, an die sie adressiert ist.
Sie kann vertrauliche und/oder nur für den/die Empfänger bestimmte Informationen enthalten. Sollten Sie nicht
der bestimmungsgemäße Empfänger sein, kontaktieren Sie bitte den Absender und löschen Sie die Mitteilung.
Jegliche unbefugte Verwendung der Informationen in dieser Nachricht ist untersagt.


<Data_library_error.png>


--
Dr. Christopher Previti
Genomics and
Proteomics Core Facility
High Throughput Sequencing (W190)
Bioinformatician

German Cancer Research Center (DKFZ)
Foundation under Public Law
Im Neuenheimer Feld 580
69120 Heidelberg
Germany
Room: B2.102 (INF580/TP3)
Phone: +49 6221 42-4661

christopher.previti@...
www.dkfz.de

Management Board: Prof. Dr. Michael Baumann, Prof. Dr. Josef Puchta
VAT-ID No.: DE143293537

Vertraulichkeitshinweis: Diese Nachricht ist ausschließlich für die Personen bestimmt, an die sie adressiert ist.
Sie kann vertrauliche und/oder nur für den/die Empfänger bestimmte Informationen enthalten. Sollten Sie nicht
der bestimmungsgemäße Empfänger sein, kontaktieren Sie bitte den Absender und löschen Sie die Mitteilung.
Jegliche unbefugte Verwendung der Informationen in dieser Nachricht ist untersagt.



___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/

--
Dr. Christopher Previti
Genomics and
Proteomics Core Facility
High Throughput Sequencing (W190)
Bioinformatician

German Cancer Research Center (DKFZ)
Foundation under Public Law
Im Neuenheimer Feld 580
69120 Heidelberg
Germany
Room: B2.102 (INF580/TP3)
Phone: +49 6221 42-4661

christopher.previti@...
www.dkfz.de

Management Board: Prof. Dr. Michael Baumann, Prof. Dr. Josef Puchta
VAT-ID No.: DE143293537

Vertraulichkeitshinweis: Diese Nachricht ist ausschließlich für die Personen bestimmt, an die sie adressiert ist.
Sie kann vertrauliche und/oder nur für den/die Empfänger bestimmte Informationen enthalten. Sollten Sie nicht
der bestimmungsgemäße Empfänger sein, kontaktieren Sie bitte den Absender und löschen Sie die Mitteilung.
Jegliche unbefugte Verwendung der Informationen in dieser Nachricht ist untersagt.



    

___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/