Python modules not found when starting with init system

classic Classic list List threaded Threaded
5 messages Options
| Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Python modules not found when starting with init system

Maximilian Friedersdorff
Hi All,

when my local galaxy is started using the init system (systemd on Ubuntu
16.04 in this instance).  Some tools will fail, indicating that some
python module do not exist.  This is the result of uploading a 2KB plain
text file:

Traceback (most recent call last):
  File "/home/galaxy/galaxy/tools/data_source/upload.py", line 20, in <module>
    from galaxy.datatypes import sniff
  File "/home/galaxy/galaxy/lib/galaxy/datatypes/sniff.py", line 31, in <module>
    from galaxy.datatypes.binary import Binary
  File "/home/galaxy/galaxy/lib/galaxy/datatypes/binary.py", line 14, in <module>
    import pysam
ImportError: No module named pysam
Traceback (most recent call last):
  File "/home/galaxy/galaxy/database/jobs_directory/000/39/set_metadata_EI05GD.py", line 1, in <module>
    from galaxy_ext.metadata.set_metadata import set_metadata; set_metadata()
  File "/home/galaxy/galaxy/lib/galaxy_ext/metadata/set_metadata.py", line 23, in <module>
    from sqlalchemy.orm import clear_mappers
ImportError: No module named sqlalchemy.orm

When the same galaxy install is started directly using the 'run.sh'
script in the install directory, everything works fine.

I've verified that I can manually import pysam and sqlalchemy.orm if I
execute the python interpreter located at <install_dir>/.venv/bin/python
directly.

Any ideas?

I've attached the init script.

TL;DR

Galaxy cannot import some python modules when started by the init
system. No idea why.

Galaxy version: 17.05
OS: Ubuntu 16.04


Many Thanks

Max


--------------------------------------------------------------------
Un o’r 4 prifysgol uchaf yn y DU a’r orau yng Nghymru am fodlonrwydd myfyrwyr.
(Arolwg Cenedlaethol y Myfyrwyr 2016)
www.aber.ac.uk

Top 4 UK university and best in Wales for student satisfaction
(National Student Survey 2016)
www.aber.ac.uk

___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/

galaxy (1K) Download Attachment
| Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Python modules not found when starting with init system

Gudrun Amedick
Hi,

i ran into that too (on debian 8). I'm not that familiar with virtual environments but this is what I found out.

It looks like the virtual environment doesn't get activated properly in the shipped ini.d-file (it is the shipped one, right?). 
If I add ". /path/to/.venv/bin/activate" to it, everything runs just fine. "run.sh" has this line. The init.d-file hasn't. It just calls the .venv-
python directly. It seems like that's not fully equivalent to activating .venv and running python in there. I don't really know why. I'm guessing
PATH-magic.

Hope that helps a bit.

        Gudrun Amedick

--
Gudrun Amedick
   IT-Systemtechnik

UNIVERSITÄT ZU LÜBECK
    IT-Service-Center
    
    Ratzeburger Allee 160
    23562 Lübeck
    Tel +49 451 3101 2035
    Fax +49 451 3101 2004
    [hidden email]
    www.itsc.uni-luebeck.de

Am Dienstag, den 27.06.2017, 12:54 +0100 schrieb Maximilian Friedersdorff:

> Hi All,
>
> when my local galaxy is started using the init system (systemd on Ubuntu
> 16.04 in this instance).  Some tools will fail, indicating that some
> python module do not exist.  This is the result of uploading a 2KB plain
> text file:
>
> Traceback (most recent call last):
>   File "/home/galaxy/galaxy/tools/data_source/upload.py", line 20, in <module>
>     from galaxy.datatypes import sniff
>   File "/home/galaxy/galaxy/lib/galaxy/datatypes/sniff.py", line 31, in <module>
>     from galaxy.datatypes.binary import Binary
>   File "/home/galaxy/galaxy/lib/galaxy/datatypes/binary.py", line 14, in <module>
>     import pysam
> ImportError: No module named pysam
> Traceback (most recent call last):
>   File "/home/galaxy/galaxy/database/jobs_directory/000/39/set_metadata_EI05GD.py", line 1, in <module>
>     from galaxy_ext.metadata.set_metadata import set_metadata; set_metadata()
>   File "/home/galaxy/galaxy/lib/galaxy_ext/metadata/set_metadata.py", line 23, in <module>
>     from sqlalchemy.orm import clear_mappers
> ImportError: No module named sqlalchemy.orm
>
> When the same galaxy install is started directly using the 'run.sh'
> script in the install directory, everything works fine.
>
> I've verified that I can manually import pysam and sqlalchemy.orm if I
> execute the python interpreter located at <install_dir>/.venv/bin/python
> directly.
>
> Any ideas?
>
> I've attached the init script.
>
> TL;DR
>
> Galaxy cannot import some python modules when started by the init
> system. No idea why.
>
> Galaxy version: 17.05
> OS: Ubuntu 16.04
>
>
> Many Thanks
>
> Max
>
>
> --------------------------------------------------------------------
> Un o’r 4 prifysgol uchaf yn y DU a’r orau yng Nghymru am fodlonrwydd myfyrwyr.
> (Arolwg Cenedlaethol y Myfyrwyr 2016)
> www.aber.ac.uk
>
> Top 4 UK university and best in Wales for student satisfaction
> (National Student Survey 2016)
> www.aber.ac.uk
> ___________________________________________________________
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>   https://lists.galaxyproject.org/
>
> To search Galaxy mailing lists use the unified search at:
>   http://galaxyproject.org/search/
___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/

smime.p7s (8K) Download Attachment
| Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Python modules not found when starting with init system

Maximilian Friedersdorff
On Tue, Jun 27, 2017 at 03:41:27PM +0200, Gudrun Amedick wrote:

>Hi,
>
>i ran into that too (on debian 8). I'm not that familiar with virtual environments but this is what I found out.
>
>It looks like the virtual environment doesn't get activated properly in the shipped ini.d-file (it is the shipped one, right?).
>If I add ". /path/to/.venv/bin/activate" to it, everything runs just fine. "run.sh" has this line. The init.d-file hasn't. It just calls the .venv-
>python directly. It seems like that's not fully equivalent to activating .venv and running python in there. I don't really know why. I'm guessing
>PATH-magic.
>
>Hope that helps a bit.
>
>       Gudrun Amedick

Hi,

The thing is that running the python interpreter in the virtual
environment _should_ take care of that anyway. The provided debian init
script even says as much:  "A simple way to activate this virtualenv is
to use the python interpreter in <GALAXY_DIR>/.venv".

In any case, explicitly activating the virtualenv solves the problem.
Thanks for the pointer Gudrun.

Maybe a dev can chime in here.

Many Thanks

Max


--------------------------------------------------------------------
Un o’r 4 prifysgol uchaf yn y DU a’r orau yng Nghymru am fodlonrwydd myfyrwyr.
(Arolwg Cenedlaethol y Myfyrwyr 2016)
www.aber.ac.uk

Top 4 UK university and best in Wales for student satisfaction
(National Student Survey 2016)
www.aber.ac.uk
___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/
| Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Python modules not found when starting with init system

Gudrun Amedick
Hi,

I know. To my understaunding of virtual environments, this shouldn't make a difference. As I said, I don't really get it. It looks like Python tries
to find pysam outside of the virtual environment for some reason, that's why I'm guessing PATH.

I'm curious though, and if someone knows why this solution works, I'd be happy to hear it too.

Glad I could help.

        Gudrun Amedick


Gudrun Amedick
   IT-Systemtechnik

UNIVERSITÄT ZU LÜBECK
    IT-Service-Center
    
    Ratzeburger Allee 160
    23562 Lübeck
    Tel +49 451 3101 2035
    Fax +49 451 3101 2004
    [hidden email]
    www.itsc.uni-luebeck.de

Am Mittwoch, den 28.06.2017, 10:59 +0100 schrieb Maximilian Friedersdorff:

> On Tue, Jun 27, 2017 at 03:41:27PM +0200, Gudrun Amedick wrote:
> >
> > Hi,
> >
> > i ran into that too (on debian 8). I'm not that familiar with virtual environments but this is what I found out.
> >
> > It looks like the virtual environment doesn't get activated properly in the shipped ini.d-file (it is the shipped one, right?).
> > If I add ". /path/to/.venv/bin/activate" to it, everything runs just fine. "run.sh" has this line. The init.d-file hasn't. It just calls the
> > .venv-
> > python directly. It seems like that's not fully equivalent to activating .venv and running python in there. I don't really know why. I'm guessing
> > PATH-magic.
> >
> > Hope that helps a bit.
> >
> >       Gudrun Amedick
> Hi,
>
> The thing is that running the python interpreter in the virtual
> environment _should_ take care of that anyway. The provided debian init
> script even says as much:  "A simple way to activate this virtualenv is
> to use the python interpreter in <GALAXY_DIR>/.venv".
>
> In any case, explicitly activating the virtualenv solves the problem.
> Thanks for the pointer Gudrun.
>
> Maybe a dev can chime in here.
>
> Many Thanks
>
> Max
>
>
> --------------------------------------------------------------------
> Un o’r 4 prifysgol uchaf yn y DU a’r orau yng Nghymru am fodlonrwydd myfyrwyr.
> (Arolwg Cenedlaethol y Myfyrwyr 2016)
> www.aber.ac.uk
>
> Top 4 UK university and best in Wales for student satisfaction
> (National Student Survey 2016)
> www.aber.ac.uk
___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/

smime.p7s (8K) Download Attachment
| Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Python modules not found when starting with init system

Abbott, James
Hi,

I've just resolved a similar problem running uwsgi under supervisord, which turned out to be a typo setting the path environment variable in the supervisord config file.  Most things worked ok but some things produced errors finding pysam and sqlalchemy (since the system python was being used in preference to the galaxy venv), while jobs would mysteriously fail with a 'file not found' error.

Took some tracking down...

James

--
Dr. James Abbott
Lead Bioinformatician
Bioinformatician Data Science Group
Imperial College London


From: galaxy-dev <[hidden email]> on behalf of Gudrun Amedick <[hidden email]>
Sent: Wednesday, June 28, 2017 1:49:44 PM
To: Maximilian Friedersdorff
Cc: [hidden email]
Subject: Re: [galaxy-dev] Python modules not found when starting with init system
 
Hi,

I know. To my understaunding of virtual environments, this shouldn't make a difference. As I said, I don't really get it. It looks like Python tries
to find pysam outside of the virtual environment for some reason, that's why I'm guessing PATH.

I'm curious though, and if someone knows why this solution works, I'd be happy to hear it too.

Glad I could help.

        Gudrun Amedick


Gudrun Amedick
   IT-Systemtechnik

UNIVERSITÄT ZU LÜBECK
    IT-Service-Center
    
    Ratzeburger Allee 160
    23562 Lübeck
    Tel +49 451 3101 2035
    Fax +49 451 3101 2004
    [hidden email]
    www.itsc.uni-luebeck.de

Am Mittwoch, den 28.06.2017, 10:59 +0100 schrieb Maximilian Friedersdorff:
> On Tue, Jun 27, 2017 at 03:41:27PM +0200, Gudrun Amedick wrote:
> >
> > Hi,
> >
> > i ran into that too (on debian 8). I'm not that familiar with virtual environments but this is what I found out.
> >
> > It looks like the virtual environment doesn't get activated properly in the shipped ini.d-file (it is the shipped one, right?).
> > If I add ". /path/to/.venv/bin/activate" to it, everything runs just fine. "run.sh" has this line. The init.d-file hasn't. It just calls the
> > .venv-
> > python directly. It seems like that's not fully equivalent to activating .venv and running python in there. I don't really know why. I'm guessing
> > PATH-magic.
> >
> > Hope that helps a bit.
> >
> >       Gudrun Amedick
> Hi,
>
> The thing is that running the python interpreter in the virtual
> environment _should_ take care of that anyway. The provided debian init
> script even says as much:  "A simple way to activate this virtualenv is
> to use the python interpreter in <GALAXY_DIR>/.venv".
>
> In any case, explicitly activating the virtualenv solves the problem.
> Thanks for the pointer Gudrun.
>
> Maybe a dev can chime in here.
>
> Many Thanks
>
> Max
>
>
> --------------------------------------------------------------------
> Un o’r 4 prifysgol uchaf yn y DU a’r orau yng Nghymru am fodlonrwydd myfyrwyr.
> (Arolwg Cenedlaethol y Myfyrwyr 2016)
> www.aber.ac.uk
>
> Top 4 UK university and best in Wales for student satisfaction
> (National Student Survey 2016)
> www.aber.ac.uk

___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/
Loading...