ImportError: No module named galaxy_utils.sequence.fastq

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

ImportError: No module named galaxy_utils.sequence.fastq

juliech
This post was updated on .
Hi,
I've this error message when I run groomer.

EDIT : I've been tested to run groomer alone and it works fine. But in my case, I launch groomer via an other tool in the xml file with this command :

<command>
                #if $fastq_format == "illumina"
        blabla_No_python_command
                #else
        python /data/galaxy-dist/tools/fastq/fastq_groomer.py 'tmp.fastqcssanger' 'cssanger' '$output' 'illumina' 'ascii' 'dont_summarize_input'
                #end if
</command>

My local instance of galaxy runs on a python virtualenv (python 2.7)
and I have make the latest update with : hg pull -u.

I've verified and this library is present in my
galaxy-dist/lib/galaxy_utils/sequence directory.

I've found no other idea than the update on this forum to run groomer
correctly, any other idea ?

Thanks.
Julie
___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
Reply | Threaded
Open this post in threaded view
|

Re: ImportError: No module named galaxy_utils.sequence.fastq

John Chilton-4
Hello Julie,

  This is odd - sorry.

  I might be worth uninstalling and re-installing this in the tool
shed admin options - in particular ensure you install tool
dependencies and that this installation process succeeds.

  Otherwise -  it might be worth trying to put some debug statements
in the fastq groomer tool to try to track down the problem. Here
(https://gist.github.com/jmchilton/2d61f7ab81d99a0f02fc) is a diff to
have that tool write out the run-time value  PYTHONPATH to Galaxy's
home directory - it might be worth running a job, finding the output,
setting PYTHONPATH to that value on the compute node and attempting to
just import it directly.

cd $HOME; python -c "import galaxy.sequences.fastq"

  Out of curiosity - what job runner are you using (local, DRMAA, etc...)?

-John

On Tue, Jul 1, 2014 at 4:50 AM, julie dubois <[hidden email]> wrote:

> Hi,
> I've this error message when I run groomer.
>
> My local instance of galaxy runs on a python virtualenv (python 2.7)
> and I have make the latest update with : hg pull -u.
>
> I've verified and this library is present in my
> galaxy-dist/lib/galaxy_utils/sequence directory.
>
> I've found no other idea than the update on this forum to run groomer
> correctly, any other idea ?
>
> Thanks.
> Julie
> ___________________________________________________________
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>   http://lists.bx.psu.edu/
>
> To search Galaxy mailing lists use the unified search at:
>   http://galaxyproject.org/search/mailinglists/
___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
Reply | Threaded
Open this post in threaded view
|

Re: ImportError: No module named galaxy_utils.sequence.fastq

juliech
Hi John,

As I specified it in the "edit" of my first message, I have this error
only if I run groomer with a command line in another xml file
surrounding by other command. When I run groomer alone with the
original xml, it works fine, so no problem with the PYTHONPATH.
 But when I use this :
<command>
                #if $fastq_format == "illumina"
        blabla_No_python_command
                #else
        python /data/galaxy-dist/tools/fastq/fastq_groomer.py
'tmp.fastqcssanger' 'cssanger' '$output' 'illumina' 'ascii'
'dont_summarize_input'
                #end if
</command>

It doesn't work.

I've checked the files contained in job_working_directory. A shell
script is generated for my job and it begin with these lines :

#!/bin/sh

GALAXY_SLOTS="1"; export GALAXY_SLOTS;
export GALAXY_SLOTS
GALAXY_LIB="None"
if [ "$GALAXY_LIB" != "None" ]; then
    if [ -n "$PYTHONPATH" ]; then
        PYTHONPATH="$GALAXY_LIB:$PYTHONPATH"
    else
        PYTHONPATH="$GALAXY_LIB"
    fi
    export PYTHONPATH
fi



So, with the run of groomer alone the line GALAXY_LIB is equal to
"/data/galaxy/galaxy-dist/lib", but here with the run of groomer with
a command line with another job the GALAXY_LIB is set to "None"!!!
I don't understand why.

For the moment, to surround the problem, I've explicitly specified the
GALAXY_LIB path int the xml command before the command line wich run
python fastq_groomer.py and it works but I'm aware that is not
beautiful.

Finally I'm using a local jobrunner.

Thanks for your help
Julie

P.S : we use such command because we build a workflow where the
quality format of the fastq must be invisible for the users.

2014-07-03 19:48 GMT+02:00 John Chilton <[hidden email]>:

> Hello Julie,
>
>   This is odd - sorry.
>
>   I might be worth uninstalling and re-installing this in the tool
> shed admin options - in particular ensure you install tool
> dependencies and that this installation process succeeds.
>
>   Otherwise -  it might be worth trying to put some debug statements
> in the fastq groomer tool to try to track down the problem. Here
> (https://gist.github.com/jmchilton/2d61f7ab81d99a0f02fc) is a diff to
> have that tool write out the run-time value  PYTHONPATH to Galaxy's
> home directory - it might be worth running a job, finding the output,
> setting PYTHONPATH to that value on the compute node and attempting to
> just import it directly.
>
> cd $HOME; python -c "import galaxy.sequences.fastq"
>
>   Out of curiosity - what job runner are you using (local, DRMAA, etc...)?
>
> -John
>
> On Tue, Jul 1, 2014 at 4:50 AM, julie dubois <[hidden email]> wrote:
>> Hi,
>> I've this error message when I run groomer.
>>
>> My local instance of galaxy runs on a python virtualenv (python 2.7)
>> and I have make the latest update with : hg pull -u.
>>
>> I've verified and this library is present in my
>> galaxy-dist/lib/galaxy_utils/sequence directory.
>>
>> I've found no other idea than the update on this forum to run groomer
>> correctly, any other idea ?
>>
>> Thanks.
>> Julie
>> ___________________________________________________________
>> Please keep all replies on the list by using "reply all"
>> in your mail client.  To manage your subscriptions to this
>> and other Galaxy lists, please use the interface at:
>>   http://lists.bx.psu.edu/
>>
>> To search Galaxy mailing lists use the unified search at:
>>   http://galaxyproject.org/search/mailinglists/
___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
Reply | Threaded
Open this post in threaded view
|

Re: ImportError: No module named galaxy_utils.sequence.fastq

John Chilton-4
Ahhh - I get it now. Galaxy only adds itself to the PYTHONPATH if it
can infer it is running a Python script (i.e. the command block starts
with python or a Python interpreter is used).

If you are building a tool for the tool shed - it would be best to
setup a tool dependency on the galaxy_sequence_utils package - this
would force every execution of the tool to have these utilities on
your PYTHONPATH. But since you are hard-encoding paths - it looks like
you are tailoring your Galaxy instance to your local environment. In
this case an easy hack is to just setup that dependency globally for
the system:

sudo easy_install
http://depot.galaxyproject.org/package/source/galaxy_sequence_utils/galaxy_sequence_utils-1.0.0.tgz

If you are setting up Galaxy env.sh files this can be installed into a
virtualenv instead - which is a better practice - let me know if you
are interested and I can send details.

-John




On Mon, Jul 7, 2014 at 4:02 AM, julie dubois <[hidden email]> wrote:

> Hi John,
>
> As I specified it in the "edit" of my first message, I have this error
> only if I run groomer with a command line in another xml file
> surrounding by other command. When I run groomer alone with the
> original xml, it works fine, so no problem with the PYTHONPATH.
>  But when I use this :
> <command>
>                 #if $fastq_format == "illumina"
>         blabla_No_python_command
>                 #else
>         python /data/galaxy-dist/tools/fastq/fastq_groomer.py
> 'tmp.fastqcssanger' 'cssanger' '$output' 'illumina' 'ascii'
> 'dont_summarize_input'
>                 #end if
> </command>
>
> It doesn't work.
>
> I've checked the files contained in job_working_directory. A shell
> script is generated for my job and it begin with these lines :
>
> #!/bin/sh
>
> GALAXY_SLOTS="1"; export GALAXY_SLOTS;
> export GALAXY_SLOTS
> GALAXY_LIB="None"
> if [ "$GALAXY_LIB" != "None" ]; then
>     if [ -n "$PYTHONPATH" ]; then
>         PYTHONPATH="$GALAXY_LIB:$PYTHONPATH"
>     else
>         PYTHONPATH="$GALAXY_LIB"
>     fi
>     export PYTHONPATH
> fi
>
>
>
> So, with the run of groomer alone the line GALAXY_LIB is equal to
> "/data/galaxy/galaxy-dist/lib", but here with the run of groomer with
> a command line with another job the GALAXY_LIB is set to "None"!!!
> I don't understand why.
>
> For the moment, to surround the problem, I've explicitly specified the
> GALAXY_LIB path int the xml command before the command line wich run
> python fastq_groomer.py and it works but I'm aware that is not
> beautiful.
>
> Finally I'm using a local jobrunner.
>
> Thanks for your help
> Julie
>
> P.S : we use such command because we build a workflow where the
> quality format of the fastq must be invisible for the users.
>
> 2014-07-03 19:48 GMT+02:00 John Chilton <[hidden email]>:
>> Hello Julie,
>>
>>   This is odd - sorry.
>>
>>   I might be worth uninstalling and re-installing this in the tool
>> shed admin options - in particular ensure you install tool
>> dependencies and that this installation process succeeds.
>>
>>   Otherwise -  it might be worth trying to put some debug statements
>> in the fastq groomer tool to try to track down the problem. Here
>> (https://gist.github.com/jmchilton/2d61f7ab81d99a0f02fc) is a diff to
>> have that tool write out the run-time value  PYTHONPATH to Galaxy's
>> home directory - it might be worth running a job, finding the output,
>> setting PYTHONPATH to that value on the compute node and attempting to
>> just import it directly.
>>
>> cd $HOME; python -c "import galaxy.sequences.fastq"
>>
>>   Out of curiosity - what job runner are you using (local, DRMAA, etc...)?
>>
>> -John
>>
>> On Tue, Jul 1, 2014 at 4:50 AM, julie dubois <[hidden email]> wrote:
>>> Hi,
>>> I've this error message when I run groomer.
>>>
>>> My local instance of galaxy runs on a python virtualenv (python 2.7)
>>> and I have make the latest update with : hg pull -u.
>>>
>>> I've verified and this library is present in my
>>> galaxy-dist/lib/galaxy_utils/sequence directory.
>>>
>>> I've found no other idea than the update on this forum to run groomer
>>> correctly, any other idea ?
>>>
>>> Thanks.
>>> Julie
>>> ___________________________________________________________
>>> Please keep all replies on the list by using "reply all"
>>> in your mail client.  To manage your subscriptions to this
>>> and other Galaxy lists, please use the interface at:
>>>   http://lists.bx.psu.edu/
>>>
>>> To search Galaxy mailing lists use the unified search at:
>>>   http://galaxyproject.org/search/mailinglists/
___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
Reply | Threaded
Open this post in threaded view
|

Re: ImportError: No module named galaxy_utils.sequence.fastq

juliech
Thanks!

Actually it's not for the toolshed, for the moment...
But I'm very intereseted by details to install it into a virtualenv.
Thank you.

Julie

2014-07-07 16:16 GMT+02:00 John Chilton <[hidden email]>:

> Ahhh - I get it now. Galaxy only adds itself to the PYTHONPATH if it
> can infer it is running a Python script (i.e. the command block starts
> with python or a Python interpreter is used).
>
> If you are building a tool for the tool shed - it would be best to
> setup a tool dependency on the galaxy_sequence_utils package - this
> would force every execution of the tool to have these utilities on
> your PYTHONPATH. But since you are hard-encoding paths - it looks like
> you are tailoring your Galaxy instance to your local environment. In
> this case an easy hack is to just setup that dependency globally for
> the system:
>
> sudo easy_install
> http://depot.galaxyproject.org/package/source/galaxy_sequence_utils/galaxy_sequence_utils-1.0.0.tgz
>
> If you are setting up Galaxy env.sh files this can be installed into a
> virtualenv instead - which is a better practice - let me know if you
> are interested and I can send details.
>
> -John
>
>
>
>
> On Mon, Jul 7, 2014 at 4:02 AM, julie dubois <[hidden email]> wrote:
>> Hi John,
>>
>> As I specified it in the "edit" of my first message, I have this error
>> only if I run groomer with a command line in another xml file
>> surrounding by other command. When I run groomer alone with the
>> original xml, it works fine, so no problem with the PYTHONPATH.
>>  But when I use this :
>> <command>
>>                 #if $fastq_format == "illumina"
>>         blabla_No_python_command
>>                 #else
>>         python /data/galaxy-dist/tools/fastq/fastq_groomer.py
>> 'tmp.fastqcssanger' 'cssanger' '$output' 'illumina' 'ascii'
>> 'dont_summarize_input'
>>                 #end if
>> </command>
>>
>> It doesn't work.
>>
>> I've checked the files contained in job_working_directory. A shell
>> script is generated for my job and it begin with these lines :
>>
>> #!/bin/sh
>>
>> GALAXY_SLOTS="1"; export GALAXY_SLOTS;
>> export GALAXY_SLOTS
>> GALAXY_LIB="None"
>> if [ "$GALAXY_LIB" != "None" ]; then
>>     if [ -n "$PYTHONPATH" ]; then
>>         PYTHONPATH="$GALAXY_LIB:$PYTHONPATH"
>>     else
>>         PYTHONPATH="$GALAXY_LIB"
>>     fi
>>     export PYTHONPATH
>> fi
>>
>>
>>
>> So, with the run of groomer alone the line GALAXY_LIB is equal to
>> "/data/galaxy/galaxy-dist/lib", but here with the run of groomer with
>> a command line with another job the GALAXY_LIB is set to "None"!!!
>> I don't understand why.
>>
>> For the moment, to surround the problem, I've explicitly specified the
>> GALAXY_LIB path int the xml command before the command line wich run
>> python fastq_groomer.py and it works but I'm aware that is not
>> beautiful.
>>
>> Finally I'm using a local jobrunner.
>>
>> Thanks for your help
>> Julie
>>
>> P.S : we use such command because we build a workflow where the
>> quality format of the fastq must be invisible for the users.
>>
>> 2014-07-03 19:48 GMT+02:00 John Chilton <[hidden email]>:
>>> Hello Julie,
>>>
>>>   This is odd - sorry.
>>>
>>>   I might be worth uninstalling and re-installing this in the tool
>>> shed admin options - in particular ensure you install tool
>>> dependencies and that this installation process succeeds.
>>>
>>>   Otherwise -  it might be worth trying to put some debug statements
>>> in the fastq groomer tool to try to track down the problem. Here
>>> (https://gist.github.com/jmchilton/2d61f7ab81d99a0f02fc) is a diff to
>>> have that tool write out the run-time value  PYTHONPATH to Galaxy's
>>> home directory - it might be worth running a job, finding the output,
>>> setting PYTHONPATH to that value on the compute node and attempting to
>>> just import it directly.
>>>
>>> cd $HOME; python -c "import galaxy.sequences.fastq"
>>>
>>>   Out of curiosity - what job runner are you using (local, DRMAA, etc...)?
>>>
>>> -John
>>>
>>> On Tue, Jul 1, 2014 at 4:50 AM, julie dubois <[hidden email]> wrote:
>>>> Hi,
>>>> I've this error message when I run groomer.
>>>>
>>>> My local instance of galaxy runs on a python virtualenv (python 2.7)
>>>> and I have make the latest update with : hg pull -u.
>>>>
>>>> I've verified and this library is present in my
>>>> galaxy-dist/lib/galaxy_utils/sequence directory.
>>>>
>>>> I've found no other idea than the update on this forum to run groomer
>>>> correctly, any other idea ?
>>>>
>>>> Thanks.
>>>> Julie
>>>> ___________________________________________________________
>>>> Please keep all replies on the list by using "reply all"
>>>> in your mail client.  To manage your subscriptions to this
>>>> and other Galaxy lists, please use the interface at:
>>>>   http://lists.bx.psu.edu/
>>>>
>>>> To search Galaxy mailing lists use the unified search at:
>>>>   http://galaxyproject.org/search/mailinglists/
___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
Reply | Threaded
Open this post in threaded view
|

Re: ImportError: No module named galaxy_utils.sequence.fastq

John Chilton-4
If your tool defines a package requirement like:

<requirements>
<requirement type="package" version="1.0.0">galaxy-sequence-utils</requirement>
</requirements>

And your tool_dependency_dir setting universe_wsgi.ini is something
like /data/galaxy_deps - then you should be able to create a file
/data/galaxy_deps/galaxy-sequence-utils/1.0.0/env.sh with contents
like

. /data/galaxy_deps/galaxy-sequence-utils/1.0.0/venv/bin/activate

This will cause that tool's jobs to source that isolated environment
before execution. You should then be able to setup this virtual
environment using the following commands:

virtualenv /data/galaxy_deps/galaxy-sequence-utils/1.0.0/venv
. /data/galaxy_deps/galaxy-sequence-utils/1.0.0/venv/bin/activate
easy_install http://depot.galaxyproject.org/package/source/galaxy_sequence_utils/galaxy_sequence_utils-1.0.0.tgz

This may require installing some system Python packages such as
python-virtualenv on Ubuntu.

Hope this helps.

-John

On Mon, Jul 7, 2014 at 9:31 AM, julie dubois <[hidden email]> wrote:

> Thanks!
>
> Actually it's not for the toolshed, for the moment...
> But I'm very intereseted by details to install it into a virtualenv.
> Thank you.
>
> Julie
>
> 2014-07-07 16:16 GMT+02:00 John Chilton <[hidden email]>:
>> Ahhh - I get it now. Galaxy only adds itself to the PYTHONPATH if it
>> can infer it is running a Python script (i.e. the command block starts
>> with python or a Python interpreter is used).
>>
>> If you are building a tool for the tool shed - it would be best to
>> setup a tool dependency on the galaxy_sequence_utils package - this
>> would force every execution of the tool to have these utilities on
>> your PYTHONPATH. But since you are hard-encoding paths - it looks like
>> you are tailoring your Galaxy instance to your local environment. In
>> this case an easy hack is to just setup that dependency globally for
>> the system:
>>
>> sudo easy_install
>> http://depot.galaxyproject.org/package/source/galaxy_sequence_utils/galaxy_sequence_utils-1.0.0.tgz
>>
>> If you are setting up Galaxy env.sh files this can be installed into a
>> virtualenv instead - which is a better practice - let me know if you
>> are interested and I can send details.
>>
>> -John
>>
>>
>>
>>
>> On Mon, Jul 7, 2014 at 4:02 AM, julie dubois <[hidden email]> wrote:
>>> Hi John,
>>>
>>> As I specified it in the "edit" of my first message, I have this error
>>> only if I run groomer with a command line in another xml file
>>> surrounding by other command. When I run groomer alone with the
>>> original xml, it works fine, so no problem with the PYTHONPATH.
>>>  But when I use this :
>>> <command>
>>>                 #if $fastq_format == "illumina"
>>>         blabla_No_python_command
>>>                 #else
>>>         python /data/galaxy-dist/tools/fastq/fastq_groomer.py
>>> 'tmp.fastqcssanger' 'cssanger' '$output' 'illumina' 'ascii'
>>> 'dont_summarize_input'
>>>                 #end if
>>> </command>
>>>
>>> It doesn't work.
>>>
>>> I've checked the files contained in job_working_directory. A shell
>>> script is generated for my job and it begin with these lines :
>>>
>>> #!/bin/sh
>>>
>>> GALAXY_SLOTS="1"; export GALAXY_SLOTS;
>>> export GALAXY_SLOTS
>>> GALAXY_LIB="None"
>>> if [ "$GALAXY_LIB" != "None" ]; then
>>>     if [ -n "$PYTHONPATH" ]; then
>>>         PYTHONPATH="$GALAXY_LIB:$PYTHONPATH"
>>>     else
>>>         PYTHONPATH="$GALAXY_LIB"
>>>     fi
>>>     export PYTHONPATH
>>> fi
>>>
>>>
>>>
>>> So, with the run of groomer alone the line GALAXY_LIB is equal to
>>> "/data/galaxy/galaxy-dist/lib", but here with the run of groomer with
>>> a command line with another job the GALAXY_LIB is set to "None"!!!
>>> I don't understand why.
>>>
>>> For the moment, to surround the problem, I've explicitly specified the
>>> GALAXY_LIB path int the xml command before the command line wich run
>>> python fastq_groomer.py and it works but I'm aware that is not
>>> beautiful.
>>>
>>> Finally I'm using a local jobrunner.
>>>
>>> Thanks for your help
>>> Julie
>>>
>>> P.S : we use such command because we build a workflow where the
>>> quality format of the fastq must be invisible for the users.
>>>
>>> 2014-07-03 19:48 GMT+02:00 John Chilton <[hidden email]>:
>>>> Hello Julie,
>>>>
>>>>   This is odd - sorry.
>>>>
>>>>   I might be worth uninstalling and re-installing this in the tool
>>>> shed admin options - in particular ensure you install tool
>>>> dependencies and that this installation process succeeds.
>>>>
>>>>   Otherwise -  it might be worth trying to put some debug statements
>>>> in the fastq groomer tool to try to track down the problem. Here
>>>> (https://gist.github.com/jmchilton/2d61f7ab81d99a0f02fc) is a diff to
>>>> have that tool write out the run-time value  PYTHONPATH to Galaxy's
>>>> home directory - it might be worth running a job, finding the output,
>>>> setting PYTHONPATH to that value on the compute node and attempting to
>>>> just import it directly.
>>>>
>>>> cd $HOME; python -c "import galaxy.sequences.fastq"
>>>>
>>>>   Out of curiosity - what job runner are you using (local, DRMAA, etc...)?
>>>>
>>>> -John
>>>>
>>>> On Tue, Jul 1, 2014 at 4:50 AM, julie dubois <[hidden email]> wrote:
>>>>> Hi,
>>>>> I've this error message when I run groomer.
>>>>>
>>>>> My local instance of galaxy runs on a python virtualenv (python 2.7)
>>>>> and I have make the latest update with : hg pull -u.
>>>>>
>>>>> I've verified and this library is present in my
>>>>> galaxy-dist/lib/galaxy_utils/sequence directory.
>>>>>
>>>>> I've found no other idea than the update on this forum to run groomer
>>>>> correctly, any other idea ?
>>>>>
>>>>> Thanks.
>>>>> Julie
>>>>> ___________________________________________________________
>>>>> Please keep all replies on the list by using "reply all"
>>>>> in your mail client.  To manage your subscriptions to this
>>>>> and other Galaxy lists, please use the interface at:
>>>>>   http://lists.bx.psu.edu/
>>>>>
>>>>> To search Galaxy mailing lists use the unified search at:
>>>>>   http://galaxyproject.org/search/mailinglists/
___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/