PATH set in tool env.sh leaking through to set_metadata.py

classic Classic list List threaded Threaded
8 messages Options
| Threaded
Open this post in threaded view
|

PATH set in tool env.sh leaking through to set_metadata.py

Wolfgang Maier
Hi,

I am not sure whether I am diagnosing this problem approximately right
or not, but it seems that with the latest release of Galaxy (stable
branch, latest_2015.01.13) enviroment variables set in a
tool-dependency's env.sh file and sourced before a job run are now
leaking through to

./scripts/set_metadata.py called after the tool run finishes.

In my specific case, the sourced env.sh adds a python 3 virtualenv
directory to PATH, which then causes set_metadata.py to fail with:

   File "./scripts/set_metadata.py", line 130
     except Exception, e:
                     ^
SyntaxError: invalid syntax

because that (amazingly enough :)) seems to be the first line not
compatible with Python 3.

I have not seen this error before (I believe not with latest_2014.10.06)
so I guess my question is: has the way that set_metadata.py is called
been changed recently and would you consider the current behavior a bug
or is this expected and ok ?

Best,
Wolfgang

___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
| Threaded
Open this post in threaded view
|

Re: PATH set in tool env.sh leaking through to set_metadata.py

Wolfgang Maier
On 01/20/2015 06:20 PM, Wolfgang Maier wrote:
>
> I have not seen this error before (I believe not with latest_2014.10.06)

update: confirmed this now. It's enough to hg update to
latest_2014.10.06 and things are working again.

The difference is that when building the dependency shell command the
latest release seems to put the call to set_metadata.sh into that
command, while before it seems it was run separately.

Wolfgang

___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
| Threaded
Open this post in threaded view
|

Re: PATH set in tool env.sh leaking through to set_metadata.py

John Chilton-4
Are you using the local job runner (this will be the case if you
haven't explicitly configured something like pbs or DRMAA in your
job_conf.xml file)?

-John

On Tue, Jan 20, 2015 at 12:34 PM, Wolfgang Maier
<[hidden email]> wrote:

> On 01/20/2015 06:20 PM, Wolfgang Maier wrote:
>>
>>
>> I have not seen this error before (I believe not with latest_2014.10.06)
>
>
> update: confirmed this now. It's enough to hg update to latest_2014.10.06
> and things are working again.
>
> The difference is that when building the dependency shell command the latest
> release seems to put the call to set_metadata.sh into that command, while
> before it seems it was run separately.
>
>
> Wolfgang
>
> ___________________________________________________________
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>  https://lists.galaxyproject.org/
>
> To search Galaxy mailing lists use the unified search at:
>  http://galaxyproject.org/search/mailinglists/
___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
| Threaded
Open this post in threaded view
|

Re: PATH set in tool env.sh leaking through to set_metadata.py

Wolfgang Maier
Yes, this is the local job runner on a single machine.


On 01/20/2015 06:52 PM, John Chilton wrote:

> Are you using the local job runner (this will be the case if you
> haven't explicitly configured something like pbs or DRMAA in your
> job_conf.xml file)?
>
> -John
>
> On Tue, Jan 20, 2015 at 12:34 PM, Wolfgang Maier
> <[hidden email]> wrote:
>> On 01/20/2015 06:20 PM, Wolfgang Maier wrote:
>>>
>>>
>>> I have not seen this error before (I believe not with latest_2014.10.06)
>>
>>
>> update: confirmed this now. It's enough to hg update to latest_2014.10.06
>> and things are working again.
>>
>> The difference is that when building the dependency shell command the latest
>> release seems to put the call to set_metadata.sh into that command, while
>> before it seems it was run separately.
>>
>>
>> Wolfgang
>>
>> ___________________________________________________________
>> Please keep all replies on the list by using "reply all"
>> in your mail client.  To manage your subscriptions to this
>> and other Galaxy lists, please use the interface at:
>>   https://lists.galaxyproject.org/
>>
>> To search Galaxy mailing lists use the unified search at:
>>   http://galaxyproject.org/search/mailinglists/
> ___________________________________________________________
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>    https://lists.galaxyproject.org/
>
> To search Galaxy mailing lists use the unified search at:
>    http://galaxyproject.org/search/mailinglists/
>

___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
| Threaded
Open this post in threaded view
|

Re: PATH set in tool env.sh leaking through to set_metadata.py

John Chilton-4
In reply to this post by John Chilton-4
So the tool dependency stuff didn't change - but the local job runner
now behaves like the other job runners (DRMAA, PBS, etc...) so you
have uncovered a bug for all of them I think. Is this a tool shed tool
or a locally created dependency?

I need to think about how to solve this more generally but if you want
a quick work around - I think it would work to just replace "python"
in $GALAXY_ROOT/set_metadata.sh with a  hard-coded path to Galaxy's
python or perhaps just "python2".

Sorry for the inconvenience and thanks for bringing this to our attention.

-John

On Tue, Jan 20, 2015 at 12:52 PM, John Chilton <[hidden email]> wrote:

> Are you using the local job runner (this will be the case if you
> haven't explicitly configured something like pbs or DRMAA in your
> job_conf.xml file)?
>
> -John
>
> On Tue, Jan 20, 2015 at 12:34 PM, Wolfgang Maier
> <[hidden email]> wrote:
>> On 01/20/2015 06:20 PM, Wolfgang Maier wrote:
>>>
>>>
>>> I have not seen this error before (I believe not with latest_2014.10.06)
>>
>>
>> update: confirmed this now. It's enough to hg update to latest_2014.10.06
>> and things are working again.
>>
>> The difference is that when building the dependency shell command the latest
>> release seems to put the call to set_metadata.sh into that command, while
>> before it seems it was run separately.
>>
>>
>> Wolfgang
>>
>> ___________________________________________________________
>> Please keep all replies on the list by using "reply all"
>> in your mail client.  To manage your subscriptions to this
>> and other Galaxy lists, please use the interface at:
>>  https://lists.galaxyproject.org/
>>
>> To search Galaxy mailing lists use the unified search at:
>>  http://galaxyproject.org/search/mailinglists/
___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
| Threaded
Open this post in threaded view
|

Re: PATH set in tool env.sh leaking through to set_metadata.py

Wolfgang Maier
On 21.01.2015 16:11, John Chilton wrote:
> So the tool dependency stuff didn't change - but the local job runner
> now behaves like the other job runners (DRMAA, PBS, etc...) so you
> have uncovered a bug for all of them I think. Is this a tool shed tool
> or a locally created dependency?
>

This is a tool shed tool (package_mimodd_0_1_5), but one I uploaded
myself (I discovered the bug when I did a test install after updating
our local Galaxy).

> I need to think about how to solve this more generally but if you want
> a quick work around - I think it would work to just replace "python"
> in $GALAXY_ROOT/set_metadata.sh with a  hard-coded path to Galaxy's
> python or perhaps just "python2".

Since I can change the tool shed tool a more "global" solution could be
to simply remove the "python" symlink from the python3 virtualenv after
its created and have the tool use "python3" explicitly at runtime. I'll
try whether that works, but I think it should.

>
> Sorry for the inconvenience and thanks for bringing this to our attention.
>

No problem. I'm glad if I could help with something instead of just
asking questions all the time.

Wolfgang


>
> On Tue, Jan 20, 2015 at 12:52 PM, John Chilton <[hidden email]> wrote:
>> Are you using the local job runner (this will be the case if you
>> haven't explicitly configured something like pbs or DRMAA in your
>> job_conf.xml file)?
>>
>> -John
>>
>> On Tue, Jan 20, 2015 at 12:34 PM, Wolfgang Maier
>> <[hidden email]> wrote:
>>> On 01/20/2015 06:20 PM, Wolfgang Maier wrote:
>>>>
>>>>
>>>> I have not seen this error before (I believe not with latest_2014.10.06)
>>>
>>>
>>> update: confirmed this now. It's enough to hg update to latest_2014.10.06
>>> and things are working again.
>>>
>>> The difference is that when building the dependency shell command the latest
>>> release seems to put the call to set_metadata.sh into that command, while
>>> before it seems it was run separately.
>>>
>>>
>>> Wolfgang
>>>

___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
| Threaded
Open this post in threaded view
|

Re: PATH set in tool env.sh leaking through to set_metadata.py

C. Ch.
In reply to this post by John Chilton-4
Hi,

Could this be related to the fact that galaxy is not seeing the tools that are installed in /usr/local/bin?? 

I'm on latest_2015.01.13 and since the two last upgrades I saw that interproscan and bsmap were not found by galaxy. Both are installed in /usr/local and when invoked on the command line, as the galaxy user, they work perfectly well. 
Before the last two upgrades both worked from within galaxy.

Cristian

> Date: Wed, 21 Jan 2015 10:11:43 -0500

> From: [hidden email]
> To: [hidden email]
> CC: [hidden email]
> Subject: Re: [galaxy-dev] PATH set in tool env.sh leaking through to set_metadata.py
>
> So the tool dependency stuff didn't change - but the local job runner
> now behaves like the other job runners (DRMAA, PBS, etc...) so you
> have uncovered a bug for all of them I think. Is this a tool shed tool
> or a locally created dependency?
>
> I need to think about how to solve this more generally but if you want
> a quick work around - I think it would work to just replace "python"
> in $GALAXY_ROOT/set_metadata.sh with a hard-coded path to Galaxy's
> python or perhaps just "python2".
>
> Sorry for the inconvenience and thanks for bringing this to our attention.
>
> -John
>
> On Tue, Jan 20, 2015 at 12:52 PM, John Chilton <[hidden email]> wrote:
> > Are you using the local job runner (this will be the case if you
> > haven't explicitly configured something like pbs or DRMAA in your
> > job_conf.xml file)?
> >
> > -John
> >
> > On Tue, Jan 20, 2015 at 12:34 PM, Wolfgang Maier
> > <[hidden email]> wrote:
> >> On 01/20/2015 06:20 PM, Wolfgang Maier wrote:
> >>>
> >>>
> >>> I have not seen this error before (I believe not with latest_2014.10.06)
> >>
> >>
> >> update: confirmed this now. It's enough to hg update to latest_2014.10.06
> >> and things are working again.
> >>
> >> The difference is that when building the dependency shell command the latest
> >> release seems to put the call to set_metadata.sh into that command, while
> >> before it seems it was run separately.
> >>
> >>
> >> Wolfgang
> >>
> >> ___________________________________________________________
> >> Please keep all replies on the list by using "reply all"
> >> in your mail client. To manage your subscriptions to this
> >> and other Galaxy lists, please use the interface at:
> >> https://lists.galaxyproject.org/
> >>
> >> To search Galaxy mailing lists use the unified search at:
> >> http://galaxyproject.org/search/mailinglists/
> ___________________________________________________________
> Please keep all replies on the list by using "reply all"
> in your mail client. To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
> https://lists.galaxyproject.org/
>
> To search Galaxy mailing lists use the unified search at:
> http://galaxyproject.org/search/mailinglists/

___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
| Threaded
Open this post in threaded view
|

Re: PATH set in tool env.sh leaking through to set_metadata.py

John Chilton-4
In reply to this post by Wolfgang Maier
On Wed, Jan 21, 2015 at 6:54 PM, John Chilton <[hidden email]> wrote:

> On Wed, Jan 21, 2015 at 10:35 AM, Wolfgang Maier
> <[hidden email]> wrote:
>>
>>
>> On 21.01.2015 16:11, John Chilton wrote:
>>>
>>> So the tool dependency stuff didn't change - but the local job runner
>>> now behaves like the other job runners (DRMAA, PBS, etc...) so you
>>> have uncovered a bug for all of them I think. Is this a tool shed tool
>>> or a locally created dependency?
>>>
>>
>> This is a tool shed tool (package_mimodd_0_1_5), but one I uploaded myself
>> (I discovered the bug when I did a test install after updating our local
>> Galaxy).
>>
>>> I need to think about how to solve this more generally but if you want
>>> a quick work around - I think it would work to just replace "python"
>>> in $GALAXY_ROOT/set_metadata.sh with a  hard-coded path to Galaxy's
>>> python or perhaps just "python2".
>>
>>
>> Since I can change the tool shed tool a more "global" solution could be to
>> simply remove the "python" symlink from the python3 virtualenv after its
>> created and have the tool use "python3" explicitly at runtime. I'll try
>> whether that works, but I think it should.
>
> This sounds like a better short-term solution to me - this way older
> versions of Galaxy will still be able to use the recipe. I created a
> Trello issue to track the longer term fix of isolating metadata
> generation from the tool environment - which is needed but could be a
> little disruptive so I would rather push it off until after the next
> release.

Opps - I forgot the link and forgot to cc the mailing list.

https://trello.com/c/aMCwYZsN

-John

>
> Thanks!
>
> -John
>
>>
>>>
>>> Sorry for the inconvenience and thanks for bringing this to our attention.
>>>
>>
>> No problem. I'm glad if I could help with something instead of just asking
>> questions all the time.
>>
>> Wolfgang
>>
>>
>>
>>>
>>> On Tue, Jan 20, 2015 at 12:52 PM, John Chilton <[hidden email]>
>>> wrote:
>>>>
>>>> Are you using the local job runner (this will be the case if you
>>>> haven't explicitly configured something like pbs or DRMAA in your
>>>> job_conf.xml file)?
>>>>
>>>> -John
>>>>
>>>> On Tue, Jan 20, 2015 at 12:34 PM, Wolfgang Maier
>>>> <[hidden email]> wrote:
>>>>>
>>>>> On 01/20/2015 06:20 PM, Wolfgang Maier wrote:
>>>>>>
>>>>>>
>>>>>>
>>>>>> I have not seen this error before (I believe not with
>>>>>> latest_2014.10.06)
>>>>>
>>>>>
>>>>>
>>>>> update: confirmed this now. It's enough to hg update to
>>>>> latest_2014.10.06
>>>>> and things are working again.
>>>>>
>>>>> The difference is that when building the dependency shell command the
>>>>> latest
>>>>> release seems to put the call to set_metadata.sh into that command,
>>>>> while
>>>>> before it seems it was run separately.
>>>>>
>>>>>
>>>>> Wolfgang
>>>>>
>>
___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/