conda management of tools dependent on python3

classic Classic list List threaded Threaded
10 messages Options
Reply | Threaded
Open this post in threaded view
|

conda management of tools dependent on python3

Wolfgang Maier
Dear all,

I'm still pretty new to conda tool dependencies management and there is
one thing I cannot find documented anywhere:
How can I let conda manage tool dependencies that involve python3?
If I let Galaxy install conda for me, it creates an environment with
python2, then if there is a tool dependency that has a conda package
built for only python3 I'm getting an error because of a conflict
between that package and python2 (not surprisingly), but AFAIU conda
environments are bound to one particular version of python (or is that
incorrect?).

Thanks for any help with this and best wishes,
Wolfgang
___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
Reply | Threaded
Open this post in threaded view
|

Re: conda management of tools dependent on python3

Björn Grüning-3
Hi Wolfgang,

can you describe step by step what you are trying to do?

A conda package ships it's own python version, you can see this in the
build-string py35_0 for example. So conda can handle all python version
independent of the python version of miniconda.

Cheers,
Bjoern

> Dear all,
>
> I'm still pretty new to conda tool dependencies management and there is
> one thing I cannot find documented anywhere:
> How can I let conda manage tool dependencies that involve python3?
> If I let Galaxy install conda for me, it creates an environment with
> python2, then if there is a tool dependency that has a conda package
> built for only python3 I'm getting an error because of a conflict
> between that package and python2 (not surprisingly), but AFAIU conda
> environments are bound to one particular version of python (or is that
> incorrect?).
>
> Thanks for any help with this and best wishes,
> Wolfgang
> ___________________________________________________________
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>  https://lists.galaxyproject.org/
>
> To search Galaxy mailing lists use the unified search at:
>  http://galaxyproject.org/search/mailinglists/
___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
Reply | Threaded
Open this post in threaded view
|

Re: conda management of tools dependent on python3

Wolfgang Maier
Hi Björn,

Hmm, I guess I should have phrased my question more clearly. What I'm
trying to do is to get Galaxy to satisfy a missing tool dependency at
*tool runtime* using a bioconda package that is only available for
Python3. I am able to satisfy the dependency *manually* by running:

conda create --name __<package>@<package_version>
<package>==<package_version python=3.5 -c bioconda

(following the example at
https://docs.galaxyproject.org/en/master/admin/conda_faq.html#how-do-conda-dependencies-work-where-do-things-get-installed),
but when I let *Galaxy autoinstall* the package it will use python2.7
for the environment, then fail with:

Error: Unsatisfiable package specifications.

and a hint that <package> conflicts with python 2.7. So my question is:
is there any way to inform Galaxy that the dependency requires a conda
environment based on python3?

Best,
Wolfgang


On 28.10.2016 19:14, Björn Grüning wrote:

> Hi Wolfgang,
>
> can you describe step by step what you are trying to do?
>
> A conda package ships it's own python version, you can see this in the
> build-string py35_0 for example. So conda can handle all python version
> independent of the python version of miniconda.
>
> Cheers,
> Bjoern
>
>> Dear all,
>>
>> I'm still pretty new to conda tool dependencies management and there is
>> one thing I cannot find documented anywhere:
>> How can I let conda manage tool dependencies that involve python3?
>> If I let Galaxy install conda for me, it creates an environment with
>> python2, then if there is a tool dependency that has a conda package
>> built for only python3 I'm getting an error because of a conflict
>> between that package and python2 (not surprisingly), but AFAIU conda
>> environments are bound to one particular version of python (or is that
>> incorrect?).
>>
>> Thanks for any help with this and best wishes,
>> Wolfgang
>> ___________________________________________________________
>> Please keep all replies on the list by using "reply all"
>> in your mail client.  To manage your subscriptions to this
>> and other Galaxy lists, please use the interface at:
>>  https://lists.galaxyproject.org/
>>
>> To search Galaxy mailing lists use the unified search at:
>>  http://galaxyproject.org/search/mailinglists/
___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
Reply | Threaded
Open this post in threaded view
|

Re: conda management of tools dependent on python3

Björn Grüning-3
Hi,

can you check the package? The package should have a dependency on
python3, if not the package seems to be broken :(

Ciao,
Bjoern

Am 03.11.2016 um 21:32 schrieb Wolfgang Maier:

> Hi Björn,
>
> Hmm, I guess I should have phrased my question more clearly. What I'm
> trying to do is to get Galaxy to satisfy a missing tool dependency at
> *tool runtime* using a bioconda package that is only available for
> Python3. I am able to satisfy the dependency *manually* by running:
>
> conda create --name __<package>@<package_version>
> <package>==<package_version python=3.5 -c bioconda
>
> (following the example at
> https://docs.galaxyproject.org/en/master/admin/conda_faq.html#how-do-conda-dependencies-work-where-do-things-get-installed),
> but when I let *Galaxy autoinstall* the package it will use python2.7
> for the environment, then fail with:
>
> Error: Unsatisfiable package specifications.
>
> and a hint that <package> conflicts with python 2.7. So my question is:
> is there any way to inform Galaxy that the dependency requires a conda
> environment based on python3?
>
> Best,
> Wolfgang
>
>
> On 28.10.2016 19:14, Björn Grüning wrote:
>> Hi Wolfgang,
>>
>> can you describe step by step what you are trying to do?
>>
>> A conda package ships it's own python version, you can see this in the
>> build-string py35_0 for example. So conda can handle all python version
>> independent of the python version of miniconda.
>>
>> Cheers,
>> Bjoern
>>
>>> Dear all,
>>>
>>> I'm still pretty new to conda tool dependencies management and there is
>>> one thing I cannot find documented anywhere:
>>> How can I let conda manage tool dependencies that involve python3?
>>> If I let Galaxy install conda for me, it creates an environment with
>>> python2, then if there is a tool dependency that has a conda package
>>> built for only python3 I'm getting an error because of a conflict
>>> between that package and python2 (not surprisingly), but AFAIU conda
>>> environments are bound to one particular version of python (or is that
>>> incorrect?).
>>>
>>> Thanks for any help with this and best wishes,
>>> Wolfgang
>>> ___________________________________________________________
>>> Please keep all replies on the list by using "reply all"
>>> in your mail client.  To manage your subscriptions to this
>>> and other Galaxy lists, please use the interface at:
>>>  https://lists.galaxyproject.org/
>>>
>>> To search Galaxy mailing lists use the unified search at:
>>>  http://galaxyproject.org/search/mailinglists/
___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
Reply | Threaded
Open this post in threaded view
|

Re: conda management of tools dependent on python3

Wolfgang Maier
I'm not quite sure if I understand you correctly, but let's see.
The original tool wrapper stated

<requirement type="package" version="3.4">python3</requirement>

as a dependency. This did not work with at runtime installation at all
because conda cannot resolve python3 and the requirement got ignored by
Galaxy. I changed the requirement tag now to:

<requirement type="package" version="3.4">python</requirement>

and now Galaxy happily installs a __python@3.4 environment, but then it
tries to set up a *new* environment for my package, which is still based
on python2.7 and fails as before.
On the bioconda side, all the recipe does is to declare that it
shouldn't be built under python2 using
skip: True # [py2k] in the build section of the recipe.


On 03.11.2016 21:36, Björn Grüning wrote:

> Hi,
>
> can you check the package? The package should have a dependency on
> python3, if not the package seems to be broken :(
>
> Ciao,
> Bjoern
>
> Am 03.11.2016 um 21:32 schrieb Wolfgang Maier:
>> Hi Björn,
>>
>> Hmm, I guess I should have phrased my question more clearly. What I'm
>> trying to do is to get Galaxy to satisfy a missing tool dependency at
>> *tool runtime* using a bioconda package that is only available for
>> Python3. I am able to satisfy the dependency *manually* by running:
>>
>> conda create --name __<package>@<package_version>
>> <package>==<package_version python=3.5 -c bioconda
>>
>> (following the example at
>> https://docs.galaxyproject.org/en/master/admin/conda_faq.html#how-do-conda-dependencies-work-where-do-things-get-installed),
>> but when I let *Galaxy autoinstall* the package it will use python2.7
>> for the environment, then fail with:
>>
>> Error: Unsatisfiable package specifications.
>>
>> and a hint that <package> conflicts with python 2.7. So my question is:
>> is there any way to inform Galaxy that the dependency requires a conda
>> environment based on python3?
>>
>> Best,
>> Wolfgang
>>
>>
>> On 28.10.2016 19:14, Björn Grüning wrote:
>>> Hi Wolfgang,
>>>
>>> can you describe step by step what you are trying to do?
>>>
>>> A conda package ships it's own python version, you can see this in the
>>> build-string py35_0 for example. So conda can handle all python version
>>> independent of the python version of miniconda.
>>>
>>> Cheers,
>>> Bjoern
>>>
>>>> Dear all,
>>>>
>>>> I'm still pretty new to conda tool dependencies management and there is
>>>> one thing I cannot find documented anywhere:
>>>> How can I let conda manage tool dependencies that involve python3?
>>>> If I let Galaxy install conda for me, it creates an environment with
>>>> python2, then if there is a tool dependency that has a conda package
>>>> built for only python3 I'm getting an error because of a conflict
>>>> between that package and python2 (not surprisingly), but AFAIU conda
>>>> environments are bound to one particular version of python (or is that
>>>> incorrect?).
>>>>
>>>> Thanks for any help with this and best wishes,
>>>> Wolfgang
>>>> ___________________________________________________________
>>>> Please keep all replies on the list by using "reply all"
>>>> in your mail client.  To manage your subscriptions to this
>>>> and other Galaxy lists, please use the interface at:
>>>>  https://lists.galaxyproject.org/
>>>>
>>>> To search Galaxy mailing lists use the unified search at:
>>>>  http://galaxyproject.org/search/mailinglists/
___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
Reply | Threaded
Open this post in threaded view
|

filter options

Vincent Cahais
Dear All,

I try to filter options for a param linked to a datatable.
I want to keep only options with the same dbkey as my input file. But i
cannot make it work.

<param name="input" type="data" format="bam" label="Bam file"/>
<param name="dbindel" type="select" force_select="true" label="indel
database">
         <options from_data_table="dbindel" />
         <filter type="data_meta" ref="input" key="dbkey" />
         <filter type="remove_value" key="dbkey"/>
</param>

##dbindels
#<unique_build_id>   <dbkey>   <display_name> <file_path>
hg38    hg38    Human
/data/genome/hg38/dbindel/Mills_and_1000G_gold_standard.indels.hg38.vcf
mm10_1  mm10    Mouse [All]
/data/genomes/mm10/dbindel/mgp.v5.merged.indels.dbSNP142.normed.vcf.gz
mm10_2  mm10    Mouse [129P2_OlaHsd]
/data/genome/mm10/dbindel/129P2_OlaHsd.mgp.v5.indels.dbSNP142.normed.vcf.gz

Thank you for help,

Vincent
___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
Reply | Threaded
Open this post in threaded view
|

Re: conda management of tools dependent on python3

Nicola Soranzo-2
In reply to this post by Wolfgang Maier
Hi Wolfgang,
I guess that the conda recipe should be updated to also specify

- python >=3

in the requirements.

Cheers,
Nicola

On 03/11/16 21:04, Wolfgang Maier wrote:

> I'm not quite sure if I understand you correctly, but let's see.
> The original tool wrapper stated
>
> <requirement type="package" version="3.4">python3</requirement>
>
> as a dependency. This did not work with at runtime installation at all
> because conda cannot resolve python3 and the requirement got ignored
> by Galaxy. I changed the requirement tag now to:
>
> <requirement type="package" version="3.4">python</requirement>
>
> and now Galaxy happily installs a __python@3.4 environment, but then
> it tries to set up a *new* environment for my package, which is still
> based on python2.7 and fails as before.
> On the bioconda side, all the recipe does is to declare that it
> shouldn't be built under python2 using
> skip: True # [py2k] in the build section of the recipe.
>
>
> On 03.11.2016 21:36, Björn Grüning wrote:
>> Hi,
>>
>> can you check the package? The package should have a dependency on
>> python3, if not the package seems to be broken :(
>>
>> Ciao,
>> Bjoern
>>
>> Am 03.11.2016 um 21:32 schrieb Wolfgang Maier:
>>> Hi Björn,
>>>
>>> Hmm, I guess I should have phrased my question more clearly. What I'm
>>> trying to do is to get Galaxy to satisfy a missing tool dependency at
>>> *tool runtime* using a bioconda package that is only available for
>>> Python3. I am able to satisfy the dependency *manually* by running:
>>>
>>> conda create --name __<package>@<package_version>
>>> <package>==<package_version python=3.5 -c bioconda
>>>
>>> (following the example at
>>> https://docs.galaxyproject.org/en/master/admin/conda_faq.html#how-do-conda-dependencies-work-where-do-things-get-installed),
>>>
>>> but when I let *Galaxy autoinstall* the package it will use python2.7
>>> for the environment, then fail with:
>>>
>>> Error: Unsatisfiable package specifications.
>>>
>>> and a hint that <package> conflicts with python 2.7. So my question is:
>>> is there any way to inform Galaxy that the dependency requires a conda
>>> environment based on python3?
>>>
>>> Best,
>>> Wolfgang
>>>
>>>
>>> On 28.10.2016 19:14, Björn Grüning wrote:
>>>> Hi Wolfgang,
>>>>
>>>> can you describe step by step what you are trying to do?
>>>>
>>>> A conda package ships it's own python version, you can see this in the
>>>> build-string py35_0 for example. So conda can handle all python
>>>> version
>>>> independent of the python version of miniconda.
>>>>
>>>> Cheers,
>>>> Bjoern
>>>>
>>>>> Dear all,
>>>>>
>>>>> I'm still pretty new to conda tool dependencies management and
>>>>> there is
>>>>> one thing I cannot find documented anywhere:
>>>>> How can I let conda manage tool dependencies that involve python3?
>>>>> If I let Galaxy install conda for me, it creates an environment with
>>>>> python2, then if there is a tool dependency that has a conda package
>>>>> built for only python3 I'm getting an error because of a conflict
>>>>> between that package and python2 (not surprisingly), but AFAIU conda
>>>>> environments are bound to one particular version of python (or is
>>>>> that
>>>>> incorrect?).
>>>>>
>>>>> Thanks for any help with this and best wishes,
>>>>> Wolfgang
>>>>> ___________________________________________________________
>>>>> Please keep all replies on the list by using "reply all"
>>>>> in your mail client.  To manage your subscriptions to this
>>>>> and other Galaxy lists, please use the interface at:
>>>>>  https://lists.galaxyproject.org/
>>>>>
>>>>> To search Galaxy mailing lists use the unified search at:
>>>>>  http://galaxyproject.org/search/mailinglists/
> ___________________________________________________________
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>  https://lists.galaxyproject.org/
>
> To search Galaxy mailing lists use the unified search at:
>  http://galaxyproject.org/search/mailinglists/

___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
Reply | Threaded
Open this post in threaded view
|

Re: filter options

Nicola Soranzo-2
In reply to this post by Vincent Cahais
Hi Vincent,
have you defined the dbindel table in config/tool_data_table_conf.xml ?

Cheers,
Nicola

On 04/11/16 08:09, Vincent Cahais wrote:

> Dear All,
>
> I try to filter options for a param linked to a datatable.
> I want to keep only options with the same dbkey as my input file. But i
> cannot make it work.
>
> <param name="input" type="data" format="bam" label="Bam file"/>
> <param name="dbindel" type="select" force_select="true" label="indel
> database">
>           <options from_data_table="dbindel" />
>           <filter type="data_meta" ref="input" key="dbkey" />
>           <filter type="remove_value" key="dbkey"/>
> </param>
>
> ##dbindels
> #<unique_build_id>   <dbkey>   <display_name> <file_path>
> hg38    hg38    Human
> /data/genome/hg38/dbindel/Mills_and_1000G_gold_standard.indels.hg38.vcf
> mm10_1  mm10    Mouse [All]
> /data/genomes/mm10/dbindel/mgp.v5.merged.indels.dbSNP142.normed.vcf.gz
> mm10_2  mm10    Mouse [129P2_OlaHsd]
> /data/genome/mm10/dbindel/129P2_OlaHsd.mgp.v5.indels.dbSNP142.normed.vcf.gz
>
> Thank you for help,
>
> Vincent
> ___________________________________________________________
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>    https://lists.galaxyproject.org/
>
> To search Galaxy mailing lists use the unified search at:
>    http://galaxyproject.org/search/mailinglists/

___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
Reply | Threaded
Open this post in threaded view
|

Re: filter options

Vincent Cahais
Sure, my combo-box is correctly filled with all options, but unfiltered.

<table comment_char="#" name="dbindel">
         <columns>value, dbkey, name, path</columns>
         <file path="****" />
     </table>
</tables>

Le 04/11/2016 à 13:42, Nicola Soranzo a écrit :

> Hi Vincent,
> have you defined the dbindel table in config/tool_data_table_conf.xml ?
>
> Cheers,
> Nicola
>
> On 04/11/16 08:09, Vincent Cahais wrote:
>> Dear All,
>>
>> I try to filter options for a param linked to a datatable.
>> I want to keep only options with the same dbkey as my input file. But i
>> cannot make it work.
>>
>> <param name="input" type="data" format="bam" label="Bam file"/>
>> <param name="dbindel" type="select" force_select="true" label="indel
>> database">
>>           <options from_data_table="dbindel" />
>>           <filter type="data_meta" ref="input" key="dbkey" />
>>           <filter type="remove_value" key="dbkey"/>
>> </param>
>>
>> ##dbindels
>> #<unique_build_id>   <dbkey>   <display_name> <file_path>
>> hg38    hg38    Human
>> /data/genome/hg38/dbindel/Mills_and_1000G_gold_standard.indels.hg38.vcf
>> mm10_1  mm10    Mouse [All]
>> /data/genomes/mm10/dbindel/mgp.v5.merged.indels.dbSNP142.normed.vcf.gz
>> mm10_2  mm10    Mouse [129P2_OlaHsd]
>> /data/genome/mm10/dbindel/129P2_OlaHsd.mgp.v5.indels.dbSNP142.normed.vcf.gz
>>
>>
>> Thank you for help,
>>
>> Vincent
>> ___________________________________________________________
>> Please keep all replies on the list by using "reply all"
>> in your mail client.  To manage your subscriptions to this
>> and other Galaxy lists, please use the interface at:
>>    https://lists.galaxyproject.org/
>>
>> To search Galaxy mailing lists use the unified search at:
>>    http://galaxyproject.org/search/mailinglists/
>
___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
Reply | Threaded
Open this post in threaded view
|

Re: filter options

Vincent Cahais

I found the solution :

<param name="input" type="data" format="bam" label="Bam file"/>

<param name="dbindel" type="select" force_select="true" label="indel database">
         <options from_data_table="dbindel">
                <filter type="data_meta" ref="input" key="dbkey" column="1"/>
        </options>
</param>


Le 04/11/2016 à 13:50, Vincent Cahais a écrit :
Sure, my combo-box is correctly filled with all options, but unfiltered.

<table comment_char="#" name="dbindel">
         <columns>value, dbkey, name, path</columns>
         <file path="****" />
     </table>
</tables>

Le 04/11/2016 à 13:42, Nicola Soranzo a écrit :
Hi Vincent,
have you defined the dbindel table in config/tool_data_table_conf.xml ?

Cheers,
Nicola

On 04/11/16 08:09, Vincent Cahais wrote:
Dear All,

I try to filter options for a param linked to a datatable.
I want to keep only options with the same dbkey as my input file. But i
cannot make it work.

<param name="input" type="data" format="bam" label="Bam file"/>
<param name="dbindel" type="select" force_select="true" label="indel
database">
          <options from_data_table="dbindel" />
          <filter type="data_meta" ref="input" key="dbkey" />
          <filter type="remove_value" key="dbkey"/>
</param>

##dbindels
#<unique_build_id>   <dbkey>   <display_name> <file_path>
hg38    hg38    Human
/data/genome/hg38/dbindel/Mills_and_1000G_gold_standard.indels.hg38.vcf
mm10_1  mm10    Mouse [All]
/data/genomes/mm10/dbindel/mgp.v5.merged.indels.dbSNP142.normed.vcf.gz
mm10_2  mm10    Mouse [129P2_OlaHsd]
/data/genome/mm10/dbindel/129P2_OlaHsd.mgp.v5.indels.dbSNP142.normed.vcf.gz 


Thank you for help,

Vincent
___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
   https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
   http://galaxyproject.org/search/mailinglists/

___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

--

Vincent Cahais, Sr Resrch Asst Data Mg/Anlys
Programming Tech, Mecanisms of Carcinogenesis Section

International Agency for Research on Cancer
World Health Organization
150 cours Albert Thomas
69372 Lyon CEDEX 08, France
Off. +33 472 738 487
Mob.
E-mail:[hidden email]


___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/