Tools using Galaxy's Python library?

classic Classic list List threaded Threaded
9 messages Options
| Threaded
Open this post in threaded view
|

Tools using Galaxy's Python library?

Peter Cock
Hi all,

A few of my tools have for a long time used Galaxy's own parsing
functionality in order to avoid an external dependency. Lately
this has stopped working on my TravisCI testing with planemo
using the Galaxy dev branch (the stable master branch is fine):
e.g.

https://travis-ci.org/peterjc/pico_galaxy/builds/196655736

The tools fail with things like:

|  from galaxy_utils.sequence.fasta import fastaReader, fastaWriter
|  ImportError: No module named galaxy_utils.sequence.fasta


or:

|  from galaxy_utils.sequence.fastq import fastqReader
|  ImportError: No module named galaxy_utils.sequence.fastq

Is this a temporary regression in Galaxy, or a deliberate change?
Do the tools need to do something to explicit have access to the
Galaxy Python library, or are they now considered private?
If so, I can update these tools to use an explicit dependency
for parsing FASTA and FASTQ (e.g. Biopython).

Thanks,

Peter
___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
| Threaded
Open this post in threaded view
|

Re: Tools using Galaxy's Python library?

Nicola Soranzo-2
Hi Peter,
adding

<requirement type="package"
version="1.0.1">galaxy_sequence_utils<requirement>

to each of these tools should solve the problem, there is a Bioconda
package which provides the Python library:

https://anaconda.org/bioconda/galaxy_sequence_utils

Cheers,
Nicola

On 31/01/17 10:39, Peter Cock wrote:

> Hi all,
>
> A few of my tools have for a long time used Galaxy's own parsing
> functionality in order to avoid an external dependency. Lately
> this has stopped working on my TravisCI testing with planemo
> using the Galaxy dev branch (the stable master branch is fine):
> e.g.
>
> https://travis-ci.org/peterjc/pico_galaxy/builds/196655736
>
> The tools fail with things like:
>
> |  from galaxy_utils.sequence.fasta import fastaReader, fastaWriter
> |  ImportError: No module named galaxy_utils.sequence.fasta
>
>
> or:
>
> |  from galaxy_utils.sequence.fastq import fastqReader
> |  ImportError: No module named galaxy_utils.sequence.fastq
>
> Is this a temporary regression in Galaxy, or a deliberate change?
> Do the tools need to do something to explicit have access to the
> Galaxy Python library, or are they now considered private?
> If so, I can update these tools to use an explicit dependency
> for parsing FASTA and FASTQ (e.g. Biopython).
>
> Thanks,
>
> Peter
> ___________________________________________________________
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>    https://lists.galaxyproject.org/
>
> To search Galaxy mailing lists use the unified search at:
>    http://galaxyproject.org/search/mailinglists/

___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
| Threaded
Open this post in threaded view
|

Re: Tools using Galaxy's Python library?

Peter Cock
Thanks Nicola,

I didn't spot the missing slash, but planemo lint did - for anyone
else copy-and-pasting:

<requirement type="package" version="1.0.1">galaxy_sequence_utils</requirement>

For anyone not yet using BioConda with Galaxy, there is also a Tool
Shed entry here:

https://toolshed.g2.bx.psu.edu/view/iuc/package_galaxy_sequence_utils_1_0_1/

I am therefore adding this to my tool_dependencies.xml files:

    <package name="galaxy_sequence_utils" version="1.0.1">
        <repository name="package_galaxy_sequence_utils_1_0_1" owner="iuc" />
    </package>

I still need to work out how best to make this available under TravisCI,
given I am not currently using BioConda for the dependencies.

@IUC: Should this also be released on PyPI for easy install via pip?

Peter


On Tue, Jan 31, 2017 at 11:15 AM, Nicola Soranzo <[hidden email]> wrote:

> Hi Peter,
> adding
>
> <requirement type="package"
> version="1.0.1">galaxy_sequence_utils<requirement>
>
> to each of these tools should solve the problem, there is a Bioconda package
> which provides the Python library:
>
> https://anaconda.org/bioconda/galaxy_sequence_utils
>
> Cheers,
> Nicola
>
>
> On 31/01/17 10:39, Peter Cock wrote:
>>
>> Hi all,
>>
>> A few of my tools have for a long time used Galaxy's own parsing
>> functionality in order to avoid an external dependency. Lately
>> this has stopped working on my TravisCI testing with planemo
>> using the Galaxy dev branch (the stable master branch is fine):
>> e.g.
>>
>> https://travis-ci.org/peterjc/pico_galaxy/builds/196655736
>>
>> The tools fail with things like:
>>
>> |  from galaxy_utils.sequence.fasta import fastaReader, fastaWriter
>> |  ImportError: No module named galaxy_utils.sequence.fasta
>>
>>
>> or:
>>
>> |  from galaxy_utils.sequence.fastq import fastqReader
>> |  ImportError: No module named galaxy_utils.sequence.fastq
>>
>> Is this a temporary regression in Galaxy, or a deliberate change?
>> Do the tools need to do something to explicit have access to the
>> Galaxy Python library, or are they now considered private?
>> If so, I can update these tools to use an explicit dependency
>> for parsing FASTA and FASTQ (e.g. Biopython).
>>
>> Thanks,
>>
>> Peter
>> ___________________________________________________________
>> Please keep all replies on the list by using "reply all"
>> in your mail client.  To manage your subscriptions to this
>> and other Galaxy lists, please use the interface at:
>>    https://lists.galaxyproject.org/
>>
>> To search Galaxy mailing lists use the unified search at:
>>    http://galaxyproject.org/search/mailinglists/
>
>
___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
| Threaded
Open this post in threaded view
|

Re: Tools using Galaxy's Python library?

Martin Čech-2
Hi folks,

the two of the Galaxy-born libraries have sources at our GitHub:


Any improvements or PyPI-readiness work are welcome.

Best,
Martin

On Tue, Jan 31, 2017 at 7:08 AM Peter Cock <[hidden email]> wrote:
Thanks Nicola,

I didn't spot the missing slash, but planemo lint did - for anyone
else copy-and-pasting:

<requirement type="package" version="1.0.1">galaxy_sequence_utils</requirement>

For anyone not yet using BioConda with Galaxy, there is also a Tool
Shed entry here:

https://toolshed.g2.bx.psu.edu/view/iuc/package_galaxy_sequence_utils_1_0_1/

I am therefore adding this to my tool_dependencies.xml files:

    <package name="galaxy_sequence_utils" version="1.0.1">
        <repository name="package_galaxy_sequence_utils_1_0_1" owner="iuc" />
    </package>

I still need to work out how best to make this available under TravisCI,
given I am not currently using BioConda for the dependencies.

@IUC: Should this also be released on PyPI for easy install via pip?

Peter


On Tue, Jan 31, 2017 at 11:15 AM, Nicola Soranzo <[hidden email]> wrote:
> Hi Peter,
> adding
>
> <requirement type="package"
> version="1.0.1">galaxy_sequence_utils<requirement>
>
> to each of these tools should solve the problem, there is a Bioconda package
> which provides the Python library:
>
> https://anaconda.org/bioconda/galaxy_sequence_utils
>
> Cheers,
> Nicola
>
>
> On 31/01/17 10:39, Peter Cock wrote:
>>
>> Hi all,
>>
>> A few of my tools have for a long time used Galaxy's own parsing
>> functionality in order to avoid an external dependency. Lately
>> this has stopped working on my TravisCI testing with planemo
>> using the Galaxy dev branch (the stable master branch is fine):
>> e.g.
>>
>> https://travis-ci.org/peterjc/pico_galaxy/builds/196655736
>>
>> The tools fail with things like:
>>
>> |  from galaxy_utils.sequence.fasta import fastaReader, fastaWriter
>> |  ImportError: No module named galaxy_utils.sequence.fasta
>>
>>
>> or:
>>
>> |  from galaxy_utils.sequence.fastq import fastqReader
>> |  ImportError: No module named galaxy_utils.sequence.fastq
>>
>> Is this a temporary regression in Galaxy, or a deliberate change?
>> Do the tools need to do something to explicit have access to the
>> Galaxy Python library, or are they now considered private?
>> If so, I can update these tools to use an explicit dependency
>> for parsing FASTA and FASTQ (e.g. Biopython).
>>
>> Thanks,
>>
>> Peter
>> ___________________________________________________________
>> Please keep all replies on the list by using "reply all"
>> in your mail client.  To manage your subscriptions to this
>> and other Galaxy lists, please use the interface at:
>>    https://lists.galaxyproject.org/
>>
>> To search Galaxy mailing lists use the unified search at:
>>    http://galaxyproject.org/search/mailinglists/
>
>
___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
| Threaded
Open this post in threaded view
|

Re: Tools using Galaxy's Python library?

John Chilton-4
In reply to this post by Peter Cock
So the PR that broke your tools is probably here https://github.com/galaxyproject/galaxy/pull/3364/files. That pull request removed Galaxy from the Python path of Galaxy tools - this gives tools a much cleaner environment and prevents certain conflicts between Conda and Galaxy.

As part of that PR, there is a list of Galaxy tools in lib/galaxy/tools/__init__.py that still get setup with Galaxy's Python environment. This includes many random devteam and even an IUC tool - I've added something to the list even though one ancient version of tool shed tool required it years ago and it has since been updated. I would open a PR to add the tool id of any of your tools that have been published to the tool shed and require these tools to this list. You can target it against 17.01 ideally and then we can merge that into dev.

This behavior can be reverted - there is a config option that is documented pretty well in the PR but I missed named it in the sample (fixing it with https://github.com/galaxyproject/galaxy/pull/3521/files).

Hopefully this helps and thanks for the bug report! 

-John


On Tue, Jan 31, 2017 at 7:08 AM Peter Cock <[hidden email]> wrote:
Thanks Nicola,

I didn't spot the missing slash, but planemo lint did - for anyone
else copy-and-pasting:

<requirement type="package" version="1.0.1">galaxy_sequence_utils</requirement>

For anyone not yet using BioConda with Galaxy, there is also a Tool
Shed entry here:

https://toolshed.g2.bx.psu.edu/view/iuc/package_galaxy_sequence_utils_1_0_1/

I am therefore adding this to my tool_dependencies.xml files:

    <package name="galaxy_sequence_utils" version="1.0.1">
        <repository name="package_galaxy_sequence_utils_1_0_1" owner="iuc" />
    </package>

I still need to work out how best to make this available under TravisCI,
given I am not currently using BioConda for the dependencies.

@IUC: Should this also be released on PyPI for easy install via pip?

Peter


On Tue, Jan 31, 2017 at 11:15 AM, Nicola Soranzo <[hidden email]> wrote:
> Hi Peter,
> adding
>
> <requirement type="package"
> version="1.0.1">galaxy_sequence_utils<requirement>
>
> to each of these tools should solve the problem, there is a Bioconda package
> which provides the Python library:
>
> https://anaconda.org/bioconda/galaxy_sequence_utils
>
> Cheers,
> Nicola
>
>
> On 31/01/17 10:39, Peter Cock wrote:
>>
>> Hi all,
>>
>> A few of my tools have for a long time used Galaxy's own parsing
>> functionality in order to avoid an external dependency. Lately
>> this has stopped working on my TravisCI testing with planemo
>> using the Galaxy dev branch (the stable master branch is fine):
>> e.g.
>>
>> https://travis-ci.org/peterjc/pico_galaxy/builds/196655736
>>
>> The tools fail with things like:
>>
>> |  from galaxy_utils.sequence.fasta import fastaReader, fastaWriter
>> |  ImportError: No module named galaxy_utils.sequence.fasta
>>
>>
>> or:
>>
>> |  from galaxy_utils.sequence.fastq import fastqReader
>> |  ImportError: No module named galaxy_utils.sequence.fastq
>>
>> Is this a temporary regression in Galaxy, or a deliberate change?
>> Do the tools need to do something to explicit have access to the
>> Galaxy Python library, or are they now considered private?
>> If so, I can update these tools to use an explicit dependency
>> for parsing FASTA and FASTQ (e.g. Biopython).
>>
>> Thanks,
>>
>> Peter
>> ___________________________________________________________
>> Please keep all replies on the list by using "reply all"
>> in your mail client.  To manage your subscriptions to this
>> and other Galaxy lists, please use the interface at:
>>    https://lists.galaxyproject.org/
>>
>> To search Galaxy mailing lists use the unified search at:
>>    http://galaxyproject.org/search/mailinglists/
>
>
___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
| Threaded
Open this post in threaded view
|

Re: Tools using Galaxy's Python library?

Nicola Soranzo-2
In reply to this post by Martin Čech-2
It would also be nice to have gops in Conda, either Bioconda or iuc channel.

Cheers,
Nicola

On 31/01/17 15:01, Martin Čech wrote:
Hi folks,

the two of the Galaxy-born libraries have sources at our GitHub:


Any improvements or PyPI-readiness work are welcome.

Best,
Martin

On Tue, Jan 31, 2017 at 7:08 AM Peter Cock <[hidden email]> wrote:
Thanks Nicola,

I didn't spot the missing slash, but planemo lint did - for anyone
else copy-and-pasting:

<requirement type="package" version="1.0.1">galaxy_sequence_utils</requirement>

For anyone not yet using BioConda with Galaxy, there is also a Tool
Shed entry here:

https://toolshed.g2.bx.psu.edu/view/iuc/package_galaxy_sequence_utils_1_0_1/

I am therefore adding this to my tool_dependencies.xml files:

    <package name="galaxy_sequence_utils" version="1.0.1">
        <repository name="package_galaxy_sequence_utils_1_0_1" owner="iuc" />
    </package>

I still need to work out how best to make this available under TravisCI,
given I am not currently using BioConda for the dependencies.

@IUC: Should this also be released on PyPI for easy install via pip?

Peter


On Tue, Jan 31, 2017 at 11:15 AM, Nicola Soranzo <[hidden email]> wrote:
> Hi Peter,
> adding
>
> <requirement type="package"
> version="1.0.1">galaxy_sequence_utils<requirement>
>
> to each of these tools should solve the problem, there is a Bioconda package
> which provides the Python library:
>
> https://anaconda.org/bioconda/galaxy_sequence_utils
>
> Cheers,
> Nicola
>
>
> On 31/01/17 10:39, Peter Cock wrote:
>>
>> Hi all,
>>
>> A few of my tools have for a long time used Galaxy's own parsing
>> functionality in order to avoid an external dependency. Lately
>> this has stopped working on my TravisCI testing with planemo
>> using the Galaxy dev branch (the stable master branch is fine):
>> e.g.
>>
>> https://travis-ci.org/peterjc/pico_galaxy/builds/196655736
>>
>> The tools fail with things like:
>>
>> |  from galaxy_utils.sequence.fasta import fastaReader, fastaWriter
>> |  ImportError: No module named galaxy_utils.sequence.fasta
>>
>>
>> or:
>>
>> |  from galaxy_utils.sequence.fastq import fastqReader
>> |  ImportError: No module named galaxy_utils.sequence.fastq
>>
>> Is this a temporary regression in Galaxy, or a deliberate change?
>> Do the tools need to do something to explicit have access to the
>> Galaxy Python library, or are they now considered private?
>> If so, I can update these tools to use an explicit dependency
>> for parsing FASTA and FASTQ (e.g. Biopython).
>>
>> Thanks,
>>
>> Peter
>> ___________________________________________________________
>> Please keep all replies on the list by using "reply all"
>> in your mail client.  To manage your subscriptions to this
>> and other Galaxy lists, please use the interface at:
>>    https://lists.galaxyproject.org/
>>
>> To search Galaxy mailing lists use the unified search at:
>>    http://galaxyproject.org/search/mailinglists/
>
>
___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/


___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
| Threaded
Open this post in threaded view
|

Re: Tools using Galaxy's Python library?

Peter Cock
In reply to this post by John Chilton-4
Thanks John - that could explain what's going wrong and offers
another approach to fixing my TravisCI testing - setting:

preserve_python_environment  = legacy_and_local

or:

preserve_python_environment = always

(Despite several attempts I didn't hit on a nice way to get the
galaxy_sequence_utils installed nicely via a "manual install",
one day I will switch my TravisCI testing to use BioConda)

I am hoping to get my TravisCI builds passing again today...

Peter

On Tue, Jan 31, 2017 at 5:27 PM, John Chilton <[hidden email]> wrote:

> So the PR that broke your tools is probably here
> https://github.com/galaxyproject/galaxy/pull/3364/files. That pull request
> removed Galaxy from the Python path of Galaxy tools - this gives tools a
> much cleaner environment and prevents certain conflicts between Conda and
> Galaxy.
>
> As part of that PR, there is a list of Galaxy tools in
> lib/galaxy/tools/__init__.py that still get setup with Galaxy's Python
> environment. This includes many random devteam and even an IUC tool - I've
> added something to the list even though one ancient version of tool shed
> tool required it years ago and it has since been updated. I would open a PR
> to add the tool id of any of your tools that have been published to the tool
> shed and require these tools to this list. You can target it against 17.01
> ideally and then we can merge that into dev.
>
> This behavior can be reverted - there is a config option that is documented
> pretty well in the PR but I missed named it in the sample (fixing it with
> https://github.com/galaxyproject/galaxy/pull/3521/files).
>
> Hopefully this helps and thanks for the bug report!
>
> -John
>
>
> On Tue, Jan 31, 2017 at 7:08 AM Peter Cock <[hidden email]>
> wrote:
>>
>> Thanks Nicola,
>>
>> I didn't spot the missing slash, but planemo lint did - for anyone
>> else copy-and-pasting:
>>
>> <requirement type="package"
>> version="1.0.1">galaxy_sequence_utils</requirement>
>>
>> For anyone not yet using BioConda with Galaxy, there is also a Tool
>> Shed entry here:
>>
>>
>> https://toolshed.g2.bx.psu.edu/view/iuc/package_galaxy_sequence_utils_1_0_1/
>>
>> I am therefore adding this to my tool_dependencies.xml files:
>>
>>     <package name="galaxy_sequence_utils" version="1.0.1">
>>         <repository name="package_galaxy_sequence_utils_1_0_1" owner="iuc"
>> />
>>     </package>
>>
>> I still need to work out how best to make this available under TravisCI,
>> given I am not currently using BioConda for the dependencies.
>>
>> @IUC: Should this also be released on PyPI for easy install via pip?
>>
>> Peter
>>
>>
>> On Tue, Jan 31, 2017 at 11:15 AM, Nicola Soranzo <[hidden email]>
>> wrote:
>> > Hi Peter,
>> > adding
>> >
>> > <requirement type="package"
>> > version="1.0.1">galaxy_sequence_utils<requirement>
>> >
>> > to each of these tools should solve the problem, there is a Bioconda
>> > package
>> > which provides the Python library:
>> >
>> > https://anaconda.org/bioconda/galaxy_sequence_utils
>> >
>> > Cheers,
>> > Nicola
>> >
>> >
>> > On 31/01/17 10:39, Peter Cock wrote:
>> >>
>> >> Hi all,
>> >>
>> >> A few of my tools have for a long time used Galaxy's own parsing
>> >> functionality in order to avoid an external dependency. Lately
>> >> this has stopped working on my TravisCI testing with planemo
>> >> using the Galaxy dev branch (the stable master branch is fine):
>> >> e.g.
>> >>
>> >> https://travis-ci.org/peterjc/pico_galaxy/builds/196655736
>> >>
>> >> The tools fail with things like:
>> >>
>> >> |  from galaxy_utils.sequence.fasta import fastaReader, fastaWriter
>> >> |  ImportError: No module named galaxy_utils.sequence.fasta
>> >>
>> >>
>> >> or:
>> >>
>> >> |  from galaxy_utils.sequence.fastq import fastqReader
>> >> |  ImportError: No module named galaxy_utils.sequence.fastq
>> >>
>> >> Is this a temporary regression in Galaxy, or a deliberate change?
>> >> Do the tools need to do something to explicit have access to the
>> >> Galaxy Python library, or are they now considered private?
>> >> If so, I can update these tools to use an explicit dependency
>> >> for parsing FASTA and FASTQ (e.g. Biopython).
>> >>
>> >> Thanks,
>> >>
>> >> Peter
>> >>
___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
| Threaded
Open this post in threaded view
|

Re: Tools using Galaxy's Python library?

Peter Cock
This went smoother today, and has served to refresh some of the
details of how the Galaxy's venv etc all work.

In my TravisCI setup for tool testing I currently "manually" install
the tool dependencies, and thus am using pip with the same URL
as the Tool Shed package:

pip install http://depot.galaxyproject.org/package/source/galaxy_sequence_utils/galaxy_sequence_utils-1.0.1-st.tgz

Commit:

https://github.com/peterjc/pico_galaxy/commit/c9b77b49281974988c790a54f05c4d1ff81b0904

The TravisCI build is still failing, but on an unrelated change
on the Galaxy dev branch.

Peter

On Wed, Feb 1, 2017 at 9:46 AM, Peter Cock <[hidden email]> wrote:

> Thanks John - that could explain what's going wrong and offers
> another approach to fixing my TravisCI testing - setting:
>
> preserve_python_environment  = legacy_and_local
>
> or:
>
> preserve_python_environment = always
>
> (Despite several attempts I didn't hit on a nice way to get the
> galaxy_sequence_utils installed nicely via a "manual install",
> one day I will switch my TravisCI testing to use BioConda)
>
> I am hoping to get my TravisCI builds passing again today...
>
> Peter
>
> On Tue, Jan 31, 2017 at 5:27 PM, John Chilton <[hidden email]> wrote:
>> So the PR that broke your tools is probably here
>> https://github.com/galaxyproject/galaxy/pull/3364/files. That pull request
>> removed Galaxy from the Python path of Galaxy tools - this gives tools a
>> much cleaner environment and prevents certain conflicts between Conda and
>> Galaxy.
>>
>> As part of that PR, there is a list of Galaxy tools in
>> lib/galaxy/tools/__init__.py that still get setup with Galaxy's Python
>> environment. This includes many random devteam and even an IUC tool - I've
>> added something to the list even though one ancient version of tool shed
>> tool required it years ago and it has since been updated. I would open a PR
>> to add the tool id of any of your tools that have been published to the tool
>> shed and require these tools to this list. You can target it against 17.01
>> ideally and then we can merge that into dev.
>>
>> This behavior can be reverted - there is a config option that is documented
>> pretty well in the PR but I missed named it in the sample (fixing it with
>> https://github.com/galaxyproject/galaxy/pull/3521/files).
>>
>> Hopefully this helps and thanks for the bug report!
>>
>> -John
>>
>>
>> On Tue, Jan 31, 2017 at 7:08 AM Peter Cock <[hidden email]>
>> wrote:
>>>
>>> Thanks Nicola,
>>>
>>> I didn't spot the missing slash, but planemo lint did - for anyone
>>> else copy-and-pasting:
>>>
>>> <requirement type="package"
>>> version="1.0.1">galaxy_sequence_utils</requirement>
>>>
>>> For anyone not yet using BioConda with Galaxy, there is also a Tool
>>> Shed entry here:
>>>
>>>
>>> https://toolshed.g2.bx.psu.edu/view/iuc/package_galaxy_sequence_utils_1_0_1/
>>>
>>> I am therefore adding this to my tool_dependencies.xml files:
>>>
>>>     <package name="galaxy_sequence_utils" version="1.0.1">
>>>         <repository name="package_galaxy_sequence_utils_1_0_1" owner="iuc"
>>> />
>>>     </package>
>>>
>>> I still need to work out how best to make this available under TravisCI,
>>> given I am not currently using BioConda for the dependencies.
>>>
>>> @IUC: Should this also be released on PyPI for easy install via pip?
>>>
>>> Peter
>>>
>>>
>>> On Tue, Jan 31, 2017 at 11:15 AM, Nicola Soranzo <[hidden email]>
>>> wrote:
>>> > Hi Peter,
>>> > adding
>>> >
>>> > <requirement type="package"
>>> > version="1.0.1">galaxy_sequence_utils<requirement>
>>> >
>>> > to each of these tools should solve the problem, there is a Bioconda
>>> > package
>>> > which provides the Python library:
>>> >
>>> > https://anaconda.org/bioconda/galaxy_sequence_utils
>>> >
>>> > Cheers,
>>> > Nicola
>>> >
>>> >
>>> > On 31/01/17 10:39, Peter Cock wrote:
>>> >>
>>> >> Hi all,
>>> >>
>>> >> A few of my tools have for a long time used Galaxy's own parsing
>>> >> functionality in order to avoid an external dependency. Lately
>>> >> this has stopped working on my TravisCI testing with planemo
>>> >> using the Galaxy dev branch (the stable master branch is fine):
>>> >> e.g.
>>> >>
>>> >> https://travis-ci.org/peterjc/pico_galaxy/builds/196655736
>>> >>
>>> >> The tools fail with things like:
>>> >>
>>> >> |  from galaxy_utils.sequence.fasta import fastaReader, fastaWriter
>>> >> |  ImportError: No module named galaxy_utils.sequence.fasta
>>> >>
>>> >>
>>> >> or:
>>> >>
>>> >> |  from galaxy_utils.sequence.fastq import fastqReader
>>> >> |  ImportError: No module named galaxy_utils.sequence.fastq
>>> >>
>>> >> Is this a temporary regression in Galaxy, or a deliberate change?
>>> >> Do the tools need to do something to explicit have access to the
>>> >> Galaxy Python library, or are they now considered private?
>>> >> If so, I can update these tools to use an explicit dependency
>>> >> for parsing FASTA and FASTQ (e.g. Biopython).
>>> >>
>>> >> Thanks,
>>> >>
>>> >> Peter
>>> >>
___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
| Threaded
Open this post in threaded view
|

Re: Tools using Galaxy's Python library?

Peter Cock
To recap, in the next release of Galaxy, there will not be a copy of the
sequence parsing code like lib/galaxy_utils/sequence/fastq.py - see
this pull request: https://github.com/galaxyproject/galaxy/pull/3551

Both Galaxy itself, and any tools needing it, will depend on the now
separately packaged galaxy_sequence_utils library, source code now
here:

https://github.com/galaxyproject/sequence_utils

The original version of this is available on the Tool Shed:

https://toolshed.g2.bx.psu.edu

The library is now available on PyPI too:

https://pypi.python.org/pypi/galaxy_sequence_utils/

This code has been updated and now does "import six" as part of
work to make sure it can be used on Python 2 and Python 3, and
it includes support for compressed FASTQ files.

My TravisCI tool tests are passing again using both Galaxy's stable
and dev branches.

Peter

On Wed, Feb 1, 2017 at 11:20 AM, Peter Cock <[hidden email]> wrote:

> This went smoother today, and has served to refresh some of the
> details of how the Galaxy's venv etc all work.
>
> In my TravisCI setup for tool testing I currently "manually" install
> the tool dependencies, and thus am using pip with the same URL
> as the Tool Shed package:
>
> pip install http://depot.galaxyproject.org/package/source/galaxy_sequence_utils/galaxy_sequence_utils-1.0.1-st.tgz
>
> Commit:
>
> https://github.com/peterjc/pico_galaxy/commit/c9b77b49281974988c790a54f05c4d1ff81b0904
>
> The TravisCI build is still failing, but on an unrelated change
> on the Galaxy dev branch.
>
> Peter
___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/