QIIME tools for Galaxy (WIP, call for collaborators)

classic Classic list List threaded Threaded
10 messages Options
| Threaded
Open this post in threaded view
|

QIIME tools for Galaxy (WIP, call for collaborators)

Lance Parsons
I was recently asked if I could provide a QIIME analysis pipeline for
16S data in Galaxy using tools in the QIIME pipeline (http://qiime.org/).

I did a bit of looking around for existing Galaxy wrappers and found an
application that generates the wrappers for QIIME scripts for Galaxy
(https://github.com/qiime/qiime-galaxy). This is a very well written
application that does a great job of wrapping the QIIME scripts for
Galaxy. However, there are a few things about it that don't quite fit my
needs.

1. The tools output tgz files of all of the output files. This means
that to execute a pipeline, the user would have to download the tgz
files, untar, and then upload whichever file(s) are needed for the next
step.
2. There is no toolshed repository to install the dependencies for these
tools, making it a tricky for administrators to automate and also
maintain various versions of QIIME going forward.
3. There are no toolshed versions of the tools themselves, which also
makes installation and integration a bit tricky and makes it hard to me
to create and manage updates, fixes, tweaks, etc. There are also no
tests, etc.

For these reasons I decided to investigate the feasibility of using the
generated wrappers as a basis for a "toolshed" version of QIIME. If
anyone is interested in helping, or has suggestions, or is working on
something related, I'd be very happy to collaborate.

The repository for the WIP is at
https://github.com/lparsons/galaxy_tools/tree/qiime/tools/qiime1.9.0.
There is also a package on the testtoolshed as well as a first pass at
package_qiime_1_9_1
(https://github.com/lparsons/galaxy_tools/tree/qiime/packages/package_qiime_1_9_1).

--
Lance Parsons - Scientific Programmer
134 Carl C. Icahn Laboratory
Lewis-Sigler Institute for Integrative Genomics
Princeton University

___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
| Threaded
Open this post in threaded view
|

Re: QIIME tools for Galaxy (WIP, call for collaborators)

Daniel Blankenberg
Hi Lance,

I looked at this a bit ago and had similar concerns, particularly with the outputs and inputs not being well-defined. In addition to the output tar ball —> local, extract —> upload not being great, as you mention, the input datatypes, etc, could use some work — in the very least, we should definitely create a nice biom datatype and have some converters available (import and export).  

Definitely worth spending some extra time to make sure that we have the data flowing well between each of the different parts/tools, and even better to make sure that its done in a way that allows mixing and matching with other non-qiime tools.


One thing that we want to avoid is large amounts of manual massaging of the automatically generated xml; fixing things up once might not be too bad, but having to do it with each new tool version can be “frustrating". Although perhaps having a good starting point and only needing to manually modify for any updates could be good enough (I’m not very familiar with the extent of typical changes between qiime versions to make a call on how much changes).




Dan

(resending since I received a message bounce from list)


On Oct 6, 2015, at 9:59 AM, Daniel Blankenberg <[hidden email]> wrote:

> Hi Lance,
>
> I looked at this a bit ago and had similar concerns, particularly with the outputs and inputs not being well-defined. In addition to the output tar ball —> local, extract —> upload not being great, as you mention, the input datatypes, etc, could use some work — in the very least, we should definitely create a nice biom datatype and have some converters available (import and export).  
>
> Definitely worth spending some extra time to make sure that we have the data flowing well between each of the different parts/tools, and even better to make sure that its done in a way that allows mixing and matching with other non-qiime tools.
>
>
> One thing that we want to avoid is large amounts of manual massaging of the automatically generated xml; fixing things up once might not be too bad, but having to do it with each new tool version can be “frustrating". Although perhaps having a good starting point and only needing to manually modify for any updates could be good enough (I’m not very familiar with the extent of typical changes between qiime versions to make a call on how much changes).
>
>
>
>
> Dan
>
>
> On Oct 5, 2015, at 5:26 PM, Lance Parsons <[hidden email]> wrote:
>
>> I was recently asked if I could provide a QIIME analysis pipeline for 16S data in Galaxy using tools in the QIIME pipeline (http://qiime.org/).
>>
>> I did a bit of looking around for existing Galaxy wrappers and found an application that generates the wrappers for QIIME scripts for Galaxy (https://github.com/qiime/qiime-galaxy). This is a very well written application that does a great job of wrapping the QIIME scripts for Galaxy. However, there are a few things about it that don't quite fit my needs.
>>
>> 1. The tools output tgz files of all of the output files. This means that to execute a pipeline, the user would have to download the tgz files, untar, and then upload whichever file(s) are needed for the next step.
>> 2. There is no toolshed repository to install the dependencies for these tools, making it a tricky for administrators to automate and also maintain various versions of QIIME going forward.
>> 3. There are no toolshed versions of the tools themselves, which also makes installation and integration a bit tricky and makes it hard to me to create and manage updates, fixes, tweaks, etc. There are also no tests, etc.
>>
>> For these reasons I decided to investigate the feasibility of using the generated wrappers as a basis for a "toolshed" version of QIIME. If anyone is interested in helping, or has suggestions, or is working on something related, I'd be very happy to collaborate.
>>
>> The repository for the WIP is at https://github.com/lparsons/galaxy_tools/tree/qiime/tools/qiime1.9.0. There is also a package on the testtoolshed as well as a first pass at package_qiime_1_9_1 (https://github.com/lparsons/galaxy_tools/tree/qiime/packages/package_qiime_1_9_1).
>>
>> --
>> Lance Parsons - Scientific Programmer
>> 134 Carl C. Icahn Laboratory
>> Lewis-Sigler Institute for Integrative Genomics
>> Princeton University
>>
>> ___________________________________________________________
>> Please keep all replies on the list by using "reply all"
>> in your mail client.  To manage your subscriptions to this
>> and other Galaxy lists, please use the interface at:
>> https://lists.galaxyproject.org/
>>
>> To search Galaxy mailing lists use the unified search at:
>> http://galaxyproject.org/search/mailinglists/
>

___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
| Threaded
Open this post in threaded view
|

Re: QIIME tools for Galaxy (WIP, call for collaborators)

Lance Parsons
I agree that it would be very nice to get the data flowing between each of the tools and to be able to mix/match with other tools. That is an area of Galaxy tool-dev that I'm less familiar with, so any help would be greatly appreciated.

As for the manual massaging, I agree, however, at this point I'm really hoping that the changes between Qiime versions are manageable, esp. once we have the data types and basic framework of the tools down.

Along that front, I would very much like to get something usable (at least for some specific workflows) done sooner rather than later, then be able to iterate, adding new tools, datatypes, etc. as we move forward. Does that sound reasonable? I love the idea of a Metagenomics toolfest (https://github.com/galaxyproject/tools-iuc/issues/299). One thing I would like to do soon then, would be to define some functionality for the first round. What would be very helpful would be if you, Bjoern, etc. could help make sure the first round lays the proper groundwork so I/we don't have to redo things for later iterations, but we can build on a solid foundation.

Thanks so much for the input, help, etc. It is very much appreciated.

- Lance

October 6, 2015 at 10:21 AM
Hi Lance,

I looked at this a bit ago and had similar concerns, particularly with the outputs and inputs not being well-defined. In addition to the output tar ball —> local, extract —> upload not being great, as you mention, the input datatypes, etc, could use some work — in the very least, we should definitely create a nice biom datatype and have some converters available (import and export).

Definitely worth spending some extra time to make sure that we have the data flowing well between each of the different parts/tools, and even better to make sure that its done in a way that allows mixing and matching with other non-qiime tools.


One thing that we want to avoid is large amounts of manual massaging of the automatically generated xml; fixing things up once might not be too bad, but having to do it with each new tool version can be “frustrating". Although perhaps having a good starting point and only needing to manually modify for any updates could be good enough (I’m not very familiar with the extent of typical changes between qiime versions to make a call on how much changes).




Dan

(resending since I received a message bounce from list)



October 6, 2015 at 9:59 AM
Hi Lance,

I looked at this a bit ago and had similar concerns, particularly with the outputs and inputs not being well-defined. In addition to the output tar ball —> local, extract —> upload not being great, as you mention, the input datatypes, etc, could use some work — in the very least, we should definitely create a nice biom datatype and have some converters available (import and export).

Definitely worth spending some extra time to make sure that we have the data flowing well between each of the different parts/tools, and even better to make sure that its done in a way that allows mixing and matching with other non-qiime tools.


One thing that we want to avoid is large amounts of manual massaging of the automatically generated xml; fixing things up once might not be too bad, but having to do it with each new tool version can be “frustrating". Although perhaps having a good starting point and only needing to manually modify for any updates could be good enough (I’m not very familiar with the extent of typical changes between qiime versions to make a call on how much changes).




Dan



October 5, 2015 at 5:26 PM
I was recently asked if I could provide a QIIME analysis pipeline for 16S data in Galaxy using tools in the QIIME pipeline (http://qiime.org/).

I did a bit of looking around for existing Galaxy wrappers and found an application that generates the wrappers for QIIME scripts for Galaxy (https://github.com/qiime/qiime-galaxy). This is a very well written application that does a great job of wrapping the QIIME scripts for Galaxy. However, there are a few things about it that don't quite fit my needs.

1. The tools output tgz files of all of the output files. This means that to execute a pipeline, the user would have to download the tgz files, untar, and then upload whichever file(s) are needed for the next step.
2. There is no toolshed repository to install the dependencies for these tools, making it a tricky for administrators to automate and also maintain various versions of QIIME going forward.
3. There are no toolshed versions of the tools themselves, which also makes installation and integration a bit tricky and makes it hard to me to create and manage updates, fixes, tweaks, etc. There are also no tests, etc.

For these reasons I decided to investigate the feasibility of using the generated wrappers as a basis for a "toolshed" version of QIIME. If anyone is interested in helping, or has suggestions, or is working on something related, I'd be very happy to collaborate.

The repository for the WIP is at https://github.com/lparsons/galaxy_tools/tree/qiime/tools/qiime1.9.0. There is also a package on the testtoolshed as well as a first pass at package_qiime_1_9_1 (https://github.com/lparsons/galaxy_tools/tree/qiime/packages/package_qiime_1_9_1).


--
Lance Parsons - Scientific Programmer
134 Carl C. Icahn Laboratory
Lewis-Sigler Institute for Integrative Genomics
Princeton University


___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
| Threaded
Open this post in threaded view
|

Re: QIIME tools for Galaxy (WIP, call for collaborators)

Björn Grüning-3
Hi Lance,

I will help if I can. I also think we should begin too write 2-3 nice
tools in the beginning to test the handshake between them and lay the
ground work. This will also help to get familiar with the tool.

I still hope some one else from the community shows interest! :)

Thanks,
Bjoern

Am 06.10.2015 um 20:33 schrieb Lance Parsons:

> I agree that it would be very nice to get the data flowing between each
> of the tools and to be able to mix/match with other tools. That is an
> area of Galaxy tool-dev that I'm less familiar with, so any help would
> be greatly appreciated.
>
> As for the manual massaging, I agree, however, at this point I'm really
> hoping that the changes between Qiime versions are manageable, esp. once
> we have the data types and basic framework of the tools down.
>
> Along that front, I would very much like to get something usable (at
> least for some specific workflows) done sooner rather than later, then
> be able to iterate, adding new tools, datatypes, etc. as we move
> forward. Does that sound reasonable? I love the idea of a Metagenomics
> toolfest (https://github.com/galaxyproject/tools-iuc/issues/299). One
> thing I would like to do soon then, would be to define some
> functionality for the first round. What would be very helpful would be
> if you, Bjoern, etc. could help make sure the first round lays the
> proper groundwork so I/we don't have to redo things for later
> iterations, but we can build on a solid foundation.
>
> Thanks so much for the input, help, etc. It is very much appreciated.
>
> - Lance
>
>> Daniel Blankenberg <mailto:[hidden email]>
>> October 6, 2015 at 10:21 AM
>> Hi Lance,
>>
>> I looked at this a bit ago and had similar concerns, particularly with
>> the outputs and inputs not being well-defined. In addition to the
>> output tar ball —> local, extract —> upload not being great, as you
>> mention, the input datatypes, etc, could use some work — in the very
>> least, we should definitely create a nice biom datatype and have some
>> converters available (import and export).
>>
>> Definitely worth spending some extra time to make sure that we have
>> the data flowing well between each of the different parts/tools, and
>> even better to make sure that its done in a way that allows mixing and
>> matching with other non-qiime tools.
>>
>>
>> One thing that we want to avoid is large amounts of manual massaging
>> of the automatically generated xml; fixing things up once might not be
>> too bad, but having to do it with each new tool version can be
>> “frustrating". Although perhaps having a good starting point and only
>> needing to manually modify for any updates could be good enough (I’m
>> not very familiar with the extent of typical changes between qiime
>> versions to make a call on how much changes).
>>
>>
>>
>>
>> Dan
>>
>> (resending since I received a message bounce from list)
>>
>>
>>
>> Daniel Blankenberg <mailto:[hidden email]>
>> October 6, 2015 at 9:59 AM
>> Hi Lance,
>>
>> I looked at this a bit ago and had similar concerns, particularly with
>> the outputs and inputs not being well-defined. In addition to the
>> output tar ball —> local, extract —> upload not being great, as you
>> mention, the input datatypes, etc, could use some work — in the very
>> least, we should definitely create a nice biom datatype and have some
>> converters available (import and export).
>>
>> Definitely worth spending some extra time to make sure that we have
>> the data flowing well between each of the different parts/tools, and
>> even better to make sure that its done in a way that allows mixing and
>> matching with other non-qiime tools.
>>
>>
>> One thing that we want to avoid is large amounts of manual massaging
>> of the automatically generated xml; fixing things up once might not be
>> too bad, but having to do it with each new tool version can be
>> “frustrating". Although perhaps having a good starting point and only
>> needing to manually modify for any updates could be good enough (I’m
>> not very familiar with the extent of typical changes between qiime
>> versions to make a call on how much changes).
>>
>>
>>
>>
>> Dan
>>
>>
>>
>> Lance Parsons <mailto:[hidden email]>
>> October 5, 2015 at 5:26 PM
>> I was recently asked if I could provide a QIIME analysis pipeline for
>> 16S data in Galaxy using tools in the QIIME pipeline (http://qiime.org/).
>>
>> I did a bit of looking around for existing Galaxy wrappers and found
>> an application that generates the wrappers for QIIME scripts for
>> Galaxy (https://github.com/qiime/qiime-galaxy). This is a very well
>> written application that does a great job of wrapping the QIIME
>> scripts for Galaxy. However, there are a few things about it that
>> don't quite fit my needs.
>>
>> 1. The tools output tgz files of all of the output files. This means
>> that to execute a pipeline, the user would have to download the tgz
>> files, untar, and then upload whichever file(s) are needed for the
>> next step.
>> 2. There is no toolshed repository to install the dependencies for
>> these tools, making it a tricky for administrators to automate and
>> also maintain various versions of QIIME going forward.
>> 3. There are no toolshed versions of the tools themselves, which also
>> makes installation and integration a bit tricky and makes it hard to
>> me to create and manage updates, fixes, tweaks, etc. There are also no
>> tests, etc.
>>
>> For these reasons I decided to investigate the feasibility of using
>> the generated wrappers as a basis for a "toolshed" version of QIIME.
>> If anyone is interested in helping, or has suggestions, or is working
>> on something related, I'd be very happy to collaborate.
>>
>> The repository for the WIP is at
>> https://github.com/lparsons/galaxy_tools/tree/qiime/tools/qiime1.9.0.
>> There is also a package on the testtoolshed as well as a first pass at
>> package_qiime_1_9_1
>> (https://github.com/lparsons/galaxy_tools/tree/qiime/packages/package_qiime_1_9_1).
>>
>>
>
>
>
> ___________________________________________________________
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>   https://lists.galaxyproject.org/
>
> To search Galaxy mailing lists use the unified search at:
>   http://galaxyproject.org/search/mailinglists/
>
___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
| Threaded
Open this post in threaded view
|

Re: QIIME tools for Galaxy (WIP, call for collaborators)

Yvan Le Bras
Hi Lance, Björn, Dan

We are several scientists working on metagenomics in France and at a previous GUGGO  (western France User Group) meeting, we mention the fact to "benchmark" our Galaxy pipelines. One is using new Qiime tools integration, one Mothur and the last one, the recently developped FROGS (Find Rapidly OTU with Galaxy Solution) pipeline using Swarm. We already have had some exchanges with Björn concerning metagenomics flavours on Docker, and I think it will be particularly interesting to exchange one with each other. The Qiime integration was started in march by colleagues from Brest Ifremer centre. I was just trying to install and test FROGS from Toulouse and Jouy-en-Josas INRA labs. If they are not on the list, I can contact them.

All the best,

Yvan

----- Mail original -----

> De: "Björn Grüning" <[hidden email]>
> À: "Lance Parsons" <[hidden email]>, "Daniel Blankenberg" <[hidden email]>
> Cc: "Galaxy Dev List" <[hidden email]>
> Envoyé: Mercredi 7 Octobre 2015 14:25:38
> Objet: Re: [galaxy-dev] QIIME tools for Galaxy (WIP, call for collaborators)
>
> Hi Lance,
>
> I will help if I can. I also think we should begin too write 2-3 nice
> tools in the beginning to test the handshake between them and lay the
> ground work. This will also help to get familiar with the tool.
>
> I still hope some one else from the community shows interest! :)
>
> Thanks,
> Bjoern
>
> Am 06.10.2015 um 20:33 schrieb Lance Parsons:
> > I agree that it would be very nice to get the data flowing between each
> > of the tools and to be able to mix/match with other tools. That is an
> > area of Galaxy tool-dev that I'm less familiar with, so any help would
> > be greatly appreciated.
> >
> > As for the manual massaging, I agree, however, at this point I'm really
> > hoping that the changes between Qiime versions are manageable, esp. once
> > we have the data types and basic framework of the tools down.
> >
> > Along that front, I would very much like to get something usable (at
> > least for some specific workflows) done sooner rather than later, then
> > be able to iterate, adding new tools, datatypes, etc. as we move
> > forward. Does that sound reasonable? I love the idea of a Metagenomics
> > toolfest (https://github.com/galaxyproject/tools-iuc/issues/299). One
> > thing I would like to do soon then, would be to define some
> > functionality for the first round. What would be very helpful would be
> > if you, Bjoern, etc. could help make sure the first round lays the
> > proper groundwork so I/we don't have to redo things for later
> > iterations, but we can build on a solid foundation.
> >
> > Thanks so much for the input, help, etc. It is very much appreciated.
> >
> > - Lance
> >
> >> Daniel Blankenberg <mailto:[hidden email]>
> >> October 6, 2015 at 10:21 AM
> >> Hi Lance,
> >>
> >> I looked at this a bit ago and had similar concerns, particularly with
> >> the outputs and inputs not being well-defined. In addition to the
> >> output tar ball —> local, extract —> upload not being great, as you
> >> mention, the input datatypes, etc, could use some work — in the very
> >> least, we should definitely create a nice biom datatype and have some
> >> converters available (import and export).
> >>
> >> Definitely worth spending some extra time to make sure that we have
> >> the data flowing well between each of the different parts/tools, and
> >> even better to make sure that its done in a way that allows mixing and
> >> matching with other non-qiime tools.
> >>
> >>
> >> One thing that we want to avoid is large amounts of manual massaging
> >> of the automatically generated xml; fixing things up once might not be
> >> too bad, but having to do it with each new tool version can be
> >> “frustrating". Although perhaps having a good starting point and only
> >> needing to manually modify for any updates could be good enough (I’m
> >> not very familiar with the extent of typical changes between qiime
> >> versions to make a call on how much changes).
> >>
> >>
> >>
> >>
> >> Dan
> >>
> >> (resending since I received a message bounce from list)
> >>
> >>
> >>
> >> Daniel Blankenberg <mailto:[hidden email]>
> >> October 6, 2015 at 9:59 AM
> >> Hi Lance,
> >>
> >> I looked at this a bit ago and had similar concerns, particularly with
> >> the outputs and inputs not being well-defined. In addition to the
> >> output tar ball —> local, extract —> upload not being great, as you
> >> mention, the input datatypes, etc, could use some work — in the very
> >> least, we should definitely create a nice biom datatype and have some
> >> converters available (import and export).
> >>
> >> Definitely worth spending some extra time to make sure that we have
> >> the data flowing well between each of the different parts/tools, and
> >> even better to make sure that its done in a way that allows mixing and
> >> matching with other non-qiime tools.
> >>
> >>
> >> One thing that we want to avoid is large amounts of manual massaging
> >> of the automatically generated xml; fixing things up once might not be
> >> too bad, but having to do it with each new tool version can be
> >> “frustrating". Although perhaps having a good starting point and only
> >> needing to manually modify for any updates could be good enough (I’m
> >> not very familiar with the extent of typical changes between qiime
> >> versions to make a call on how much changes).
> >>
> >>
> >>
> >>
> >> Dan
> >>
> >>
> >>
> >> Lance Parsons <mailto:[hidden email]>
> >> October 5, 2015 at 5:26 PM
> >> I was recently asked if I could provide a QIIME analysis pipeline for
> >> 16S data in Galaxy using tools in the QIIME pipeline (http://qiime.org/).
> >>
> >> I did a bit of looking around for existing Galaxy wrappers and found
> >> an application that generates the wrappers for QIIME scripts for
> >> Galaxy (https://github.com/qiime/qiime-galaxy). This is a very well
> >> written application that does a great job of wrapping the QIIME
> >> scripts for Galaxy. However, there are a few things about it that
> >> don't quite fit my needs.
> >>
> >> 1. The tools output tgz files of all of the output files. This means
> >> that to execute a pipeline, the user would have to download the tgz
> >> files, untar, and then upload whichever file(s) are needed for the
> >> next step.
> >> 2. There is no toolshed repository to install the dependencies for
> >> these tools, making it a tricky for administrators to automate and
> >> also maintain various versions of QIIME going forward.
> >> 3. There are no toolshed versions of the tools themselves, which also
> >> makes installation and integration a bit tricky and makes it hard to
> >> me to create and manage updates, fixes, tweaks, etc. There are also no
> >> tests, etc.
> >>
> >> For these reasons I decided to investigate the feasibility of using
> >> the generated wrappers as a basis for a "toolshed" version of QIIME.
> >> If anyone is interested in helping, or has suggestions, or is working
> >> on something related, I'd be very happy to collaborate.
> >>
> >> The repository for the WIP is at
> >> https://github.com/lparsons/galaxy_tools/tree/qiime/tools/qiime1.9.0.
> >> There is also a package on the testtoolshed as well as a first pass at
> >> package_qiime_1_9_1
> >> (https://github.com/lparsons/galaxy_tools/tree/qiime/packages/package_qiime_1_9_1).
> >>
> >>
> >
> >
> >
> > ___________________________________________________________
> > Please keep all replies on the list by using "reply all"
> > in your mail client.  To manage your subscriptions to this
> > and other Galaxy lists, please use the interface at:
> >   https://lists.galaxyproject.org/
> >
> > To search Galaxy mailing lists use the unified search at:
> >   http://galaxyproject.org/search/mailinglists/
> >
> ___________________________________________________________
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>   https://lists.galaxyproject.org/
>
> To search Galaxy mailing lists use the unified search at:
>   http://galaxyproject.org/search/mailinglists/

--
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Yvan Le Bras, PhD @Yvan2935 <°))))><
e-Biogenouest project http://www.e-biogenouest.org 
CNRS UMR 6074 IRISA-INRIA, Campus de Beaulieu, 35042 Rennes Cedex
tél.: +33 (0) 2 99 84 71 79 / +33 (0) 6.10.43.96.51
[hidden email]
___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
| Threaded
Open this post in threaded view
|

Re: QIIME tools for Galaxy (WIP, call for collaborators)

Björn Grüning-3
Hi Yvan,

> Hi Lance, Björn, Dan
>
> We are several scientists working on metagenomics in France and at a
> previous GUGGO  (western France User Group) meeting, we mention the
> fact to "benchmark" our Galaxy pipelines. One is using new Qiime
> tools integration, one Mothur and the last one, the recently
> developped FROGS (Find Rapidly OTU with Galaxy Solution) pipeline
> using Swarm. We already have had some exchanges with Björn concerning
> metagenomics flavours on Docker, and I think it will be particularly
> interesting to exchange one with each other. The Qiime integration
> was started in march by colleagues from Brest Ifremer centre. I was
> just trying to install and test FROGS from Toulouse and Jouy-en-Josas
> INRA labs. If they are not on the list, I can contact them.

Please do so. This is great to know. Please also consider to join our
hackathon:

https://github.com/galaxyproject/tools-iuc/issues/299

We are always open for new ideas. Would be great to get a nice suite and
a few workflows for metagenomic in the near future! :)

Thanks,
Bjoern

> All the best,
>
> Yvan
>
> ----- Mail original -----
>> De: "Björn Grüning" <[hidden email]> À: "Lance Parsons"
>> <[hidden email]>, "Daniel Blankenberg" <[hidden email]> Cc:
>> "Galaxy Dev List" <[hidden email]> Envoyé:
>> Mercredi 7 Octobre 2015 14:25:38 Objet: Re: [galaxy-dev] QIIME
>> tools for Galaxy (WIP, call for collaborators)
>>
>> Hi Lance,
>>
>> I will help if I can. I also think we should begin too write 2-3
>> nice tools in the beginning to test the handshake between them and
>> lay the ground work. This will also help to get familiar with the
>> tool.
>>
>> I still hope some one else from the community shows interest! :)
>>
>> Thanks, Bjoern
>>
>> Am 06.10.2015 um 20:33 schrieb Lance Parsons:
>>> I agree that it would be very nice to get the data flowing
>>> between each of the tools and to be able to mix/match with other
>>> tools. That is an area of Galaxy tool-dev that I'm less familiar
>>> with, so any help would be greatly appreciated.
>>>
>>> As for the manual massaging, I agree, however, at this point I'm
>>> really hoping that the changes between Qiime versions are
>>> manageable, esp. once we have the data types and basic framework
>>> of the tools down.
>>>
>>> Along that front, I would very much like to get something usable
>>> (at least for some specific workflows) done sooner rather than
>>> later, then be able to iterate, adding new tools, datatypes, etc.
>>> as we move forward. Does that sound reasonable? I love the idea
>>> of a Metagenomics toolfest
>>> (https://github.com/galaxyproject/tools-iuc/issues/299). One
>>> thing I would like to do soon then, would be to define some
>>> functionality for the first round. What would be very helpful
>>> would be if you, Bjoern, etc. could help make sure the first
>>> round lays the proper groundwork so I/we don't have to redo
>>> things for later iterations, but we can build on a solid
>>> foundation.
>>>
>>> Thanks so much for the input, help, etc. It is very much
>>> appreciated.
>>>
>>> - Lance
>>>
>>>> Daniel Blankenberg <mailto:[hidden email]> October 6, 2015 at
>>>> 10:21 AM Hi Lance,
>>>>
>>>> I looked at this a bit ago and had similar concerns,
>>>> particularly with the outputs and inputs not being
>>>> well-defined. In addition to the output tar ball —> local,
>>>> extract —> upload not being great, as you mention, the input
>>>> datatypes, etc, could use some work — in the very least, we
>>>> should definitely create a nice biom datatype and have some
>>>> converters available (import and export).
>>>>
>>>> Definitely worth spending some extra time to make sure that we
>>>> have the data flowing well between each of the different
>>>> parts/tools, and even better to make sure that its done in a
>>>> way that allows mixing and matching with other non-qiime
>>>> tools.
>>>>
>>>>
>>>> One thing that we want to avoid is large amounts of manual
>>>> massaging of the automatically generated xml; fixing things up
>>>> once might not be too bad, but having to do it with each new
>>>> tool version can be “frustrating". Although perhaps having a
>>>> good starting point and only needing to manually modify for any
>>>> updates could be good enough (I’m not very familiar with the
>>>> extent of typical changes between qiime versions to make a call
>>>> on how much changes).
>>>>
>>>>
>>>>
>>>>
>>>> Dan
>>>>
>>>> (resending since I received a message bounce from list)
>>>>
>>>>
>>>>
>>>> Daniel Blankenberg <mailto:[hidden email]> October 6, 2015 at
>>>> 9:59 AM Hi Lance,
>>>>
>>>> I looked at this a bit ago and had similar concerns,
>>>> particularly with the outputs and inputs not being
>>>> well-defined. In addition to the output tar ball —> local,
>>>> extract —> upload not being great, as you mention, the input
>>>> datatypes, etc, could use some work — in the very least, we
>>>> should definitely create a nice biom datatype and have some
>>>> converters available (import and export).
>>>>
>>>> Definitely worth spending some extra time to make sure that we
>>>> have the data flowing well between each of the different
>>>> parts/tools, and even better to make sure that its done in a
>>>> way that allows mixing and matching with other non-qiime
>>>> tools.
>>>>
>>>>
>>>> One thing that we want to avoid is large amounts of manual
>>>> massaging of the automatically generated xml; fixing things up
>>>> once might not be too bad, but having to do it with each new
>>>> tool version can be “frustrating". Although perhaps having a
>>>> good starting point and only needing to manually modify for any
>>>> updates could be good enough (I’m not very familiar with the
>>>> extent of typical changes between qiime versions to make a call
>>>> on how much changes).
>>>>
>>>>
>>>>
>>>>
>>>> Dan
>>>>
>>>>
>>>>
>>>> Lance Parsons <mailto:[hidden email]> October 5, 2015
>>>> at 5:26 PM I was recently asked if I could provide a QIIME
>>>> analysis pipeline for 16S data in Galaxy using tools in the
>>>> QIIME pipeline (http://qiime.org/).
>>>>
>>>> I did a bit of looking around for existing Galaxy wrappers and
>>>> found an application that generates the wrappers for QIIME
>>>> scripts for Galaxy (https://github.com/qiime/qiime-galaxy).
>>>> This is a very well written application that does a great job
>>>> of wrapping the QIIME scripts for Galaxy. However, there are a
>>>> few things about it that don't quite fit my needs.
>>>>
>>>> 1. The tools output tgz files of all of the output files. This
>>>> means that to execute a pipeline, the user would have to
>>>> download the tgz files, untar, and then upload whichever
>>>> file(s) are needed for the next step. 2. There is no toolshed
>>>> repository to install the dependencies for these tools, making
>>>> it a tricky for administrators to automate and also maintain
>>>> various versions of QIIME going forward. 3. There are no
>>>> toolshed versions of the tools themselves, which also makes
>>>> installation and integration a bit tricky and makes it hard to
>>>> me to create and manage updates, fixes, tweaks, etc. There are
>>>> also no tests, etc.
>>>>
>>>> For these reasons I decided to investigate the feasibility of
>>>> using the generated wrappers as a basis for a "toolshed"
>>>> version of QIIME. If anyone is interested in helping, or has
>>>> suggestions, or is working on something related, I'd be very
>>>> happy to collaborate.
>>>>
>>>> The repository for the WIP is at
>>>> https://github.com/lparsons/galaxy_tools/tree/qiime/tools/qiime1.9.0.
>>>>
>>>>
There is also a package on the testtoolshed as well as a first pass at
>>>> package_qiime_1_9_1
>>>> (https://github.com/lparsons/galaxy_tools/tree/qiime/packages/package_qiime_1_9_1).
>>>>
>>>>
>>>
>>>
>>>
>>>
>>>>
___________________________________________________________

>>> Please keep all replies on the list by using "reply all" in your
>>> mail client.  To manage your subscriptions to this and other
>>> Galaxy lists, please use the interface at:
>>> https://lists.galaxyproject.org/
>>>
>>> To search Galaxy mailing lists use the unified search at:
>>> http://galaxyproject.org/search/mailinglists/
>>>
>> ___________________________________________________________ Please
>> keep all replies on the list by using "reply all" in your mail
>> client.  To manage your subscriptions to this and other Galaxy
>> lists, please use the interface at:
>> https://lists.galaxyproject.org/
>>
>> To search Galaxy mailing lists use the unified search at:
>> http://galaxyproject.org/search/mailinglists/
>
___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
| Threaded
Open this post in threaded view
|

Re: QIIME tools for Galaxy (WIP, call for collaborators)

Sebastian Schaaf
Hey there,

Any news on this?

My colleague Aarif made me aware of the discussion, because one of our
students, Ashok, worked on this months ago. QIIME was requested by a
project which never started and therefore the developed codes did not
gain priority. Basically we faced the same issues, and at least the
'dead end' problem with the resulting tgz archives was solved. We are
currently assembling the codes into a toolshed entry, but I need to ask
Ashok tomorrow about the current progress.

Is there still interest on it or is everything already on the move..?

Cheers,

Sebastian


Björn Grüning schrieb:

> Hi Yvan,
>
>> Hi Lance, Björn, Dan
>>
>> We are several scientists working on metagenomics in France and at a
>> previous GUGGO  (western France User Group) meeting, we mention the
>> fact to "benchmark" our Galaxy pipelines. One is using new Qiime
>> tools integration, one Mothur and the last one, the recently
>> developped FROGS (Find Rapidly OTU with Galaxy Solution) pipeline
>> using Swarm. We already have had some exchanges with Björn concerning
>> metagenomics flavours on Docker, and I think it will be particularly
>> interesting to exchange one with each other. The Qiime integration
>> was started in march by colleagues from Brest Ifremer centre. I was
>> just trying to install and test FROGS from Toulouse and Jouy-en-Josas
>> INRA labs. If they are not on the list, I can contact them.
> Please do so. This is great to know. Please also consider to join our
> hackathon:
>
> https://github.com/galaxyproject/tools-iuc/issues/299
>
> We are always open for new ideas. Would be great to get a nice suite and
> a few workflows for metagenomic in the near future! :)
>
> Thanks,
> Bjoern
>
>> All the best,
>>
>> Yvan
>>
>> ----- Mail original -----
>>> De: "Björn Grüning" <[hidden email]> À: "Lance Parsons"
>>> <[hidden email]>, "Daniel Blankenberg" <[hidden email]> Cc:
>>> "Galaxy Dev List" <[hidden email]> Envoyé:
>>> Mercredi 7 Octobre 2015 14:25:38 Objet: Re: [galaxy-dev] QIIME
>>> tools for Galaxy (WIP, call for collaborators)
>>>
>>> Hi Lance,
>>>
>>> I will help if I can. I also think we should begin too write 2-3
>>> nice tools in the beginning to test the handshake between them and
>>> lay the ground work. This will also help to get familiar with the
>>> tool.
>>>
>>> I still hope some one else from the community shows interest! :)
>>>
>>> Thanks, Bjoern
>>>
>>> Am 06.10.2015 um 20:33 schrieb Lance Parsons:
>>>> I agree that it would be very nice to get the data flowing
>>>> between each of the tools and to be able to mix/match with other
>>>> tools. That is an area of Galaxy tool-dev that I'm less familiar
>>>> with, so any help would be greatly appreciated.
>>>>
>>>> As for the manual massaging, I agree, however, at this point I'm
>>>> really hoping that the changes between Qiime versions are
>>>> manageable, esp. once we have the data types and basic framework
>>>> of the tools down.
>>>>
>>>> Along that front, I would very much like to get something usable
>>>> (at least for some specific workflows) done sooner rather than
>>>> later, then be able to iterate, adding new tools, datatypes, etc.
>>>> as we move forward. Does that sound reasonable? I love the idea
>>>> of a Metagenomics toolfest
>>>> (https://github.com/galaxyproject/tools-iuc/issues/299). One
>>>> thing I would like to do soon then, would be to define some
>>>> functionality for the first round. What would be very helpful
>>>> would be if you, Bjoern, etc. could help make sure the first
>>>> round lays the proper groundwork so I/we don't have to redo
>>>> things for later iterations, but we can build on a solid
>>>> foundation.
>>>>
>>>> Thanks so much for the input, help, etc. It is very much
>>>> appreciated.
>>>>
>>>> - Lance
>>>>
>>>>> Daniel Blankenberg <mailto:[hidden email]> October 6, 2015 at
>>>>> 10:21 AM Hi Lance,
>>>>>
>>>>> I looked at this a bit ago and had similar concerns,
>>>>> particularly with the outputs and inputs not being
>>>>> well-defined. In addition to the output tar ball —> local,
>>>>> extract —> upload not being great, as you mention, the input
>>>>> datatypes, etc, could use some work — in the very least, we
>>>>> should definitely create a nice biom datatype and have some
>>>>> converters available (import and export).
>>>>>
>>>>> Definitely worth spending some extra time to make sure that we
>>>>> have the data flowing well between each of the different
>>>>> parts/tools, and even better to make sure that its done in a
>>>>> way that allows mixing and matching with other non-qiime
>>>>> tools.
>>>>>
>>>>>
>>>>> One thing that we want to avoid is large amounts of manual
>>>>> massaging of the automatically generated xml; fixing things up
>>>>> once might not be too bad, but having to do it with each new
>>>>> tool version can be “frustrating". Although perhaps having a
>>>>> good starting point and only needing to manually modify for any
>>>>> updates could be good enough (I’m not very familiar with the
>>>>> extent of typical changes between qiime versions to make a call
>>>>> on how much changes).
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> Dan
>>>>>
>>>>> (resending since I received a message bounce from list)
>>>>>
>>>>>
>>>>>
>>>>> Daniel Blankenberg <mailto:[hidden email]> October 6, 2015 at
>>>>> 9:59 AM Hi Lance,
>>>>>
>>>>> I looked at this a bit ago and had similar concerns,
>>>>> particularly with the outputs and inputs not being
>>>>> well-defined. In addition to the output tar ball —> local,
>>>>> extract —> upload not being great, as you mention, the input
>>>>> datatypes, etc, could use some work — in the very least, we
>>>>> should definitely create a nice biom datatype and have some
>>>>> converters available (import and export).
>>>>>
>>>>> Definitely worth spending some extra time to make sure that we
>>>>> have the data flowing well between each of the different
>>>>> parts/tools, and even better to make sure that its done in a
>>>>> way that allows mixing and matching with other non-qiime
>>>>> tools.
>>>>>
>>>>>
>>>>> One thing that we want to avoid is large amounts of manual
>>>>> massaging of the automatically generated xml; fixing things up
>>>>> once might not be too bad, but having to do it with each new
>>>>> tool version can be “frustrating". Although perhaps having a
>>>>> good starting point and only needing to manually modify for any
>>>>> updates could be good enough (I’m not very familiar with the
>>>>> extent of typical changes between qiime versions to make a call
>>>>> on how much changes).
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> Dan
>>>>>
>>>>>
>>>>>
>>>>> Lance Parsons <mailto:[hidden email]> October 5, 2015
>>>>> at 5:26 PM I was recently asked if I could provide a QIIME
>>>>> analysis pipeline for 16S data in Galaxy using tools in the
>>>>> QIIME pipeline (http://qiime.org/).
>>>>>
>>>>> I did a bit of looking around for existing Galaxy wrappers and
>>>>> found an application that generates the wrappers for QIIME
>>>>> scripts for Galaxy (https://github.com/qiime/qiime-galaxy).
>>>>> This is a very well written application that does a great job
>>>>> of wrapping the QIIME scripts for Galaxy. However, there are a
>>>>> few things about it that don't quite fit my needs.
>>>>>
>>>>> 1. The tools output tgz files of all of the output files. This
>>>>> means that to execute a pipeline, the user would have to
>>>>> download the tgz files, untar, and then upload whichever
>>>>> file(s) are needed for the next step. 2. There is no toolshed
>>>>> repository to install the dependencies for these tools, making
>>>>> it a tricky for administrators to automate and also maintain
>>>>> various versions of QIIME going forward. 3. There are no
>>>>> toolshed versions of the tools themselves, which also makes
>>>>> installation and integration a bit tricky and makes it hard to
>>>>> me to create and manage updates, fixes, tweaks, etc. There are
>>>>> also no tests, etc.
>>>>>
>>>>> For these reasons I decided to investigate the feasibility of
>>>>> using the generated wrappers as a basis for a "toolshed"
>>>>> version of QIIME. If anyone is interested in helping, or has
>>>>> suggestions, or is working on something related, I'd be very
>>>>> happy to collaborate.
>>>>>
>>>>> The repository for the WIP is at
>>>>> https://github.com/lparsons/galaxy_tools/tree/qiime/tools/qiime1.9.0.
>>>>>
>>>>>
> There is also a package on the testtoolshed as well as a first pass at
>>>>> package_qiime_1_9_1
>>>>> (https://github.com/lparsons/galaxy_tools/tree/qiime/packages/package_qiime_1_9_1).
>>>>>
>>>>>
>>>>
>>>>
>>>>
> ___________________________________________________________
>>>> Please keep all replies on the list by using "reply all" in your
>>>> mail client.  To manage your subscriptions to this and other
>>>> Galaxy lists, please use the interface at:
>>>> https://lists.galaxyproject.org/
>>>>
>>>> To search Galaxy mailing lists use the unified search at:
>>>> http://galaxyproject.org/search/mailinglists/
>>>>
>>> ___________________________________________________________ Please
>>> keep all replies on the list by using "reply all" in your mail
>>> client.  To manage your subscriptions to this and other Galaxy
>>> lists, please use the interface at:
>>> https://lists.galaxyproject.org/
>>>
>>> To search Galaxy mailing lists use the unified search at:
>>> http://galaxyproject.org/search/mailinglists/
> ___________________________________________________________
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>    https://lists.galaxyproject.org/
>
> To search Galaxy mailing lists use the unified search at:
>    http://galaxyproject.org/search/mailinglists/

--
Sebastian Schaaf, M.Sc. Bioinformatics
Faculty Coordinator NGS Infrastructure
Chair of Biometry and Bioinformatics
Department of Medical Informatics,
  Biometry and Epidemiology (IBE)
University of Munich
DKTK Munich
Marchioninistr. 15, K U1 808
D-81377 Munich (Germany)
Tel: +49 89 4400 77499

___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
| Threaded
Open this post in threaded view
|

Re: QIIME tools for Galaxy (WIP, call for collaborators)

Björn Grüning-3
Hi Sebastian,

I think we try to work on this during our upcoming hackathon.

https://github.com/galaxyproject/tools-iuc/issues/299

It would be nice to join and contribute your work to it.

Thanks,
Bjoern

Am 13.10.2015 um 15:03 schrieb Sebastian Schaaf:

> Hey there,
>
> Any news on this?
>
> My colleague Aarif made me aware of the discussion, because one of our
> students, Ashok, worked on this months ago. QIIME was requested by a
> project which never started and therefore the developed codes did not
> gain priority. Basically we faced the same issues, and at least the
> 'dead end' problem with the resulting tgz archives was solved. We are
> currently assembling the codes into a toolshed entry, but I need to ask
> Ashok tomorrow about the current progress.
>
> Is there still interest on it or is everything already on the move..?
>
> Cheers,
>
> Sebastian
>
>
> Björn Grüning schrieb:
>> Hi Yvan,
>>
>>> Hi Lance, Björn, Dan
>>>
>>> We are several scientists working on metagenomics in France and at a
>>> previous GUGGO  (western France User Group) meeting, we mention the
>>> fact to "benchmark" our Galaxy pipelines. One is using new Qiime
>>> tools integration, one Mothur and the last one, the recently
>>> developped FROGS (Find Rapidly OTU with Galaxy Solution) pipeline
>>> using Swarm. We already have had some exchanges with Björn concerning
>>> metagenomics flavours on Docker, and I think it will be particularly
>>> interesting to exchange one with each other. The Qiime integration
>>> was started in march by colleagues from Brest Ifremer centre. I was
>>> just trying to install and test FROGS from Toulouse and Jouy-en-Josas
>>> INRA labs. If they are not on the list, I can contact them.
>> Please do so. This is great to know. Please also consider to join our
>> hackathon:
>>
>> https://github.com/galaxyproject/tools-iuc/issues/299
>>
>> We are always open for new ideas. Would be great to get a nice suite and
>> a few workflows for metagenomic in the near future! :)
>>
>> Thanks,
>> Bjoern
>>
>>> All the best,
>>>
>>> Yvan
>>>
>>> ----- Mail original -----
>>>> De: "Björn Grüning" <[hidden email]> À: "Lance Parsons"
>>>> <[hidden email]>, "Daniel Blankenberg" <[hidden email]> Cc:
>>>> "Galaxy Dev List" <[hidden email]> Envoyé:
>>>> Mercredi 7 Octobre 2015 14:25:38 Objet: Re: [galaxy-dev] QIIME
>>>> tools for Galaxy (WIP,    call for collaborators)
>>>>
>>>> Hi Lance,
>>>>
>>>> I will help if I can. I also think we should begin too write 2-3
>>>> nice tools in the beginning to test the handshake between them and
>>>> lay the ground work. This will also help to get familiar with the
>>>> tool.
>>>>
>>>> I still hope some one else from the community shows interest! :)
>>>>
>>>> Thanks, Bjoern
>>>>
>>>> Am 06.10.2015 um 20:33 schrieb Lance Parsons:
>>>>> I agree that it would be very nice to get the data flowing
>>>>> between each of the tools and to be able to mix/match with other
>>>>> tools. That is an area of Galaxy tool-dev that I'm less familiar
>>>>> with, so any help would be greatly appreciated.
>>>>>
>>>>> As for the manual massaging, I agree, however, at this point I'm
>>>>> really hoping that the changes between Qiime versions are
>>>>> manageable, esp. once we have the data types and basic framework
>>>>> of the tools down.
>>>>>
>>>>> Along that front, I would very much like to get something usable
>>>>> (at least for some specific workflows) done sooner rather than
>>>>> later, then be able to iterate, adding new tools, datatypes, etc.
>>>>> as we move forward. Does that sound reasonable? I love the idea
>>>>> of a Metagenomics toolfest
>>>>> (https://github.com/galaxyproject/tools-iuc/issues/299). One
>>>>> thing I would like to do soon then, would be to define some
>>>>> functionality for the first round. What would be very helpful
>>>>> would be if you, Bjoern, etc. could help make sure the first
>>>>> round lays the proper groundwork so I/we don't have to redo
>>>>> things for later iterations, but we can build on a solid
>>>>> foundation.
>>>>>
>>>>> Thanks so much for the input, help, etc. It is very much
>>>>> appreciated.
>>>>>
>>>>> - Lance
>>>>>
>>>>>> Daniel Blankenberg <mailto:[hidden email]> October 6, 2015 at
>>>>>> 10:21 AM Hi Lance,
>>>>>>
>>>>>> I looked at this a bit ago and had similar concerns,
>>>>>> particularly with the outputs and inputs not being
>>>>>> well-defined. In addition to the output tar ball —> local,
>>>>>> extract —> upload not being great, as you mention, the input
>>>>>> datatypes, etc, could use some work — in the very least, we
>>>>>> should definitely create a nice biom datatype and have some
>>>>>> converters available (import and export).
>>>>>>
>>>>>> Definitely worth spending some extra time to make sure that we
>>>>>> have the data flowing well between each of the different
>>>>>> parts/tools, and even better to make sure that its done in a
>>>>>> way that allows mixing and matching with other non-qiime
>>>>>> tools.
>>>>>>
>>>>>>
>>>>>> One thing that we want to avoid is large amounts of manual
>>>>>> massaging of the automatically generated xml; fixing things up
>>>>>> once might not be too bad, but having to do it with each new
>>>>>> tool version can be “frustrating". Although perhaps having a
>>>>>> good starting point and only needing to manually modify for any
>>>>>> updates could be good enough (I’m not very familiar with the
>>>>>> extent of typical changes between qiime versions to make a call
>>>>>> on how much changes).
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> Dan
>>>>>>
>>>>>> (resending since I received a message bounce from list)
>>>>>>
>>>>>>
>>>>>>
>>>>>> Daniel Blankenberg <mailto:[hidden email]> October 6, 2015 at
>>>>>> 9:59 AM Hi Lance,
>>>>>>
>>>>>> I looked at this a bit ago and had similar concerns,
>>>>>> particularly with the outputs and inputs not being
>>>>>> well-defined. In addition to the output tar ball —> local,
>>>>>> extract —> upload not being great, as you mention, the input
>>>>>> datatypes, etc, could use some work — in the very least, we
>>>>>> should definitely create a nice biom datatype and have some
>>>>>> converters available (import and export).
>>>>>>
>>>>>> Definitely worth spending some extra time to make sure that we
>>>>>> have the data flowing well between each of the different
>>>>>> parts/tools, and even better to make sure that its done in a
>>>>>> way that allows mixing and matching with other non-qiime
>>>>>> tools.
>>>>>>
>>>>>>
>>>>>> One thing that we want to avoid is large amounts of manual
>>>>>> massaging of the automatically generated xml; fixing things up
>>>>>> once might not be too bad, but having to do it with each new
>>>>>> tool version can be “frustrating". Although perhaps having a
>>>>>> good starting point and only needing to manually modify for any
>>>>>> updates could be good enough (I’m not very familiar with the
>>>>>> extent of typical changes between qiime versions to make a call
>>>>>> on how much changes).
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> Dan
>>>>>>
>>>>>>
>>>>>>
>>>>>> Lance Parsons <mailto:[hidden email]> October 5, 2015
>>>>>> at 5:26 PM I was recently asked if I could provide a QIIME
>>>>>> analysis pipeline for 16S data in Galaxy using tools in the
>>>>>> QIIME pipeline (http://qiime.org/).
>>>>>>
>>>>>> I did a bit of looking around for existing Galaxy wrappers and
>>>>>> found an application that generates the wrappers for QIIME
>>>>>> scripts for Galaxy (https://github.com/qiime/qiime-galaxy).
>>>>>> This is a very well written application that does a great job
>>>>>> of wrapping the QIIME scripts for Galaxy. However, there are a
>>>>>> few things about it that don't quite fit my needs.
>>>>>>
>>>>>> 1. The tools output tgz files of all of the output files. This
>>>>>> means that to execute a pipeline, the user would have to
>>>>>> download the tgz files, untar, and then upload whichever
>>>>>> file(s) are needed for the next step. 2. There is no toolshed
>>>>>> repository to install the dependencies for these tools, making
>>>>>> it a tricky for administrators to automate and also maintain
>>>>>> various versions of QIIME going forward. 3. There are no
>>>>>> toolshed versions of the tools themselves, which also makes
>>>>>> installation and integration a bit tricky and makes it hard to
>>>>>> me to create and manage updates, fixes, tweaks, etc. There are
>>>>>> also no tests, etc.
>>>>>>
>>>>>> For these reasons I decided to investigate the feasibility of
>>>>>> using the generated wrappers as a basis for a "toolshed"
>>>>>> version of QIIME. If anyone is interested in helping, or has
>>>>>> suggestions, or is working on something related, I'd be very
>>>>>> happy to collaborate.
>>>>>>
>>>>>> The repository for the WIP is at
>>>>>> https://github.com/lparsons/galaxy_tools/tree/qiime/tools/qiime1.9.0.
>>>>>>
>>>>>>
>> There is also a package on the testtoolshed as well as a first pass at
>>>>>> package_qiime_1_9_1
>>>>>> (https://github.com/lparsons/galaxy_tools/tree/qiime/packages/package_qiime_1_9_1).
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>>
>> ___________________________________________________________
>>>>> Please keep all replies on the list by using "reply all" in your
>>>>> mail client.  To manage your subscriptions to this and other
>>>>> Galaxy lists, please use the interface at:
>>>>> https://lists.galaxyproject.org/
>>>>>
>>>>> To search Galaxy mailing lists use the unified search at:
>>>>> http://galaxyproject.org/search/mailinglists/
>>>>>
>>>> ___________________________________________________________ Please
>>>> keep all replies on the list by using "reply all" in your mail
>>>> client.  To manage your subscriptions to this and other Galaxy
>>>> lists, please use the interface at:
>>>> https://lists.galaxyproject.org/
>>>>
>>>> To search Galaxy mailing lists use the unified search at:
>>>> http://galaxyproject.org/search/mailinglists/
>> ___________________________________________________________
>> Please keep all replies on the list by using "reply all"
>> in your mail client.  To manage your subscriptions to this
>> and other Galaxy lists, please use the interface at:
>>    https://lists.galaxyproject.org/
>>
>> To search Galaxy mailing lists use the unified search at:
>>    http://galaxyproject.org/search/mailinglists/
>
___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
| Threaded
Open this post in threaded view
|

Re: QIIME tools for Galaxy (WIP, call for collaborators)

Sebastian Schaaf
Hi Björn,

The discussion looks good, although (or right because?) it gained a
somewhat more generic scope, where QIIME is one of several construction
sites in the metagenomics field. Do I get it right that the date for the
hackathon is tba? Will there be a notification? I think we would like to
join :).

Anyhow we discussed yesterday in the office and came up the following:

1. Ashok collected all dependencies (names/versions of tools/libraries),
he also formalized them in recipes. The community has to check whether
it fits the needs/expectations, as we are relatively new to tool sheds
and started to load a private one recently with tools. But the plain
list should already save some work. As we run SLES as OS (which is close
to blank in terms of up to date software...) the dependency list should
go quite far.

2. The codes which were written by him for an internship will anyhow get
some improvements: currently the "qiime_connector" is simply a kind of
archive extractor which makes an arbitrary file from a QIIME tool output
archive available in the Galaxy history, so that there is neither a dead
end nor the need for a "download/upload" step. For user who know QIIME
from the command line (and therefore know how the modules work) this may
be a suitable solution, but not for beginners who start with
metagenomics. Somehow it does not feel "galaxy'sh"...

So far, looking forward to joining forces :).

Cheers,
Sebastian


Björn Grüning schrieb:

> Hi Sebastian,
>
> I think we try to work on this during our upcoming hackathon.
>
> https://github.com/galaxyproject/tools-iuc/issues/299
>
> It would be nice to join and contribute your work to it.
>
> Thanks,
> Bjoern
>
> Am 13.10.2015 um 15:03 schrieb Sebastian Schaaf:
>> Hey there,
>>
>> Any news on this?
>>
>> My colleague Aarif made me aware of the discussion, because one of our
>> students, Ashok, worked on this months ago. QIIME was requested by a
>> project which never started and therefore the developed codes did not
>> gain priority. Basically we faced the same issues, and at least the
>> 'dead end' problem with the resulting tgz archives was solved. We are
>> currently assembling the codes into a toolshed entry, but I need to ask
>> Ashok tomorrow about the current progress.
>>
>> Is there still interest on it or is everything already on the move..?
>>
>> Cheers,
>>
>> Sebastian
>>
>>
>> Björn Grüning schrieb:
>>> Hi Yvan,
>>>
>>>> Hi Lance, Björn, Dan
>>>>
>>>> We are several scientists working on metagenomics in France and at a
>>>> previous GUGGO  (western France User Group) meeting, we mention the
>>>> fact to "benchmark" our Galaxy pipelines. One is using new Qiime
>>>> tools integration, one Mothur and the last one, the recently
>>>> developped FROGS (Find Rapidly OTU with Galaxy Solution) pipeline
>>>> using Swarm. We already have had some exchanges with Björn concerning
>>>> metagenomics flavours on Docker, and I think it will be particularly
>>>> interesting to exchange one with each other. The Qiime integration
>>>> was started in march by colleagues from Brest Ifremer centre. I was
>>>> just trying to install and test FROGS from Toulouse and Jouy-en-Josas
>>>> INRA labs. If they are not on the list, I can contact them.
>>> Please do so. This is great to know. Please also consider to join our
>>> hackathon:
>>>
>>> https://github.com/galaxyproject/tools-iuc/issues/299
>>>
>>> We are always open for new ideas. Would be great to get a nice suite and
>>> a few workflows for metagenomic in the near future! :)
>>>
>>> Thanks,
>>> Bjoern
>>>
>>>> All the best,
>>>>
>>>> Yvan
>>>>
>>>> ----- Mail original -----
>>>>> De: "Björn Grüning" <[hidden email]> À: "Lance Parsons"
>>>>> <[hidden email]>, "Daniel Blankenberg" <[hidden email]> Cc:
>>>>> "Galaxy Dev List" <[hidden email]> Envoyé:
>>>>> Mercredi 7 Octobre 2015 14:25:38 Objet: Re: [galaxy-dev] QIIME
>>>>> tools for Galaxy (WIP,    call for collaborators)
>>>>>
>>>>> Hi Lance,
>>>>>
>>>>> I will help if I can. I also think we should begin too write 2-3
>>>>> nice tools in the beginning to test the handshake between them and
>>>>> lay the ground work. This will also help to get familiar with the
>>>>> tool.
>>>>>
>>>>> I still hope some one else from the community shows interest! :)
>>>>>
>>>>> Thanks, Bjoern
>>>>>
>>>>> Am 06.10.2015 um 20:33 schrieb Lance Parsons:
>>>>>> I agree that it would be very nice to get the data flowing
>>>>>> between each of the tools and to be able to mix/match with other
>>>>>> tools. That is an area of Galaxy tool-dev that I'm less familiar
>>>>>> with, so any help would be greatly appreciated.
>>>>>>
>>>>>> As for the manual massaging, I agree, however, at this point I'm
>>>>>> really hoping that the changes between Qiime versions are
>>>>>> manageable, esp. once we have the data types and basic framework
>>>>>> of the tools down.
>>>>>>
>>>>>> Along that front, I would very much like to get something usable
>>>>>> (at least for some specific workflows) done sooner rather than
>>>>>> later, then be able to iterate, adding new tools, datatypes, etc.
>>>>>> as we move forward. Does that sound reasonable? I love the idea
>>>>>> of a Metagenomics toolfest
>>>>>> (https://github.com/galaxyproject/tools-iuc/issues/299). One
>>>>>> thing I would like to do soon then, would be to define some
>>>>>> functionality for the first round. What would be very helpful
>>>>>> would be if you, Bjoern, etc. could help make sure the first
>>>>>> round lays the proper groundwork so I/we don't have to redo
>>>>>> things for later iterations, but we can build on a solid
>>>>>> foundation.
>>>>>>
>>>>>> Thanks so much for the input, help, etc. It is very much
>>>>>> appreciated.
>>>>>>
>>>>>> - Lance
>>>>>>
>>>>>>> Daniel Blankenberg <mailto:[hidden email]> October 6, 2015 at
>>>>>>> 10:21 AM Hi Lance,
>>>>>>>
>>>>>>> I looked at this a bit ago and had similar concerns,
>>>>>>> particularly with the outputs and inputs not being
>>>>>>> well-defined. In addition to the output tar ball —> local,
>>>>>>> extract —> upload not being great, as you mention, the input
>>>>>>> datatypes, etc, could use some work — in the very least, we
>>>>>>> should definitely create a nice biom datatype and have some
>>>>>>> converters available (import and export).
>>>>>>>
>>>>>>> Definitely worth spending some extra time to make sure that we
>>>>>>> have the data flowing well between each of the different
>>>>>>> parts/tools, and even better to make sure that its done in a
>>>>>>> way that allows mixing and matching with other non-qiime
>>>>>>> tools.
>>>>>>>
>>>>>>>
>>>>>>> One thing that we want to avoid is large amounts of manual
>>>>>>> massaging of the automatically generated xml; fixing things up
>>>>>>> once might not be too bad, but having to do it with each new
>>>>>>> tool version can be “frustrating". Although perhaps having a
>>>>>>> good starting point and only needing to manually modify for any
>>>>>>> updates could be good enough (I’m not very familiar with the
>>>>>>> extent of typical changes between qiime versions to make a call
>>>>>>> on how much changes).
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Dan
>>>>>>>
>>>>>>> (resending since I received a message bounce from list)
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Daniel Blankenberg <mailto:[hidden email]> October 6, 2015 at
>>>>>>> 9:59 AM Hi Lance,
>>>>>>>
>>>>>>> I looked at this a bit ago and had similar concerns,
>>>>>>> particularly with the outputs and inputs not being
>>>>>>> well-defined. In addition to the output tar ball —> local,
>>>>>>> extract —> upload not being great, as you mention, the input
>>>>>>> datatypes, etc, could use some work — in the very least, we
>>>>>>> should definitely create a nice biom datatype and have some
>>>>>>> converters available (import and export).
>>>>>>>
>>>>>>> Definitely worth spending some extra time to make sure that we
>>>>>>> have the data flowing well between each of the different
>>>>>>> parts/tools, and even better to make sure that its done in a
>>>>>>> way that allows mixing and matching with other non-qiime
>>>>>>> tools.
>>>>>>>
>>>>>>>
>>>>>>> One thing that we want to avoid is large amounts of manual
>>>>>>> massaging of the automatically generated xml; fixing things up
>>>>>>> once might not be too bad, but having to do it with each new
>>>>>>> tool version can be “frustrating". Although perhaps having a
>>>>>>> good starting point and only needing to manually modify for any
>>>>>>> updates could be good enough (I’m not very familiar with the
>>>>>>> extent of typical changes between qiime versions to make a call
>>>>>>> on how much changes).
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Dan
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Lance Parsons <mailto:[hidden email]> October 5, 2015
>>>>>>> at 5:26 PM I was recently asked if I could provide a QIIME
>>>>>>> analysis pipeline for 16S data in Galaxy using tools in the
>>>>>>> QIIME pipeline (http://qiime.org/).
>>>>>>>
>>>>>>> I did a bit of looking around for existing Galaxy wrappers and
>>>>>>> found an application that generates the wrappers for QIIME
>>>>>>> scripts for Galaxy (https://github.com/qiime/qiime-galaxy).
>>>>>>> This is a very well written application that does a great job
>>>>>>> of wrapping the QIIME scripts for Galaxy. However, there are a
>>>>>>> few things about it that don't quite fit my needs.
>>>>>>>
>>>>>>> 1. The tools output tgz files of all of the output files. This
>>>>>>> means that to execute a pipeline, the user would have to
>>>>>>> download the tgz files, untar, and then upload whichever
>>>>>>> file(s) are needed for the next step. 2. There is no toolshed
>>>>>>> repository to install the dependencies for these tools, making
>>>>>>> it a tricky for administrators to automate and also maintain
>>>>>>> various versions of QIIME going forward. 3. There are no
>>>>>>> toolshed versions of the tools themselves, which also makes
>>>>>>> installation and integration a bit tricky and makes it hard to
>>>>>>> me to create and manage updates, fixes, tweaks, etc. There are
>>>>>>> also no tests, etc.
>>>>>>>
>>>>>>> For these reasons I decided to investigate the feasibility of
>>>>>>> using the generated wrappers as a basis for a "toolshed"
>>>>>>> version of QIIME. If anyone is interested in helping, or has
>>>>>>> suggestions, or is working on something related, I'd be very
>>>>>>> happy to collaborate.
>>>>>>>
>>>>>>> The repository for the WIP is at
>>>>>>> https://github.com/lparsons/galaxy_tools/tree/qiime/tools/qiime1.9.0.
>>>>>>>
>>>>>>>
>>> There is also a package on the testtoolshed as well as a first pass at
>>>>>>> package_qiime_1_9_1
>>>>>>> (https://github.com/lparsons/galaxy_tools/tree/qiime/packages/package_qiime_1_9_1).
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>> ___________________________________________________________
>>>>>> Please keep all replies on the list by using "reply all" in your
>>>>>> mail client.  To manage your subscriptions to this and other
>>>>>> Galaxy lists, please use the interface at:
>>>>>> https://lists.galaxyproject.org/
>>>>>>
>>>>>> To search Galaxy mailing lists use the unified search at:
>>>>>> http://galaxyproject.org/search/mailinglists/
>>>>>>
>>>>> ___________________________________________________________ Please
>>>>> keep all replies on the list by using "reply all" in your mail
>>>>> client.  To manage your subscriptions to this and other Galaxy
>>>>> lists, please use the interface at:
>>>>> https://lists.galaxyproject.org/
>>>>>
>>>>> To search Galaxy mailing lists use the unified search at:
>>>>> http://galaxyproject.org/search/mailinglists/
>>> ___________________________________________________________
>>> Please keep all replies on the list by using "reply all"
>>> in your mail client.  To manage your subscriptions to this
>>> and other Galaxy lists, please use the interface at:
>>>     https://lists.galaxyproject.org/
>>>
>>> To search Galaxy mailing lists use the unified search at:
>>>     http://galaxyproject.org/search/mailinglists/

--
Sebastian Schaaf, M.Sc. Bioinformatics
Faculty Coordinator NGS Infrastructure
Chair of Biometry and Bioinformatics
Department of Medical Informatics,
  Biometry and Epidemiology (IBE)
University of Munich
DKTK Munich
Marchioninistr. 15, K U1 808
D-81377 Munich (Germany)
Tel: +49 89 4400 77499

___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
| Threaded
Open this post in threaded view
|

Re: QIIME tools for Galaxy (WIP, call for collaborators)

Björn Grüning-3


Am 15.10.2015 um 09:46 schrieb Sebastian Schaaf:
> Hi Björn,
>
> The discussion looks good, although (or right because?) it gained a
> somewhat more generic scope, where QIIME is one of several construction
> sites in the metagenomics field. Do I get it right that the date for the
> hackathon is tba? Will there be a notification? I think we would like to
> join :).

Yes, we are trying find a schedule :)
Subscribe to the github issue and you will know. We will also announce
it via twitter and this ML.

> Anyhow we discussed yesterday in the office and came up the following:
>
> 1. Ashok collected all dependencies (names/versions of tools/libraries),
> he also formalized them in recipes. The community has to check whether
> it fits the needs/expectations, as we are relatively new to tool sheds
> and started to load a private one recently with tools. But the plain
> list should already save some work. As we run SLES as OS (which is close
> to blank in terms of up to date software...) the dependency list should
> go quite far.


Awesome. Please see also this one here:
https://github.com/galaxyproject/tools-iuc/pull/297

It takes care of all python libs. Testing and enhancements are welcome!
Depending on how far we will get with wrapping tools we also need the
other dependencies.


> 2. The codes which were written by him for an internship will anyhow get
> some improvements: currently the "qiime_connector" is simply a kind of
> archive extractor which makes an arbitrary file from a QIIME tool output
> archive available in the Galaxy history, so that there is neither a dead
> end nor the need for a "download/upload" step. For user who know QIIME
> from the command line (and therefore know how the modules work) this may
> be a suitable solution, but not for beginners who start with
> metagenomics. Somehow it does not feel "galaxy'sh"...

Also a nice ideas. I guess we discuss this in more detail during the
hackathon and trying to get some proper filetypes into Galaxy for this.

> So far, looking forward to joining forces :).

!!!
Greetings to Munich!
Bjoern

> Cheers,
> Sebastian
>
>
> Björn Grüning schrieb:
>> Hi Sebastian,
>>
>> I think we try to work on this during our upcoming hackathon.
>>
>> https://github.com/galaxyproject/tools-iuc/issues/299
>>
>> It would be nice to join and contribute your work to it.
>>
>> Thanks,
>> Bjoern
>>
>> Am 13.10.2015 um 15:03 schrieb Sebastian Schaaf:
>>> Hey there,
>>>
>>> Any news on this?
>>>
>>> My colleague Aarif made me aware of the discussion, because one of our
>>> students, Ashok, worked on this months ago. QIIME was requested by a
>>> project which never started and therefore the developed codes did not
>>> gain priority. Basically we faced the same issues, and at least the
>>> 'dead end' problem with the resulting tgz archives was solved. We are
>>> currently assembling the codes into a toolshed entry, but I need to ask
>>> Ashok tomorrow about the current progress.
>>>
>>> Is there still interest on it or is everything already on the move..?
>>>
>>> Cheers,
>>>
>>> Sebastian
>>>
>>>
>>> Björn Grüning schrieb:
>>>> Hi Yvan,
>>>>
>>>>> Hi Lance, Björn, Dan
>>>>>
>>>>> We are several scientists working on metagenomics in France and at a
>>>>> previous GUGGO  (western France User Group) meeting, we mention the
>>>>> fact to "benchmark" our Galaxy pipelines. One is using new Qiime
>>>>> tools integration, one Mothur and the last one, the recently
>>>>> developped FROGS (Find Rapidly OTU with Galaxy Solution) pipeline
>>>>> using Swarm. We already have had some exchanges with Björn concerning
>>>>> metagenomics flavours on Docker, and I think it will be particularly
>>>>> interesting to exchange one with each other. The Qiime integration
>>>>> was started in march by colleagues from Brest Ifremer centre. I was
>>>>> just trying to install and test FROGS from Toulouse and Jouy-en-Josas
>>>>> INRA labs. If they are not on the list, I can contact them.
>>>> Please do so. This is great to know. Please also consider to join our
>>>> hackathon:
>>>>
>>>> https://github.com/galaxyproject/tools-iuc/issues/299
>>>>
>>>> We are always open for new ideas. Would be great to get a nice suite
>>>> and
>>>> a few workflows for metagenomic in the near future! :)
>>>>
>>>> Thanks,
>>>> Bjoern
>>>>
>>>>> All the best,
>>>>>
>>>>> Yvan
>>>>>
>>>>> ----- Mail original -----
>>>>>> De: "Björn Grüning" <[hidden email]> À: "Lance Parsons"
>>>>>> <[hidden email]>, "Daniel Blankenberg" <[hidden email]> Cc:
>>>>>> "Galaxy Dev List" <[hidden email]> Envoyé:
>>>>>> Mercredi 7 Octobre 2015 14:25:38 Objet: Re: [galaxy-dev] QIIME
>>>>>> tools for Galaxy (WIP,    call for collaborators)
>>>>>>
>>>>>> Hi Lance,
>>>>>>
>>>>>> I will help if I can. I also think we should begin too write 2-3
>>>>>> nice tools in the beginning to test the handshake between them and
>>>>>> lay the ground work. This will also help to get familiar with the
>>>>>> tool.
>>>>>>
>>>>>> I still hope some one else from the community shows interest! :)
>>>>>>
>>>>>> Thanks, Bjoern
>>>>>>
>>>>>> Am 06.10.2015 um 20:33 schrieb Lance Parsons:
>>>>>>> I agree that it would be very nice to get the data flowing
>>>>>>> between each of the tools and to be able to mix/match with other
>>>>>>> tools. That is an area of Galaxy tool-dev that I'm less familiar
>>>>>>> with, so any help would be greatly appreciated.
>>>>>>>
>>>>>>> As for the manual massaging, I agree, however, at this point I'm
>>>>>>> really hoping that the changes between Qiime versions are
>>>>>>> manageable, esp. once we have the data types and basic framework
>>>>>>> of the tools down.
>>>>>>>
>>>>>>> Along that front, I would very much like to get something usable
>>>>>>> (at least for some specific workflows) done sooner rather than
>>>>>>> later, then be able to iterate, adding new tools, datatypes, etc.
>>>>>>> as we move forward. Does that sound reasonable? I love the idea
>>>>>>> of a Metagenomics toolfest
>>>>>>> (https://github.com/galaxyproject/tools-iuc/issues/299). One
>>>>>>> thing I would like to do soon then, would be to define some
>>>>>>> functionality for the first round. What would be very helpful
>>>>>>> would be if you, Bjoern, etc. could help make sure the first
>>>>>>> round lays the proper groundwork so I/we don't have to redo
>>>>>>> things for later iterations, but we can build on a solid
>>>>>>> foundation.
>>>>>>>
>>>>>>> Thanks so much for the input, help, etc. It is very much
>>>>>>> appreciated.
>>>>>>>
>>>>>>> - Lance
>>>>>>>
>>>>>>>> Daniel Blankenberg <mailto:[hidden email]> October 6, 2015 at
>>>>>>>> 10:21 AM Hi Lance,
>>>>>>>>
>>>>>>>> I looked at this a bit ago and had similar concerns,
>>>>>>>> particularly with the outputs and inputs not being
>>>>>>>> well-defined. In addition to the output tar ball —> local,
>>>>>>>> extract —> upload not being great, as you mention, the input
>>>>>>>> datatypes, etc, could use some work — in the very least, we
>>>>>>>> should definitely create a nice biom datatype and have some
>>>>>>>> converters available (import and export).
>>>>>>>>
>>>>>>>> Definitely worth spending some extra time to make sure that we
>>>>>>>> have the data flowing well between each of the different
>>>>>>>> parts/tools, and even better to make sure that its done in a
>>>>>>>> way that allows mixing and matching with other non-qiime
>>>>>>>> tools.
>>>>>>>>
>>>>>>>>
>>>>>>>> One thing that we want to avoid is large amounts of manual
>>>>>>>> massaging of the automatically generated xml; fixing things up
>>>>>>>> once might not be too bad, but having to do it with each new
>>>>>>>> tool version can be “frustrating". Although perhaps having a
>>>>>>>> good starting point and only needing to manually modify for any
>>>>>>>> updates could be good enough (I’m not very familiar with the
>>>>>>>> extent of typical changes between qiime versions to make a call
>>>>>>>> on how much changes).
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Dan
>>>>>>>>
>>>>>>>> (resending since I received a message bounce from list)
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Daniel Blankenberg <mailto:[hidden email]> October 6, 2015 at
>>>>>>>> 9:59 AM Hi Lance,
>>>>>>>>
>>>>>>>> I looked at this a bit ago and had similar concerns,
>>>>>>>> particularly with the outputs and inputs not being
>>>>>>>> well-defined. In addition to the output tar ball —> local,
>>>>>>>> extract —> upload not being great, as you mention, the input
>>>>>>>> datatypes, etc, could use some work — in the very least, we
>>>>>>>> should definitely create a nice biom datatype and have some
>>>>>>>> converters available (import and export).
>>>>>>>>
>>>>>>>> Definitely worth spending some extra time to make sure that we
>>>>>>>> have the data flowing well between each of the different
>>>>>>>> parts/tools, and even better to make sure that its done in a
>>>>>>>> way that allows mixing and matching with other non-qiime
>>>>>>>> tools.
>>>>>>>>
>>>>>>>>
>>>>>>>> One thing that we want to avoid is large amounts of manual
>>>>>>>> massaging of the automatically generated xml; fixing things up
>>>>>>>> once might not be too bad, but having to do it with each new
>>>>>>>> tool version can be “frustrating". Although perhaps having a
>>>>>>>> good starting point and only needing to manually modify for any
>>>>>>>> updates could be good enough (I’m not very familiar with the
>>>>>>>> extent of typical changes between qiime versions to make a call
>>>>>>>> on how much changes).
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Dan
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Lance Parsons <mailto:[hidden email]> October 5, 2015
>>>>>>>> at 5:26 PM I was recently asked if I could provide a QIIME
>>>>>>>> analysis pipeline for 16S data in Galaxy using tools in the
>>>>>>>> QIIME pipeline (http://qiime.org/).
>>>>>>>>
>>>>>>>> I did a bit of looking around for existing Galaxy wrappers and
>>>>>>>> found an application that generates the wrappers for QIIME
>>>>>>>> scripts for Galaxy (https://github.com/qiime/qiime-galaxy).
>>>>>>>> This is a very well written application that does a great job
>>>>>>>> of wrapping the QIIME scripts for Galaxy. However, there are a
>>>>>>>> few things about it that don't quite fit my needs.
>>>>>>>>
>>>>>>>> 1. The tools output tgz files of all of the output files. This
>>>>>>>> means that to execute a pipeline, the user would have to
>>>>>>>> download the tgz files, untar, and then upload whichever
>>>>>>>> file(s) are needed for the next step. 2. There is no toolshed
>>>>>>>> repository to install the dependencies for these tools, making
>>>>>>>> it a tricky for administrators to automate and also maintain
>>>>>>>> various versions of QIIME going forward. 3. There are no
>>>>>>>> toolshed versions of the tools themselves, which also makes
>>>>>>>> installation and integration a bit tricky and makes it hard to
>>>>>>>> me to create and manage updates, fixes, tweaks, etc. There are
>>>>>>>> also no tests, etc.
>>>>>>>>
>>>>>>>> For these reasons I decided to investigate the feasibility of
>>>>>>>> using the generated wrappers as a basis for a "toolshed"
>>>>>>>> version of QIIME. If anyone is interested in helping, or has
>>>>>>>> suggestions, or is working on something related, I'd be very
>>>>>>>> happy to collaborate.
>>>>>>>>
>>>>>>>> The repository for the WIP is at
>>>>>>>> https://github.com/lparsons/galaxy_tools/tree/qiime/tools/qiime1.9.0.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>> There is also a package on the testtoolshed as well as a first pass at
>>>>>>>> package_qiime_1_9_1
>>>>>>>> (https://github.com/lparsons/galaxy_tools/tree/qiime/packages/package_qiime_1_9_1).
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>> ___________________________________________________________
>>>>>>> Please keep all replies on the list by using "reply all" in your
>>>>>>> mail client.  To manage your subscriptions to this and other
>>>>>>> Galaxy lists, please use the interface at:
>>>>>>> https://lists.galaxyproject.org/
>>>>>>>
>>>>>>> To search Galaxy mailing lists use the unified search at:
>>>>>>> http://galaxyproject.org/search/mailinglists/
>>>>>>>
>>>>>> ___________________________________________________________ Please
>>>>>> keep all replies on the list by using "reply all" in your mail
>>>>>> client.  To manage your subscriptions to this and other Galaxy
>>>>>> lists, please use the interface at:
>>>>>> https://lists.galaxyproject.org/
>>>>>>
>>>>>> To search Galaxy mailing lists use the unified search at:
>>>>>> http://galaxyproject.org/search/mailinglists/
>>>> ___________________________________________________________
>>>> Please keep all replies on the list by using "reply all"
>>>> in your mail client.  To manage your subscriptions to this
>>>> and other Galaxy lists, please use the interface at:
>>>>     https://lists.galaxyproject.org/
>>>>
>>>> To search Galaxy mailing lists use the unified search at:
>>>>     http://galaxyproject.org/search/mailinglists/
>
___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/