How to install bowtie2 tool in galaxy

classic Classic list List threaded Threaded
12 messages Options
Reply | Threaded
Open this post in threaded view
|

How to install bowtie2 tool in galaxy

Ravi Alla
Hi guys,
I am new to galaxy and am in the process of setting it up on a cluster with some help. I am trying to install bowtie2 to my local galaxy through the toolshed. I have a few questions.

- Which bowtie2 should I install? When I search for bowtie2 a bunch of results come up?
- What are repository dependencies? Why would I need these if bowtie2 is already installed on my system?
- I have been installing bowtie2 by unchecking the repository dependencies and tool dependencies boxes. Is this correct?
- Finally when I install bowtie2 this way it creates 2 bowtie2_indices.loc files, one in galaxy/tool-data and the other in shed_tools/toolshed.g2.gx.psu.edu/dev/bowtie2, which one do I need to edit to point to my index files? No matter which one I change I can't seem to see the indices.

I hope someone on here can help me out.
Thanks a lot
Ravi.
___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
Reply | Threaded
Open this post in threaded view
|

Re: How to install bowtie2 tool in galaxy

Björn Grüning-3
Hi Ravi,

> Hi guys,
> I am new to galaxy and am in the process of setting it up on a cluster with some help. I am trying to install bowtie2 to my local galaxy through the toolshed. I have a few questions.
>
> - Which bowtie2 should I install? When I search for bowtie2 a bunch of results come up?

Only two, or? Please make sure you are using the main toolshed. bowtie2
and package_bowtie2_2_1_0 are connected together. One is the binary and
the other contains the wrapper. Use devteam respositories is always a
good choice.

> - What are repository dependencies? Why would I need these if bowtie2 is already installed on my system?

repository dependecies containing all dependencies of your wrappers,
that can be some binaries, but also R, Perl, python libraries that are
needed to execute your wrappers. We recommend you to use the
dependencies because then you have full control over all your used
versions (wrapper, dependencies, tool) and that enables reproducibility.
To put in other words: With the toolshed you can have different tool and
wrappers versions with different dependencies at the same time.

> - I have been installing bowtie2 by unchecking the repository dependencies and tool dependencies boxes. Is this correct?

It will work, if you have a system installed version. But you need to
take care about binary updates by your own. And reproducibility of your
results is not guaranteed.

> - Finally when I install bowtie2 this way it creates 2 bowtie2_indices.loc files, one in galaxy/tool-data and the other in shed_tools/toolshed.g2.gx.psu.edu/dev/bowtie2, which one do I need to edit to point to my index files? No matter which one I change I can't seem to see the indices.

That is strange. Are you sure you don't have a spelling mistake in it.
It should be the bowtie2_indices.los I think. I will CC Greg, he is one
of the Tool Shed Developers and should know more about it.
Please do not forget to restart your Galaxy instance once you have
update the *loc files.

> I hope someone on here can help me out.

Hope it helped a little bit,
Bjoern

> Thanks a lot
> Ravi.
> ___________________________________________________________
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>    http://lists.bx.psu.edu/
>
> To search Galaxy mailing lists use the unified search at:
>    http://galaxyproject.org/search/mailinglists/
>
___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
Reply | Threaded
Open this post in threaded view
|

Re: How to install bowtie2 tool in galaxy

Ravi Alla
Hi Bjoern,
Thank you for your clarifications. I have tried installing bowtie2 using both variations, once with both wrapper and dependencies and once with only wrapper (since I have dependencies installed on the system). In either case even after editing the .loc file I cannot see the indices in galaxy. I tried the same edits to .loc file with bwa and the indices show right up, but not with bowtie2 and even bowtie for that matter. I really don't know what to do about this.
Thanks
Ravi
On Mar 4, 2014, at 3:06 AM, Björn Grüning <[hidden email]> wrote:

> Hi Ravi,
>
>> Hi guys,
>> I am new to galaxy and am in the process of setting it up on a cluster with some help. I am trying to install bowtie2 to my local galaxy through the toolshed. I have a few questions.
>>
>> - Which bowtie2 should I install? When I search for bowtie2 a bunch of results come up?
>
> Only two, or? Please make sure you are using the main toolshed. bowtie2 and package_bowtie2_2_1_0 are connected together. One is the binary and the other contains the wrapper. Use devteam respositories is always a good choice.
>
>> - What are repository dependencies? Why would I need these if bowtie2 is already installed on my system?
>
> repository dependecies containing all dependencies of your wrappers, that can be some binaries, but also R, Perl, python libraries that are needed to execute your wrappers. We recommend you to use the dependencies because then you have full control over all your used versions (wrapper, dependencies, tool) and that enables reproducibility. To put in other words: With the toolshed you can have different tool and wrappers versions with different dependencies at the same time.
>
>> - I have been installing bowtie2 by unchecking the repository dependencies and tool dependencies boxes. Is this correct?
>
> It will work, if you have a system installed version. But you need to take care about binary updates by your own. And reproducibility of your results is not guaranteed.
>
>> - Finally when I install bowtie2 this way it creates 2 bowtie2_indices.loc files, one in galaxy/tool-data and the other in shed_tools/toolshed.g2.gx.psu.edu/dev/bowtie2, which one do I need to edit to point to my index files? No matter which one I change I can't seem to see the indices.
>
> That is strange. Are you sure you don't have a spelling mistake in it. It should be the bowtie2_indices.los I think. I will CC Greg, he is one of the Tool Shed Developers and should know more about it.
> Please do not forget to restart your Galaxy instance once you have update the *loc files.
>
>> I hope someone on here can help me out.
>
> Hope it helped a little bit,
> Bjoern
>
>> Thanks a lot
>> Ravi.
>> ___________________________________________________________
>> Please keep all replies on the list by using "reply all"
>> in your mail client.  To manage your subscriptions to this
>> and other Galaxy lists, please use the interface at:
>>   http://lists.bx.psu.edu/
>>
>> To search Galaxy mailing lists use the unified search at:
>>   http://galaxyproject.org/search/mailinglists/
>>


___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
Reply | Threaded
Open this post in threaded view
|

Re: How to install bowtie2 tool in galaxy

Björn Grüning-3
Hi Ravi,

can you attach the loc file, do you see anything in the Galaxy log files
about bowtie2, try grepping for "bowtie2".

Cheers,
Bjoern

Am 04.03.2014 18:07, schrieb Ravi Alla:

> Hi Bjoern,
> Thank you for your clarifications. I have tried installing bowtie2 using both variations, once with both wrapper and dependencies and once with only wrapper (since I have dependencies installed on the system). In either case even after editing the .loc file I cannot see the indices in galaxy. I tried the same edits to .loc file with bwa and the indices show right up, but not with bowtie2 and even bowtie for that matter. I really don't know what to do about this.
> Thanks
> Ravi
> On Mar 4, 2014, at 3:06 AM, Björn Grüning <[hidden email]> wrote:
>
>> Hi Ravi,
>>
>>> Hi guys,
>>> I am new to galaxy and am in the process of setting it up on a cluster with some help. I am trying to install bowtie2 to my local galaxy through the toolshed. I have a few questions.
>>>
>>> - Which bowtie2 should I install? When I search for bowtie2 a bunch of results come up?
>>
>> Only two, or? Please make sure you are using the main toolshed. bowtie2 and package_bowtie2_2_1_0 are connected together. One is the binary and the other contains the wrapper. Use devteam respositories is always a good choice.
>>
>>> - What are repository dependencies? Why would I need these if bowtie2 is already installed on my system?
>>
>> repository dependecies containing all dependencies of your wrappers, that can be some binaries, but also R, Perl, python libraries that are needed to execute your wrappers. We recommend you to use the dependencies because then you have full control over all your used versions (wrapper, dependencies, tool) and that enables reproducibility. To put in other words: With the toolshed you can have different tool and wrappers versions with different dependencies at the same time.
>>
>>> - I have been installing bowtie2 by unchecking the repository dependencies and tool dependencies boxes. Is this correct?
>>
>> It will work, if you have a system installed version. But you need to take care about binary updates by your own. And reproducibility of your results is not guaranteed.
>>
>>> - Finally when I install bowtie2 this way it creates 2 bowtie2_indices.loc files, one in galaxy/tool-data and the other in shed_tools/toolshed.g2.gx.psu.edu/dev/bowtie2, which one do I need to edit to point to my index files? No matter which one I change I can't seem to see the indices.
>>
>> That is strange. Are you sure you don't have a spelling mistake in it. It should be the bowtie2_indices.los I think. I will CC Greg, he is one of the Tool Shed Developers and should know more about it.
>> Please do not forget to restart your Galaxy instance once you have update the *loc files.
>>
>>> I hope someone on here can help me out.
>>
>> Hope it helped a little bit,
>> Bjoern
>>
>>> Thanks a lot
>>> Ravi.
>>> ___________________________________________________________
>>> Please keep all replies on the list by using "reply all"
>>> in your mail client.  To manage your subscriptions to this
>>> and other Galaxy lists, please use the interface at:
>>>    http://lists.bx.psu.edu/
>>>
>>> To search Galaxy mailing lists use the unified search at:
>>>    http://galaxyproject.org/search/mailinglists/
>>>
>
___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
Reply | Threaded
Open this post in threaded view
|

Re: How to install bowtie2 tool in galaxy

Ravi Alla
Hi Bjoern,
Please find the bowtie2 and bowtie .loc files. In the paster.log file I see these entries for bowtie

galaxy.tools.data DEBUG 2014-03-04 10:14:11,048 Loaded tool data table 'bowtie_indexes'
galaxy.tools.data DEBUG 2014-03-04 10:14:11,051 Loading another instance of data table 'bowtie_indexes', attempting to merge content.
galaxy.tools.parameters.dynamic_options WARNING 2014-03-04 10:14:33,335 Data table named 'bowtie2_indexes' is required by tool but not configured

And ls -l /global/referenceData/databases/bowtie2/hg19 is 
drwxr-xr-x 2 ralla cgrl      2048 Sep 10 11:14 .
drwxr-xr-x 3 ralla cgrl      2048 Sep 10 11:06 ..
-rw-rw-r-- 1 ralla cgrl 960018873 May  2  2012 hg19.1.bt2
-rw-rw-r-- 1 ralla cgrl 716863572 May  2  2012 hg19.2.bt2
-rw-rw-r-- 1 ralla cgrl      3833 May  2  2012 hg19.3.bt2
-rw-rw-r-- 1 ralla cgrl 716863565 May  2  2012 hg19.4.bt2
-rw-rw-r-- 1 ralla cgrl 960018873 May  2  2012 hg19.rev.1.bt2
-rw-rw-r-- 1 ralla cgrl 716863572 May  2  2012 hg19.rev.2.bt2

and ls -l /global/referenceData/databases/bowtie/hg19 is
drwxr-xr-x 2 ralla cgrl      2048 Mar  4 10:07 .
drwxr-xr-x 3 ralla cgrl      2048 Nov 30  2012 ..
-rw-rw-r-- 1 ralla cgrl 821725563 Nov 13  2009 hg19.1.ebwt
-rw-rw-r-- 1 ralla cgrl 357667968 Nov 13  2009 hg19.2.ebwt
-rw-rw-r-- 1 ralla cgrl      3284 Nov 13  2009 hg19.3.ebwt
-rw-rw-r-- 1 ralla cgrl 715335926 Nov 13  2009 hg19.4.ebwt
lrwxrwxrwx 1 ralla cgrl        45 Aug 21  2013 hg19.fa -> /global/referenceData/genomes/hs/hg19/hg19.fa
-rw-rw-r-- 1 ralla cgrl 821725563 Nov 13  2009 hg19.rev.1.ebwt
-rw-rw-r-- 1 ralla cgrl 357667968 Nov 13  2009 hg19.rev.2.ebwt

Thanks for looking into this.
Ravi.
On Mar 4, 2014, at 9:13 AM, Björn Grüning <[hidden email]> wrote:

Hi Ravi,

can you attach the loc file, do you see anything in the Galaxy log files about bowtie2, try grepping for "bowtie2".

Cheers,
Bjoern

Am 04.03.2014 18:07, schrieb Ravi Alla:
Hi Bjoern,
Thank you for your clarifications. I have tried installing bowtie2 using both variations, once with both wrapper and dependencies and once with only wrapper (since I have dependencies installed on the system). In either case even after editing the .loc file I cannot see the indices in galaxy. I tried the same edits to .loc file with bwa and the indices show right up, but not with bowtie2 and even bowtie for that matter. I really don't know what to do about this.
Thanks
Ravi
On Mar 4, 2014, at 3:06 AM, Björn Grüning <[hidden email]> wrote:

Hi Ravi,

Hi guys,
I am new to galaxy and am in the process of setting it up on a cluster with some help. I am trying to install bowtie2 to my local galaxy through the toolshed. I have a few questions.

- Which bowtie2 should I install? When I search for bowtie2 a bunch of results come up?

Only two, or? Please make sure you are using the main toolshed. bowtie2 and package_bowtie2_2_1_0 are connected together. One is the binary and the other contains the wrapper. Use devteam respositories is always a good choice.

- What are repository dependencies? Why would I need these if bowtie2 is already installed on my system?

repository dependecies containing all dependencies of your wrappers, that can be some binaries, but also R, Perl, python libraries that are needed to execute your wrappers. We recommend you to use the dependencies because then you have full control over all your used versions (wrapper, dependencies, tool) and that enables reproducibility. To put in other words: With the toolshed you can have different tool and wrappers versions with different dependencies at the same time.

- I have been installing bowtie2 by unchecking the repository dependencies and tool dependencies boxes. Is this correct?

It will work, if you have a system installed version. But you need to take care about binary updates by your own. And reproducibility of your results is not guaranteed.

- Finally when I install bowtie2 this way it creates 2 bowtie2_indices.loc files, one in galaxy/tool-data and the other in shed_tools/toolshed.g2.gx.psu.edu/dev/bowtie2, which one do I need to edit to point to my index files? No matter which one I change I can't seem to see the indices.

That is strange. Are you sure you don't have a spelling mistake in it. It should be the bowtie2_indices.los I think. I will CC Greg, he is one of the Tool Shed Developers and should know more about it.
Please do not forget to restart your Galaxy instance once you have update the *loc files.

I hope someone on here can help me out.

Hope it helped a little bit,
Bjoern

Thanks a lot
Ravi.
___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/




___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
Reply | Threaded
Open this post in threaded view
|

Re: How to install bowtie2 tool in galaxy

Björn Grüning-3
Hi,

can you check if the file tool_data_table_conf.xml contains an entry
about bowtie2_indexes ...  or the table shed_data_table_conf.xml ... in
one of them should be one entry with bowtie2_indexes.

That warning should not be there I think:

 > galaxy.tools.parameters.dynamic_options WARNING 2014-03-04
10:14:33,335 Data table named 'bowtie2_indexes' is required by tool but
not configured

Hope you can figure it out,
Bjoern

Am 04.03.2014 19:19, schrieb Ravi Alla:

> Hi Bjoern,
> Please find the bowtie2 and bowtie .loc files. In the paster.log file I see these entries for bowtie
>
> galaxy.tools.data DEBUG 2014-03-04 10:14:11,048 Loaded tool data table 'bowtie_indexes'
> galaxy.tools.data DEBUG 2014-03-04 10:14:11,051 Loading another instance of data table 'bowtie_indexes', attempting to merge content.
> galaxy.tools.parameters.dynamic_options WARNING 2014-03-04 10:14:33,335 Data table named 'bowtie2_indexes' is required by tool but not configured
>
> And ls -l /global/referenceData/databases/bowtie2/hg19 is
> drwxr-xr-x 2 ralla cgrl      2048 Sep 10 11:14 .
> drwxr-xr-x 3 ralla cgrl      2048 Sep 10 11:06 ..
> -rw-rw-r-- 1 ralla cgrl 960018873 May  2  2012 hg19.1.bt2
> -rw-rw-r-- 1 ralla cgrl 716863572 May  2  2012 hg19.2.bt2
> -rw-rw-r-- 1 ralla cgrl      3833 May  2  2012 hg19.3.bt2
> -rw-rw-r-- 1 ralla cgrl 716863565 May  2  2012 hg19.4.bt2
> -rw-rw-r-- 1 ralla cgrl 960018873 May  2  2012 hg19.rev.1.bt2
> -rw-rw-r-- 1 ralla cgrl 716863572 May  2  2012 hg19.rev.2.bt2
>
> and ls -l /global/referenceData/databases/bowtie/hg19 is
> drwxr-xr-x 2 ralla cgrl      2048 Mar  4 10:07 .
> drwxr-xr-x 3 ralla cgrl      2048 Nov 30  2012 ..
> -rw-rw-r-- 1 ralla cgrl 821725563 Nov 13  2009 hg19.1.ebwt
> -rw-rw-r-- 1 ralla cgrl 357667968 Nov 13  2009 hg19.2.ebwt
> -rw-rw-r-- 1 ralla cgrl      3284 Nov 13  2009 hg19.3.ebwt
> -rw-rw-r-- 1 ralla cgrl 715335926 Nov 13  2009 hg19.4.ebwt
> lrwxrwxrwx 1 ralla cgrl        45 Aug 21  2013 hg19.fa -> /global/referenceData/genomes/hs/hg19/hg19.fa
> -rw-rw-r-- 1 ralla cgrl 821725563 Nov 13  2009 hg19.rev.1.ebwt
> -rw-rw-r-- 1 ralla cgrl 357667968 Nov 13  2009 hg19.rev.2.ebwt
>
> Thanks for looking into this.
> Ravi.
> On Mar 4, 2014, at 9:13 AM, Björn Grüning <[hidden email]> wrote:
>
>> Hi Ravi,
>>
>> can you attach the loc file, do you see anything in the Galaxy log files about bowtie2, try grepping for "bowtie2".
>>
>> Cheers,
>> Bjoern
>>
>> Am 04.03.2014 18:07, schrieb Ravi Alla:
>>> Hi Bjoern,
>>> Thank you for your clarifications. I have tried installing bowtie2 using both variations, once with both wrapper and dependencies and once with only wrapper (since I have dependencies installed on the system). In either case even after editing the .loc file I cannot see the indices in galaxy. I tried the same edits to .loc file with bwa and the indices show right up, but not with bowtie2 and even bowtie for that matter. I really don't know what to do about this.
>>> Thanks
>>> Ravi
>>> On Mar 4, 2014, at 3:06 AM, Björn Grüning <[hidden email]> wrote:
>>>
>>>> Hi Ravi,
>>>>
>>>>> Hi guys,
>>>>> I am new to galaxy and am in the process of setting it up on a cluster with some help. I am trying to install bowtie2 to my local galaxy through the toolshed. I have a few questions.
>>>>>
>>>>> - Which bowtie2 should I install? When I search for bowtie2 a bunch of results come up?
>>>>
>>>> Only two, or? Please make sure you are using the main toolshed. bowtie2 and package_bowtie2_2_1_0 are connected together. One is the binary and the other contains the wrapper. Use devteam respositories is always a good choice.
>>>>
>>>>> - What are repository dependencies? Why would I need these if bowtie2 is already installed on my system?
>>>>
>>>> repository dependecies containing all dependencies of your wrappers, that can be some binaries, but also R, Perl, python libraries that are needed to execute your wrappers. We recommend you to use the dependencies because then you have full control over all your used versions (wrapper, dependencies, tool) and that enables reproducibility. To put in other words: With the toolshed you can have different tool and wrappers versions with different dependencies at the same time.
>>>>
>>>>> - I have been installing bowtie2 by unchecking the repository dependencies and tool dependencies boxes. Is this correct?
>>>>
>>>> It will work, if you have a system installed version. But you need to take care about binary updates by your own. And reproducibility of your results is not guaranteed.
>>>>
>>>>> - Finally when I install bowtie2 this way it creates 2 bowtie2_indices.loc files, one in galaxy/tool-data and the other in shed_tools/toolshed.g2.gx.psu.edu/dev/bowtie2, which one do I need to edit to point to my index files? No matter which one I change I can't seem to see the indices.
>>>>
>>>> That is strange. Are you sure you don't have a spelling mistake in it. It should be the bowtie2_indices.los I think. I will CC Greg, he is one of the Tool Shed Developers and should know more about it.
>>>> Please do not forget to restart your Galaxy instance once you have update the *loc files.
>>>>
>>>>> I hope someone on here can help me out.
>>>>
>>>> Hope it helped a little bit,
>>>> Bjoern
>>>>
>>>>> Thanks a lot
>>>>> Ravi.
>>>>> ___________________________________________________________
>>>>> Please keep all replies on the list by using "reply all"
>>>>> in your mail client.  To manage your subscriptions to this
>>>>> and other Galaxy lists, please use the interface at:
>>>>>    http://lists.bx.psu.edu/
>>>>>
>>>>> To search Galaxy mailing lists use the unified search at:
>>>>>    http://galaxyproject.org/search/mailinglists/
>>>>>
>>>
>
>
___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
Reply | Threaded
Open this post in threaded view
|

Re: How to install bowtie2 tool in galaxy

Ravi Alla
Bjoern,
This is getting frustrating.
There are three places bowtie2_indices.loc file is expected. I really don't know which one is the one I should modify.

galaxy-dist/tool-data/bowtie2_indices.loc

What is the difference between these 3 files?

I changed the 2nd file to include the path to the indices and this shows up as a valid preloaded index for tophat2 but does not for bowtie2. I also changed the normal bowtie_indices.loc file at the second location and this works and I can see the bowtie1 indices for that bowtie tool. Currently only the bowtie2 indices are acting up.

I wish this was easier.
Thanks
Ravi.
On Mar 4, 2014, at 11:28 AM, Björn Grüning <[hidden email]> wrote:

Hi,

can you check if the file tool_data_table_conf.xml contains an entry about bowtie2_indexes ...  or the table shed_data_table_conf.xml ... in one of them should be one entry with bowtie2_indexes.

That warning should not be there I think:

> galaxy.tools.parameters.dynamic_options WARNING 2014-03-04 10:14:33,335 Data table named 'bowtie2_indexes' is required by tool but not configured

Hope you can figure it out,
Bjoern

Am 04.03.2014 19:19, schrieb Ravi Alla:
Hi Bjoern,
Please find the bowtie2 and bowtie .loc files. In the paster.log file I see these entries for bowtie

galaxy.tools.data DEBUG 2014-03-04 10:14:11,048 Loaded tool data table 'bowtie_indexes'
galaxy.tools.data DEBUG 2014-03-04 10:14:11,051 Loading another instance of data table 'bowtie_indexes', attempting to merge content.
galaxy.tools.parameters.dynamic_options WARNING 2014-03-04 10:14:33,335 Data table named 'bowtie2_indexes' is required by tool but not configured

And ls -l /global/referenceData/databases/bowtie2/hg19 is
drwxr-xr-x 2 ralla cgrl      2048 Sep 10 11:14 .
drwxr-xr-x 3 ralla cgrl      2048 Sep 10 11:06 ..
-rw-rw-r-- 1 ralla cgrl 960018873 May  2  2012 hg19.1.bt2
-rw-rw-r-- 1 ralla cgrl 716863572 May  2  2012 hg19.2.bt2
-rw-rw-r-- 1 ralla cgrl      3833 May  2  2012 hg19.3.bt2
-rw-rw-r-- 1 ralla cgrl 716863565 May  2  2012 hg19.4.bt2
-rw-rw-r-- 1 ralla cgrl 960018873 May  2  2012 hg19.rev.1.bt2
-rw-rw-r-- 1 ralla cgrl 716863572 May  2  2012 hg19.rev.2.bt2

and ls -l /global/referenceData/databases/bowtie/hg19 is
drwxr-xr-x 2 ralla cgrl      2048 Mar  4 10:07 .
drwxr-xr-x 3 ralla cgrl      2048 Nov 30  2012 ..
-rw-rw-r-- 1 ralla cgrl 821725563 Nov 13  2009 hg19.1.ebwt
-rw-rw-r-- 1 ralla cgrl 357667968 Nov 13  2009 hg19.2.ebwt
-rw-rw-r-- 1 ralla cgrl      3284 Nov 13  2009 hg19.3.ebwt
-rw-rw-r-- 1 ralla cgrl 715335926 Nov 13  2009 hg19.4.ebwt
lrwxrwxrwx 1 ralla cgrl        45 Aug 21  2013 hg19.fa -> /global/referenceData/genomes/hs/hg19/hg19.fa
-rw-rw-r-- 1 ralla cgrl 821725563 Nov 13  2009 hg19.rev.1.ebwt
-rw-rw-r-- 1 ralla cgrl 357667968 Nov 13  2009 hg19.rev.2.ebwt

Thanks for looking into this.
Ravi.
On Mar 4, 2014, at 9:13 AM, Björn Grüning <[hidden email]> wrote:

Hi Ravi,

can you attach the loc file, do you see anything in the Galaxy log files about bowtie2, try grepping for "bowtie2".

Cheers,
Bjoern

Am 04.03.2014 18:07, schrieb Ravi Alla:
Hi Bjoern,
Thank you for your clarifications. I have tried installing bowtie2 using both variations, once with both wrapper and dependencies and once with only wrapper (since I have dependencies installed on the system). In either case even after editing the .loc file I cannot see the indices in galaxy. I tried the same edits to .loc file with bwa and the indices show right up, but not with bowtie2 and even bowtie for that matter. I really don't know what to do about this.
Thanks
Ravi
On Mar 4, 2014, at 3:06 AM, Björn Grüning <[hidden email]> wrote:

Hi Ravi,

Hi guys,
I am new to galaxy and am in the process of setting it up on a cluster with some help. I am trying to install bowtie2 to my local galaxy through the toolshed. I have a few questions.

- Which bowtie2 should I install? When I search for bowtie2 a bunch of results come up?

Only two, or? Please make sure you are using the main toolshed. bowtie2 and package_bowtie2_2_1_0 are connected together. One is the binary and the other contains the wrapper. Use devteam respositories is always a good choice.

- What are repository dependencies? Why would I need these if bowtie2 is already installed on my system?

repository dependecies containing all dependencies of your wrappers, that can be some binaries, but also R, Perl, python libraries that are needed to execute your wrappers. We recommend you to use the dependencies because then you have full control over all your used versions (wrapper, dependencies, tool) and that enables reproducibility. To put in other words: With the toolshed you can have different tool and wrappers versions with different dependencies at the same time.

- I have been installing bowtie2 by unchecking the repository dependencies and tool dependencies boxes. Is this correct?

It will work, if you have a system installed version. But you need to take care about binary updates by your own. And reproducibility of your results is not guaranteed.

- Finally when I install bowtie2 this way it creates 2 bowtie2_indices.loc files, one in galaxy/tool-data and the other in shed_tools/toolshed.g2.gx.psu.edu/dev/bowtie2, which one do I need to edit to point to my index files? No matter which one I change I can't seem to see the indices.

That is strange. Are you sure you don't have a spelling mistake in it. It should be the bowtie2_indices.los I think. I will CC Greg, he is one of the Tool Shed Developers and should know more about it.
Please do not forget to restart your Galaxy instance once you have update the *loc files.

I hope someone on here can help me out.

Hope it helped a little bit,
Bjoern

Thanks a lot
Ravi.
___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/






___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
Reply | Threaded
Open this post in threaded view
|

Re: How to install bowtie2 tool in galaxy

Daniel Blankenberg
Hi Ravi,

I checked and the bowtie2 tool repository had a misconfigured data table name in its tool_data_table_conf.xml.sample file that was erroneously creating a tophat2_indexes data table, whereas the tool itself uses bowtie2_indexes. 

Can you try to uninstall and reinstall bowtie2 from the toolshed and see if that will fix your problem?

More information on ways to debug this are below (wrote email before checking the repository):

You’ll need to check that the various location files are being loaded into the proper data tables, see: https://wiki.galaxyproject.org/Admin/Tools/Data%20Tables. Galaxy has 2 files that specify the link between a location file and a data table: tool_data_table_conf.xml and shed_tool_data_table_conf.xml, for distribution/locally defined data tables and shed installed data tables, respectively — these are merged together during run time for each named data table from the various specified location files.

Can you check these files to confirm that you have a data table named ‘bowtie2_indexes’ defined and that one of the files referenced from there has your additions?


Thanks for using Galaxy,

Dan


On Mar 4, 2014, at 3:21 PM, Ravi Alla <[hidden email]> wrote:

Bjoern,
This is getting frustrating.
There are three places bowtie2_indices.loc file is expected. I really don't know which one is the one I should modify.

galaxy-dist/tool-data/bowtie2_indices.loc

What is the difference between these 3 files?

I changed the 2nd file to include the path to the indices and this shows up as a valid preloaded index for tophat2 but does not for bowtie2. I also changed the normal bowtie_indices.loc file at the second location and this works and I can see the bowtie1 indices for that bowtie tool. Currently only the bowtie2 indices are acting up.

I wish this was easier.
Thanks
Ravi.
On Mar 4, 2014, at 11:28 AM, Björn Grüning <[hidden email]> wrote:

Hi,

can you check if the file tool_data_table_conf.xml contains an entry about bowtie2_indexes ...  or the table shed_data_table_conf.xml ... in one of them should be one entry with bowtie2_indexes.

That warning should not be there I think:

> galaxy.tools.parameters.dynamic_options WARNING 2014-03-04 10:14:33,335 Data table named 'bowtie2_indexes' is required by tool but not configured

Hope you can figure it out,
Bjoern

Am 04.03.2014 19:19, schrieb Ravi Alla:
Hi Bjoern,
Please find the bowtie2 and bowtie .loc files. In the paster.log file I see these entries for bowtie

galaxy.tools.data DEBUG 2014-03-04 10:14:11,048 Loaded tool data table 'bowtie_indexes'
galaxy.tools.data DEBUG 2014-03-04 10:14:11,051 Loading another instance of data table 'bowtie_indexes', attempting to merge content.
galaxy.tools.parameters.dynamic_options WARNING 2014-03-04 10:14:33,335 Data table named 'bowtie2_indexes' is required by tool but not configured

And ls -l /global/referenceData/databases/bowtie2/hg19 is
drwxr-xr-x 2 ralla cgrl      2048 Sep 10 11:14 .
drwxr-xr-x 3 ralla cgrl      2048 Sep 10 11:06 ..
-rw-rw-r-- 1 ralla cgrl 960018873 May  2  2012 hg19.1.bt2
-rw-rw-r-- 1 ralla cgrl 716863572 May  2  2012 hg19.2.bt2
-rw-rw-r-- 1 ralla cgrl      3833 May  2  2012 hg19.3.bt2
-rw-rw-r-- 1 ralla cgrl 716863565 May  2  2012 hg19.4.bt2
-rw-rw-r-- 1 ralla cgrl 960018873 May  2  2012 hg19.rev.1.bt2
-rw-rw-r-- 1 ralla cgrl 716863572 May  2  2012 hg19.rev.2.bt2

and ls -l /global/referenceData/databases/bowtie/hg19 is
drwxr-xr-x 2 ralla cgrl      2048 Mar  4 10:07 .
drwxr-xr-x 3 ralla cgrl      2048 Nov 30  2012 ..
-rw-rw-r-- 1 ralla cgrl 821725563 Nov 13  2009 hg19.1.ebwt
-rw-rw-r-- 1 ralla cgrl 357667968 Nov 13  2009 hg19.2.ebwt
-rw-rw-r-- 1 ralla cgrl      3284 Nov 13  2009 hg19.3.ebwt
-rw-rw-r-- 1 ralla cgrl 715335926 Nov 13  2009 hg19.4.ebwt
lrwxrwxrwx 1 ralla cgrl        45 Aug 21  2013 hg19.fa -> /global/referenceData/genomes/hs/hg19/hg19.fa
-rw-rw-r-- 1 ralla cgrl 821725563 Nov 13  2009 hg19.rev.1.ebwt
-rw-rw-r-- 1 ralla cgrl 357667968 Nov 13  2009 hg19.rev.2.ebwt

Thanks for looking into this.
Ravi.
On Mar 4, 2014, at 9:13 AM, Björn Grüning <[hidden email]> wrote:

Hi Ravi,

can you attach the loc file, do you see anything in the Galaxy log files about bowtie2, try grepping for "bowtie2".

Cheers,
Bjoern

Am 04.03.2014 18:07, schrieb Ravi Alla:
Hi Bjoern,
Thank you for your clarifications. I have tried installing bowtie2 using both variations, once with both wrapper and dependencies and once with only wrapper (since I have dependencies installed on the system). In either case even after editing the .loc file I cannot see the indices in galaxy. I tried the same edits to .loc file with bwa and the indices show right up, but not with bowtie2 and even bowtie for that matter. I really don't know what to do about this.
Thanks
Ravi
On Mar 4, 2014, at 3:06 AM, Björn Grüning <[hidden email]> wrote:

Hi Ravi,

Hi guys,
I am new to galaxy and am in the process of setting it up on a cluster with some help. I am trying to install bowtie2 to my local galaxy through the toolshed. I have a few questions.

- Which bowtie2 should I install? When I search for bowtie2 a bunch of results come up?

Only two, or? Please make sure you are using the main toolshed. bowtie2 and package_bowtie2_2_1_0 are connected together. One is the binary and the other contains the wrapper. Use devteam respositories is always a good choice.

- What are repository dependencies? Why would I need these if bowtie2 is already installed on my system?

repository dependecies containing all dependencies of your wrappers, that can be some binaries, but also R, Perl, python libraries that are needed to execute your wrappers. We recommend you to use the dependencies because then you have full control over all your used versions (wrapper, dependencies, tool) and that enables reproducibility. To put in other words: With the toolshed you can have different tool and wrappers versions with different dependencies at the same time.

- I have been installing bowtie2 by unchecking the repository dependencies and tool dependencies boxes. Is this correct?

It will work, if you have a system installed version. But you need to take care about binary updates by your own. And reproducibility of your results is not guaranteed.

- Finally when I install bowtie2 this way it creates 2 bowtie2_indices.loc files, one in galaxy/tool-data and the other in shed_tools/toolshed.g2.gx.psu.edu/dev/bowtie2, which one do I need to edit to point to my index files? No matter which one I change I can't seem to see the indices.

That is strange. Are you sure you don't have a spelling mistake in it. It should be the bowtie2_indices.los I think. I will CC Greg, he is one of the Tool Shed Developers and should know more about it.
Please do not forget to restart your Galaxy instance once you have update the *loc files.

I hope someone on here can help me out.

Hope it helped a little bit,
Bjoern

Thanks a lot
Ravi.
___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/





___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
 http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
 http://galaxyproject.org/search/mailinglists/


___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
Reply | Threaded
Open this post in threaded view
|

Re: How to install bowtie2 tool in galaxy

Jennifer Jackson
In reply to this post by Ravi Alla
Hi Ravi,

The directory structure for the installation of ToolShed tools changed, which is why you have three directories. You perhaps had bowtie2 installed once before, then reinstalled (without completely removing the older version and associated data)? Or updated without resetting the metadata? In either case, the ../shed_tools directory is the new one. Having this as the path in your .xml configuration files (as Bjoern suggested earlier) and moving all contents & data to be under the same location will be the simplest global solution ongoing. Links near the end of my reply can help explain how-to.

For the specific reason why bowtie2 indices are not working: I noticed that the reference genome ".fa" file is not linked from the directory containing the indexes. This is required. Adding it in, the same way that you did for the bowtie2 indexes, into this dir: "/global/referenceData/databases/bowtie2/hg19" will probably solve that part of the problem. I didn't see this posted - but I apologize if I am duplicating advice already given.

I also tend to advise keeping all data under the same master "data" directory - all indexes and sequence data - as symbolic links to additional file system paths that are unknown to the 'galaxy user' cause a different set of problems. However, that said, this doesn't seem to be an issue in your specific case: if the bowtie2 indexes are functioning correctly - then environment is set up so that the other dir hierarchy where the .fa files are kept must included in the 'galaxy' user's ENV. Symbolic links that go outside of the local dir structure are known to cause problems unless the ENV config is carefully set up, and to my knowledge are best avoided entirely for certain uses such as "upload by file path" into libraries.

For reference:

NGS data set-up is described in this wiki - including expected content for each type of index:
https://wiki.galaxyproject.org/Admin/NGS%20Local%20Setup

For examples, you can rsync a genome or two and examine the contents. Or, our /location dir and have a look at the .loc files.
https://wiki.galaxyproject.org/Admin/DataIntegration

Tool Shed help (very detailed):
https://wiki.galaxyproject.org/Tool%20Shed
In particular, if you had previously installed repositories (this is not clear, just suspected from the duplications), updating the Metadata with certain distribution updates can be very important. This has been necessary for the last few releases to update to changes. The News Brief noted this, and included a link to this wiki page. Also see the "Related Pages" lower down on the wiki.
https://wiki.galaxyproject.org/ResettingMetadataForInstalledRepositories
This may also be useful:
https://wiki.galaxyproject.org/RepairingInstalledRepositories

Hopefully you have sorted most of this out by now, or this helps!

Jen
Galaxy team


On 3/4/14 12:21 PM, Ravi Alla wrote:
Bjoern,
This is getting frustrating.
There are three places bowtie2_indices.loc file is expected. I really don't know which one is the one I should modify.

galaxy-dist/tool-data/bowtie2_indices.loc

What is the difference between these 3 files?

I changed the 2nd file to include the path to the indices and this shows up as a valid preloaded index for tophat2 but does not for bowtie2. I also changed the normal bowtie_indices.loc file at the second location and this works and I can see the bowtie1 indices for that bowtie tool. Currently only the bowtie2 indices are acting up.

I wish this was easier.
Thanks
Ravi.
On Mar 4, 2014, at 11:28 AM, Björn Grüning <[hidden email]> wrote:

Hi,

can you check if the file tool_data_table_conf.xml contains an entry about bowtie2_indexes ...  or the table shed_data_table_conf.xml ... in one of them should be one entry with bowtie2_indexes.

That warning should not be there I think:

> galaxy.tools.parameters.dynamic_options WARNING 2014-03-04 10:14:33,335 Data table named 'bowtie2_indexes' is required by tool but not configured

Hope you can figure it out,
Bjoern

Am 04.03.2014 19:19, schrieb Ravi Alla:
Hi Bjoern,
Please find the bowtie2 and bowtie .loc files. In the paster.log file I see these entries for bowtie

galaxy.tools.data DEBUG 2014-03-04 10:14:11,048 Loaded tool data table 'bowtie_indexes'
galaxy.tools.data DEBUG 2014-03-04 10:14:11,051 Loading another instance of data table 'bowtie_indexes', attempting to merge content.
galaxy.tools.parameters.dynamic_options WARNING 2014-03-04 10:14:33,335 Data table named 'bowtie2_indexes' is required by tool but not configured

And ls -l /global/referenceData/databases/bowtie2/hg19 is
drwxr-xr-x 2 ralla cgrl      2048 Sep 10 11:14 .
drwxr-xr-x 3 ralla cgrl      2048 Sep 10 11:06 ..
-rw-rw-r-- 1 ralla cgrl 960018873 May  2  2012 hg19.1.bt2
-rw-rw-r-- 1 ralla cgrl 716863572 May  2  2012 hg19.2.bt2
-rw-rw-r-- 1 ralla cgrl      3833 May  2  2012 hg19.3.bt2
-rw-rw-r-- 1 ralla cgrl 716863565 May  2  2012 hg19.4.bt2
-rw-rw-r-- 1 ralla cgrl 960018873 May  2  2012 hg19.rev.1.bt2
-rw-rw-r-- 1 ralla cgrl 716863572 May  2  2012 hg19.rev.2.bt2

and ls -l /global/referenceData/databases/bowtie/hg19 is
drwxr-xr-x 2 ralla cgrl      2048 Mar  4 10:07 .
drwxr-xr-x 3 ralla cgrl      2048 Nov 30  2012 ..
-rw-rw-r-- 1 ralla cgrl 821725563 Nov 13  2009 hg19.1.ebwt
-rw-rw-r-- 1 ralla cgrl 357667968 Nov 13  2009 hg19.2.ebwt
-rw-rw-r-- 1 ralla cgrl      3284 Nov 13  2009 hg19.3.ebwt
-rw-rw-r-- 1 ralla cgrl 715335926 Nov 13  2009 hg19.4.ebwt
lrwxrwxrwx 1 ralla cgrl        45 Aug 21  2013 hg19.fa -> /global/referenceData/genomes/hs/hg19/hg19.fa
-rw-rw-r-- 1 ralla cgrl 821725563 Nov 13  2009 hg19.rev.1.ebwt
-rw-rw-r-- 1 ralla cgrl 357667968 Nov 13  2009 hg19.rev.2.ebwt

Thanks for looking into this.
Ravi.
On Mar 4, 2014, at 9:13 AM, Björn Grüning <[hidden email]> wrote:

Hi Ravi,

can you attach the loc file, do you see anything in the Galaxy log files about bowtie2, try grepping for "bowtie2".

Cheers,
Bjoern

Am 04.03.2014 18:07, schrieb Ravi Alla:
Hi Bjoern,
Thank you for your clarifications. I have tried installing bowtie2 using both variations, once with both wrapper and dependencies and once with only wrapper (since I have dependencies installed on the system). In either case even after editing the .loc file I cannot see the indices in galaxy. I tried the same edits to .loc file with bwa and the indices show right up, but not with bowtie2 and even bowtie for that matter. I really don't know what to do about this.
Thanks
Ravi
On Mar 4, 2014, at 3:06 AM, Björn Grüning <[hidden email]> wrote:

Hi Ravi,

Hi guys,
I am new to galaxy and am in the process of setting it up on a cluster with some help. I am trying to install bowtie2 to my local galaxy through the toolshed. I have a few questions.

- Which bowtie2 should I install? When I search for bowtie2 a bunch of results come up?

Only two, or? Please make sure you are using the main toolshed. bowtie2 and package_bowtie2_2_1_0 are connected together. One is the binary and the other contains the wrapper. Use devteam respositories is always a good choice.

- What are repository dependencies? Why would I need these if bowtie2 is already installed on my system?

repository dependecies containing all dependencies of your wrappers, that can be some binaries, but also R, Perl, python libraries that are needed to execute your wrappers. We recommend you to use the dependencies because then you have full control over all your used versions (wrapper, dependencies, tool) and that enables reproducibility. To put in other words: With the toolshed you can have different tool and wrappers versions with different dependencies at the same time.

- I have been installing bowtie2 by unchecking the repository dependencies and tool dependencies boxes. Is this correct?

It will work, if you have a system installed version. But you need to take care about binary updates by your own. And reproducibility of your results is not guaranteed.

- Finally when I install bowtie2 this way it creates 2 bowtie2_indices.loc files, one in galaxy/tool-data and the other in shed_tools/toolshed.g2.gx.psu.edu/dev/bowtie2, which one do I need to edit to point to my index files? No matter which one I change I can't seem to see the indices.

That is strange. Are you sure you don't have a spelling mistake in it. It should be the bowtie2_indices.los I think. I will CC Greg, he is one of the Tool Shed Developers and should know more about it.
Please do not forget to restart your Galaxy instance once you have update the *loc files.

I hope someone on here can help me out.

Hope it helped a little bit,
Bjoern

Thanks a lot
Ravi.
___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/







___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

-- 
Jennifer Hillman-Jackson
http://galaxyproject.org

___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
Jennifer Hillman-Jackson
Galaxy Support and Training
http://galaxyproject.org
Reply | Threaded
Open this post in threaded view
|

Re: How to install bowtie2 tool in galaxy

Ravi Alla
Hi Jennifer,
Thank you for this information. I was able to troubleshoot the bowtie2_indices. Like Bjoern said the bowtie2 tool needed to be reinstalled because the tool_table does not load the correct indices.

I am still trying to wrap my head around the different toolsheds.

The main tool shed resides in the galaxy-dist folder and uses tool_conf.xml to load tools into the side bar and tool_data_table_conf.xml to load indices (and other data) for the default tools that come with galaxy.

The shed tools reside in the ../shed_tools/ directory and use shed_tools_conf.xml to load tools into the side bar and the shed_tools_data_table_conf.xml to load indices.

The shed_tool xml is setup in the universe.ini file.

The .loc files for default main tools reside in galaxy-dist/tool-data and the .loc files for the shed_tools reside in ../shed_tools/......./<tool-name>/<tool_id>/<tool-name>/

And when I uninstall tools and reinstall I would have to update the metadata for that tool.

If ../shed_tools dir is not present then shed-tools get installed under galaxy-dist/tool-data/toolshed.g2.bx.psu.edu/


And it is always better to install tools as wrappers + dependencies when possible.

Am I on the right track with this tool organization?
Thanks
Ravi



On Mar 5, 2014, at 11:06 AM, Jennifer Jackson <[hidden email]> wrote:

Hi Ravi,

The directory structure for the installation of ToolShed tools changed, which is why you have three directories. You perhaps had bowtie2 installed once before, then reinstalled (without completely removing the older version and associated data)? Or updated without resetting the metadata? In either case, the ../shed_tools directory is the new one. Having this as the path in your .xml configuration files (as Bjoern suggested earlier) and moving all contents & data to be under the same location will be the simplest global solution ongoing. Links near the end of my reply can help explain how-to.

For the specific reason why bowtie2 indices are not working: I noticed that the reference genome ".fa" file is not linked from the directory containing the indexes. This is required. Adding it in, the same way that you did for the bowtie2 indexes, into this dir: "/global/referenceData/databases/bowtie2/hg19" will probably solve that part of the problem. I didn't see this posted - but I apologize if I am duplicating advice already given.

I also tend to advise keeping all data under the same master "data" directory - all indexes and sequence data - as symbolic links to additional file system paths that are unknown to the 'galaxy user' cause a different set of problems. However, that said, this doesn't seem to be an issue in your specific case: if the bowtie2 indexes are functioning correctly - then environment is set up so that the other dir hierarchy where the .fa files are kept must included in the 'galaxy' user's ENV. Symbolic links that go outside of the local dir structure are known to cause problems unless the ENV config is carefully set up, and to my knowledge are best avoided entirely for certain uses such as "upload by file path" into libraries.

For reference:

NGS data set-up is described in this wiki - including expected content for each type of index:
https://wiki.galaxyproject.org/Admin/NGS%20Local%20Setup

For examples, you can rsync a genome or two and examine the contents. Or, our /location dir and have a look at the .loc files.
https://wiki.galaxyproject.org/Admin/DataIntegration

Tool Shed help (very detailed):
https://wiki.galaxyproject.org/Tool%20Shed
In particular, if you had previously installed repositories (this is not clear, just suspected from the duplications), updating the Metadata with certain distribution updates can be very important. This has been necessary for the last few releases to update to changes. The News Brief noted this, and included a link to this wiki page. Also see the "Related Pages" lower down on the wiki.
https://wiki.galaxyproject.org/ResettingMetadataForInstalledRepositories
This may also be useful:
https://wiki.galaxyproject.org/RepairingInstalledRepositories

Hopefully you have sorted most of this out by now, or this helps!

Jen
Galaxy team


On 3/4/14 12:21 PM, Ravi Alla wrote:
Bjoern,
This is getting frustrating.
There are three places bowtie2_indices.loc file is expected. I really don't know which one is the one I should modify.

galaxy-dist/tool-data/bowtie2_indices.loc

What is the difference between these 3 files?

I changed the 2nd file to include the path to the indices and this shows up as a valid preloaded index for tophat2 but does not for bowtie2. I also changed the normal bowtie_indices.loc file at the second location and this works and I can see the bowtie1 indices for that bowtie tool. Currently only the bowtie2 indices are acting up.

I wish this was easier.
Thanks
Ravi.
On Mar 4, 2014, at 11:28 AM, Björn Grüning <[hidden email]> wrote:

Hi,

can you check if the file tool_data_table_conf.xml contains an entry about bowtie2_indexes ...  or the table shed_data_table_conf.xml ... in one of them should be one entry with bowtie2_indexes.

That warning should not be there I think:

> galaxy.tools.parameters.dynamic_options WARNING 2014-03-04 10:14:33,335 Data table named 'bowtie2_indexes' is required by tool but not configured

Hope you can figure it out,
Bjoern

Am 04.03.2014 19:19, schrieb Ravi Alla:
Hi Bjoern,
Please find the bowtie2 and bowtie .loc files. In the paster.log file I see these entries for bowtie

galaxy.tools.data DEBUG 2014-03-04 10:14:11,048 Loaded tool data table 'bowtie_indexes'
galaxy.tools.data DEBUG 2014-03-04 10:14:11,051 Loading another instance of data table 'bowtie_indexes', attempting to merge content.
galaxy.tools.parameters.dynamic_options WARNING 2014-03-04 10:14:33,335 Data table named 'bowtie2_indexes' is required by tool but not configured

And ls -l /global/referenceData/databases/bowtie2/hg19 is
drwxr-xr-x 2 ralla cgrl      2048 Sep 10 11:14 .
drwxr-xr-x 3 ralla cgrl      2048 Sep 10 11:06 ..
-rw-rw-r-- 1 ralla cgrl 960018873 May  2  2012 hg19.1.bt2
-rw-rw-r-- 1 ralla cgrl 716863572 May  2  2012 hg19.2.bt2
-rw-rw-r-- 1 ralla cgrl      3833 May  2  2012 hg19.3.bt2
-rw-rw-r-- 1 ralla cgrl 716863565 May  2  2012 hg19.4.bt2
-rw-rw-r-- 1 ralla cgrl 960018873 May  2  2012 hg19.rev.1.bt2
-rw-rw-r-- 1 ralla cgrl 716863572 May  2  2012 hg19.rev.2.bt2

and ls -l /global/referenceData/databases/bowtie/hg19 is
drwxr-xr-x 2 ralla cgrl      2048 Mar  4 10:07 .
drwxr-xr-x 3 ralla cgrl      2048 Nov 30  2012 ..
-rw-rw-r-- 1 ralla cgrl 821725563 Nov 13  2009 hg19.1.ebwt
-rw-rw-r-- 1 ralla cgrl 357667968 Nov 13  2009 hg19.2.ebwt
-rw-rw-r-- 1 ralla cgrl      3284 Nov 13  2009 hg19.3.ebwt
-rw-rw-r-- 1 ralla cgrl 715335926 Nov 13  2009 hg19.4.ebwt
lrwxrwxrwx 1 ralla cgrl        45 Aug 21  2013 hg19.fa -> /global/referenceData/genomes/hs/hg19/hg19.fa
-rw-rw-r-- 1 ralla cgrl 821725563 Nov 13  2009 hg19.rev.1.ebwt
-rw-rw-r-- 1 ralla cgrl 357667968 Nov 13  2009 hg19.rev.2.ebwt

Thanks for looking into this.
Ravi.
On Mar 4, 2014, at 9:13 AM, Björn Grüning <[hidden email]> wrote:

Hi Ravi,

can you attach the loc file, do you see anything in the Galaxy log files about bowtie2, try grepping for "bowtie2".

Cheers,
Bjoern

Am 04.03.2014 18:07, schrieb Ravi Alla:
Hi Bjoern,
Thank you for your clarifications. I have tried installing bowtie2 using both variations, once with both wrapper and dependencies and once with only wrapper (since I have dependencies installed on the system). In either case even after editing the .loc file I cannot see the indices in galaxy. I tried the same edits to .loc file with bwa and the indices show right up, but not with bowtie2 and even bowtie for that matter. I really don't know what to do about this.
Thanks
Ravi
On Mar 4, 2014, at 3:06 AM, Björn Grüning <[hidden email]> wrote:

Hi Ravi,

Hi guys,
I am new to galaxy and am in the process of setting it up on a cluster with some help. I am trying to install bowtie2 to my local galaxy through the toolshed. I have a few questions.

- Which bowtie2 should I install? When I search for bowtie2 a bunch of results come up?

Only two, or? Please make sure you are using the main toolshed. bowtie2 and package_bowtie2_2_1_0 are connected together. One is the binary and the other contains the wrapper. Use devteam respositories is always a good choice.

- What are repository dependencies? Why would I need these if bowtie2 is already installed on my system?

repository dependecies containing all dependencies of your wrappers, that can be some binaries, but also R, Perl, python libraries that are needed to execute your wrappers. We recommend you to use the dependencies because then you have full control over all your used versions (wrapper, dependencies, tool) and that enables reproducibility. To put in other words: With the toolshed you can have different tool and wrappers versions with different dependencies at the same time.

- I have been installing bowtie2 by unchecking the repository dependencies and tool dependencies boxes. Is this correct?

It will work, if you have a system installed version. But you need to take care about binary updates by your own. And reproducibility of your results is not guaranteed.

- Finally when I install bowtie2 this way it creates 2 bowtie2_indices.loc files, one in galaxy/tool-data and the other in shed_tools/toolshed.g2.gx.psu.edu/dev/bowtie2, which one do I need to edit to point to my index files? No matter which one I change I can't seem to see the indices.

That is strange. Are you sure you don't have a spelling mistake in it. It should be the bowtie2_indices.los I think. I will CC Greg, he is one of the Tool Shed Developers and should know more about it.
Please do not forget to restart your Galaxy instance once you have update the *loc files.

I hope someone on here can help me out.

Hope it helped a little bit,
Bjoern

Thanks a lot
Ravi.
___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/







___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

-- 
Jennifer Hillman-Jackson
http://galaxyproject.org


___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
Reply | Threaded
Open this post in threaded view
|

Re: How to install bowtie2 tool in galaxy

Nate Coraor (nate@bx.psu.edu)
On Wed, Mar 5, 2014 at 8:18 PM, Ravi Alla <[hidden email]> wrote:

> Hi Jennifer,
> Thank you for this information. I was able to troubleshoot the
> bowtie2_indices. Like Bjoern said the bowtie2 tool needed to be reinstalled
> because the tool_table does not load the correct indices.
>
> I am still trying to wrap my head around the different toolsheds.
>
> The main tool shed resides in the galaxy-dist folder and uses tool_conf.xml
> to load tools into the side bar and tool_data_table_conf.xml to load indices
> (and other data) for the default tools that come with galaxy.

Hi Ravi,

The tools in the galaxy-dist directory and controlled via
tool_conf.xml are not a part of the tool shed. The inclusion of tools
directly in Galaxy predates the existence of the tool shed. Other than
that, what you have above is correct.

> The shed tools reside in the ../shed_tools/ directory and use
> shed_tools_conf.xml to load tools into the side bar and the
> shed_tools_data_table_conf.xml to load indices.
>
> The shed_tool xml is setup in the universe.ini file.

Right.

> The .loc files for default main tools reside in galaxy-dist/tool-data and
> the .loc files for the shed_tools reside in
> ../shed_tools/......./<tool-name>/<tool_id>/<tool-name>/

By default they'll all be under galaxy-dist/tool-data/.  The versions
in ../shed_tools are the sample location files provided by tool
authors.

> And when I uninstall tools and reinstall I would have to update the metadata
> for that tool.

I am not sure what you mean by this.

> If ../shed_tools dir is not present then shed-tools get installed under
> galaxy-dist/tool-data/toolshed.g2.bx.psu.edu/

../shed_tools should be created if it does not exist. Tools won't be
installed in galaxy-dist/tool-data/

> And it is always better to install tools as wrappers + dependencies when
> possible.

I agree.

--nate

>
> Am I on the right track with this tool organization?
> Thanks
> Ravi
>
>
>
> On Mar 5, 2014, at 11:06 AM, Jennifer Jackson <[hidden email]> wrote:
>
> Hi Ravi,
>
> The directory structure for the installation of ToolShed tools changed,
> which is why you have three directories. You perhaps had bowtie2 installed
> once before, then reinstalled (without completely removing the older version
> and associated data)? Or updated without resetting the metadata? In either
> case, the ../shed_tools directory is the new one. Having this as the path in
> your .xml configuration files (as Bjoern suggested earlier) and moving all
> contents & data to be under the same location will be the simplest global
> solution ongoing. Links near the end of my reply can help explain how-to.
>
> For the specific reason why bowtie2 indices are not working: I noticed that
> the reference genome ".fa" file is not linked from the directory containing
> the indexes. This is required. Adding it in, the same way that you did for
> the bowtie2 indexes, into this dir:
> "/global/referenceData/databases/bowtie2/hg19" will probably solve that part
> of the problem. I didn't see this posted - but I apologize if I am
> duplicating advice already given.
>
> I also tend to advise keeping all data under the same master "data"
> directory - all indexes and sequence data - as symbolic links to additional
> file system paths that are unknown to the 'galaxy user' cause a different
> set of problems. However, that said, this doesn't seem to be an issue in
> your specific case: if the bowtie2 indexes are functioning correctly - then
> environment is set up so that the other dir hierarchy where the .fa files
> are kept must included in the 'galaxy' user's ENV. Symbolic links that go
> outside of the local dir structure are known to cause problems unless the
> ENV config is carefully set up, and to my knowledge are best avoided
> entirely for certain uses such as "upload by file path" into libraries.
>
> For reference:
>
> NGS data set-up is described in this wiki - including expected content for
> each type of index:
> https://wiki.galaxyproject.org/Admin/NGS%20Local%20Setup
>
> For examples, you can rsync a genome or two and examine the contents. Or,
> our /location dir and have a look at the .loc files.
> https://wiki.galaxyproject.org/Admin/DataIntegration
>
> Tool Shed help (very detailed):
> https://wiki.galaxyproject.org/Tool%20Shed
> In particular, if you had previously installed repositories (this is not
> clear, just suspected from the duplications), updating the Metadata with
> certain distribution updates can be very important. This has been necessary
> for the last few releases to update to changes. The News Brief noted this,
> and included a link to this wiki page. Also see the "Related Pages" lower
> down on the wiki.
> https://wiki.galaxyproject.org/ResettingMetadataForInstalledRepositories
> This may also be useful:
> https://wiki.galaxyproject.org/RepairingInstalledRepositories
>
> Hopefully you have sorted most of this out by now, or this helps!
>
> Jen
> Galaxy team
>
>
> On 3/4/14 12:21 PM, Ravi Alla wrote:
>
> Bjoern,
> This is getting frustrating.
> There are three places bowtie2_indices.loc file is expected. I really don't
> know which one is the one I should modify.
>
> galaxy-dist/tool-data/bowtie2_indices.loc
> galaxy-dist/tool-data/toolshed.g2.bx.psu.edu/repos/devteam/bowtie2/96d2e31a3938/bowtie2_indices.loc
> ../shed_tools/toolshed.g2.bx.psu.edu/repos/devteam/bowtie2/96d2e31a3938/bowtie2/tool-data/bowtie2_indices.loc
>
> What is the difference between these 3 files?
>
> I changed the 2nd file to include the path to the indices and this shows up
> as a valid preloaded index for tophat2 but does not for bowtie2. I also
> changed the normal bowtie_indices.loc file at the second location and this
> works and I can see the bowtie1 indices for that bowtie tool. Currently only
> the bowtie2 indices are acting up.
>
> I wish this was easier.
> Thanks
> Ravi.
> On Mar 4, 2014, at 11:28 AM, Björn Grüning <[hidden email]>
> wrote:
>
> Hi,
>
> can you check if the file tool_data_table_conf.xml contains an entry about
> bowtie2_indexes ...  or the table shed_data_table_conf.xml ... in one of
> them should be one entry with bowtie2_indexes.
>
> That warning should not be there I think:
>
>> galaxy.tools.parameters.dynamic_options WARNING 2014-03-04 10:14:33,335
>> Data table named 'bowtie2_indexes' is required by tool but not configured
>
> Hope you can figure it out,
> Bjoern
>
> Am 04.03.2014 19:19, schrieb Ravi Alla:
>
> Hi Bjoern,
> Please find the bowtie2 and bowtie .loc files. In the paster.log file I see
> these entries for bowtie
>
> galaxy.tools.data DEBUG 2014-03-04 10:14:11,048 Loaded tool data table
> 'bowtie_indexes'
> galaxy.tools.data DEBUG 2014-03-04 10:14:11,051 Loading another instance of
> data table 'bowtie_indexes', attempting to merge content.
> galaxy.tools.parameters.dynamic_options WARNING 2014-03-04 10:14:33,335 Data
> table named 'bowtie2_indexes' is required by tool but not configured
>
> And ls -l /global/referenceData/databases/bowtie2/hg19 is
> drwxr-xr-x 2 ralla cgrl      2048 Sep 10 11:14 .
> drwxr-xr-x 3 ralla cgrl      2048 Sep 10 11:06 ..
> -rw-rw-r-- 1 ralla cgrl 960018873 May  2  2012 hg19.1.bt2
> -rw-rw-r-- 1 ralla cgrl 716863572 May  2  2012 hg19.2.bt2
> -rw-rw-r-- 1 ralla cgrl      3833 May  2  2012 hg19.3.bt2
> -rw-rw-r-- 1 ralla cgrl 716863565 May  2  2012 hg19.4.bt2
> -rw-rw-r-- 1 ralla cgrl 960018873 May  2  2012 hg19.rev.1.bt2
> -rw-rw-r-- 1 ralla cgrl 716863572 May  2  2012 hg19.rev.2.bt2
>
> and ls -l /global/referenceData/databases/bowtie/hg19 is
> drwxr-xr-x 2 ralla cgrl      2048 Mar  4 10:07 .
> drwxr-xr-x 3 ralla cgrl      2048 Nov 30  2012 ..
> -rw-rw-r-- 1 ralla cgrl 821725563 Nov 13  2009 hg19.1.ebwt
> -rw-rw-r-- 1 ralla cgrl 357667968 Nov 13  2009 hg19.2.ebwt
> -rw-rw-r-- 1 ralla cgrl      3284 Nov 13  2009 hg19.3.ebwt
> -rw-rw-r-- 1 ralla cgrl 715335926 Nov 13  2009 hg19.4.ebwt
> lrwxrwxrwx 1 ralla cgrl        45 Aug 21  2013 hg19.fa ->
> /global/referenceData/genomes/hs/hg19/hg19.fa
> -rw-rw-r-- 1 ralla cgrl 821725563 Nov 13  2009 hg19.rev.1.ebwt
> -rw-rw-r-- 1 ralla cgrl 357667968 Nov 13  2009 hg19.rev.2.ebwt
>
> Thanks for looking into this.
> Ravi.
> On Mar 4, 2014, at 9:13 AM, Björn Grüning <[hidden email]> wrote:
>
> Hi Ravi,
>
> can you attach the loc file, do you see anything in the Galaxy log files
> about bowtie2, try grepping for "bowtie2".
>
> Cheers,
> Bjoern
>
> Am 04.03.2014 18:07, schrieb Ravi Alla:
>
> Hi Bjoern,
> Thank you for your clarifications. I have tried installing bowtie2 using
> both variations, once with both wrapper and dependencies and once with only
> wrapper (since I have dependencies installed on the system). In either case
> even after editing the .loc file I cannot see the indices in galaxy. I tried
> the same edits to .loc file with bwa and the indices show right up, but not
> with bowtie2 and even bowtie for that matter. I really don't know what to do
> about this.
> Thanks
> Ravi
> On Mar 4, 2014, at 3:06 AM, Björn Grüning <[hidden email]> wrote:
>
> Hi Ravi,
>
> Hi guys,
> I am new to galaxy and am in the process of setting it up on a cluster with
> some help. I am trying to install bowtie2 to my local galaxy through the
> toolshed. I have a few questions.
>
> - Which bowtie2 should I install? When I search for bowtie2 a bunch of
> results come up?
>
>
> Only two, or? Please make sure you are using the main toolshed. bowtie2 and
> package_bowtie2_2_1_0 are connected together. One is the binary and the
> other contains the wrapper. Use devteam respositories is always a good
> choice.
>
> - What are repository dependencies? Why would I need these if bowtie2 is
> already installed on my system?
>
>
> repository dependecies containing all dependencies of your wrappers, that
> can be some binaries, but also R, Perl, python libraries that are needed to
> execute your wrappers. We recommend you to use the dependencies because then
> you have full control over all your used versions (wrapper, dependencies,
> tool) and that enables reproducibility. To put in other words: With the
> toolshed you can have different tool and wrappers versions with different
> dependencies at the same time.
>
> - I have been installing bowtie2 by unchecking the repository dependencies
> and tool dependencies boxes. Is this correct?
>
>
> It will work, if you have a system installed version. But you need to take
> care about binary updates by your own. And reproducibility of your results
> is not guaranteed.
>
> - Finally when I install bowtie2 this way it creates 2 bowtie2_indices.loc
> files, one in galaxy/tool-data and the other in
> shed_tools/toolshed.g2.gx.psu.edu/dev/bowtie2, which one do I need to edit
> to point to my index files? No matter which one I change I can't seem to see
> the indices.
>
>
> That is strange. Are you sure you don't have a spelling mistake in it. It
> should be the bowtie2_indices.los I think. I will CC Greg, he is one of the
> Tool Shed Developers and should know more about it.
> Please do not forget to restart your Galaxy instance once you have update
> the *loc files.
>
> I hope someone on here can help me out.
>
>
> Hope it helped a little bit,
> Bjoern
>
> Thanks a lot
> Ravi.
> ___________________________________________________________
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>   http://lists.bx.psu.edu/
>
> To search Galaxy mailing lists use the unified search at:
>   http://galaxyproject.org/search/mailinglists/
>
>
>
>
>
>
>
> ___________________________________________________________
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>   http://lists.bx.psu.edu/
>
> To search Galaxy mailing lists use the unified search at:
>   http://galaxyproject.org/search/mailinglists/
>
>
> --
> Jennifer Hillman-Jackson
> http://galaxyproject.org
>
>
>
> ___________________________________________________________
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>   http://lists.bx.psu.edu/
>
> To search Galaxy mailing lists use the unified search at:
>   http://galaxyproject.org/search/mailinglists/

___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
Reply | Threaded
Open this post in threaded view
|

Re: How to install bowtie2 tool in galaxy

Ravi Alla
Hi Nate 
Thanks for the clarifications. Is there anything specific I should watch out for when updating and deleting tools? The reason I ask is that Jennifer mentioned in previous emails the need to update metadata for tools, which I am sure I follow.
Thanks
Ravi
On Mar 14, 2014, at 9:02 AM, Nate Coraor <[hidden email]> wrote:

On Wed, Mar 5, 2014 at 8:18 PM, Ravi Alla <[hidden email]> wrote:
Hi Jennifer,
Thank you for this information. I was able to troubleshoot the
bowtie2_indices. Like Bjoern said the bowtie2 tool needed to be reinstalled
because the tool_table does not load the correct indices.

I am still trying to wrap my head around the different toolsheds.

The main tool shed resides in the galaxy-dist folder and uses tool_conf.xml
to load tools into the side bar and tool_data_table_conf.xml to load indices
(and other data) for the default tools that come with galaxy.

Hi Ravi,

The tools in the galaxy-dist directory and controlled via
tool_conf.xml are not a part of the tool shed. The inclusion of tools
directly in Galaxy predates the existence of the tool shed. Other than
that, what you have above is correct.

The shed tools reside in the ../shed_tools/ directory and use
shed_tools_conf.xml to load tools into the side bar and the
shed_tools_data_table_conf.xml to load indices.

The shed_tool xml is setup in the universe.ini file.

Right.

The .loc files for default main tools reside in galaxy-dist/tool-data and
the .loc files for the shed_tools reside in
../shed_tools/......./<tool-name>/<tool_id>/<tool-name>/

By default they'll all be under galaxy-dist/tool-data/.  The versions
in ../shed_tools are the sample location files provided by tool
authors.

And when I uninstall tools and reinstall I would have to update the metadata
for that tool.

I am not sure what you mean by this.

If ../shed_tools dir is not present then shed-tools get installed under
galaxy-dist/tool-data/toolshed.g2.bx.psu.edu/

../shed_tools should be created if it does not exist. Tools won't be
installed in galaxy-dist/tool-data/

And it is always better to install tools as wrappers + dependencies when
possible.

I agree.

--nate


Am I on the right track with this tool organization?
Thanks
Ravi



On Mar 5, 2014, at 11:06 AM, Jennifer Jackson <[hidden email]> wrote:

Hi Ravi,

The directory structure for the installation of ToolShed tools changed,
which is why you have three directories. You perhaps had bowtie2 installed
once before, then reinstalled (without completely removing the older version
and associated data)? Or updated without resetting the metadata? In either
case, the ../shed_tools directory is the new one. Having this as the path in
your .xml configuration files (as Bjoern suggested earlier) and moving all
contents & data to be under the same location will be the simplest global
solution ongoing. Links near the end of my reply can help explain how-to.

For the specific reason why bowtie2 indices are not working: I noticed that
the reference genome ".fa" file is not linked from the directory containing
the indexes. This is required. Adding it in, the same way that you did for
the bowtie2 indexes, into this dir:
"/global/referenceData/databases/bowtie2/hg19" will probably solve that part
of the problem. I didn't see this posted - but I apologize if I am
duplicating advice already given.

I also tend to advise keeping all data under the same master "data"
directory - all indexes and sequence data - as symbolic links to additional
file system paths that are unknown to the 'galaxy user' cause a different
set of problems. However, that said, this doesn't seem to be an issue in
your specific case: if the bowtie2 indexes are functioning correctly - then
environment is set up so that the other dir hierarchy where the .fa files
are kept must included in the 'galaxy' user's ENV. Symbolic links that go
outside of the local dir structure are known to cause problems unless the
ENV config is carefully set up, and to my knowledge are best avoided
entirely for certain uses such as "upload by file path" into libraries.

For reference:

NGS data set-up is described in this wiki - including expected content for
each type of index:
https://wiki.galaxyproject.org/Admin/NGS%20Local%20Setup

For examples, you can rsync a genome or two and examine the contents. Or,
our /location dir and have a look at the .loc files.
https://wiki.galaxyproject.org/Admin/DataIntegration

Tool Shed help (very detailed):
https://wiki.galaxyproject.org/Tool%20Shed
In particular, if you had previously installed repositories (this is not
clear, just suspected from the duplications), updating the Metadata with
certain distribution updates can be very important. This has been necessary
for the last few releases to update to changes. The News Brief noted this,
and included a link to this wiki page. Also see the "Related Pages" lower
down on the wiki.
https://wiki.galaxyproject.org/ResettingMetadataForInstalledRepositories
This may also be useful:
https://wiki.galaxyproject.org/RepairingInstalledRepositories

Hopefully you have sorted most of this out by now, or this helps!

Jen
Galaxy team


On 3/4/14 12:21 PM, Ravi Alla wrote:

Bjoern,
This is getting frustrating.
There are three places bowtie2_indices.loc file is expected. I really don't
know which one is the one I should modify.

galaxy-dist/tool-data/bowtie2_indices.loc
galaxy-dist/tool-data/toolshed.g2.bx.psu.edu/repos/devteam/bowtie2/96d2e31a3938/bowtie2_indices.loc
../shed_tools/toolshed.g2.bx.psu.edu/repos/devteam/bowtie2/96d2e31a3938/bowtie2/tool-data/bowtie2_indices.loc

What is the difference between these 3 files?

I changed the 2nd file to include the path to the indices and this shows up
as a valid preloaded index for tophat2 but does not for bowtie2. I also
changed the normal bowtie_indices.loc file at the second location and this
works and I can see the bowtie1 indices for that bowtie tool. Currently only
the bowtie2 indices are acting up.

I wish this was easier.
Thanks
Ravi.
On Mar 4, 2014, at 11:28 AM, Björn Grüning <[hidden email]>
wrote:

Hi,

can you check if the file tool_data_table_conf.xml contains an entry about
bowtie2_indexes ...  or the table shed_data_table_conf.xml ... in one of
them should be one entry with bowtie2_indexes.

That warning should not be there I think:

galaxy.tools.parameters.dynamic_options WARNING 2014-03-04 10:14:33,335
Data table named 'bowtie2_indexes' is required by tool but not configured

Hope you can figure it out,
Bjoern

Am 04.03.2014 19:19, schrieb Ravi Alla:

Hi Bjoern,
Please find the bowtie2 and bowtie .loc files. In the paster.log file I see
these entries for bowtie

galaxy.tools.data DEBUG 2014-03-04 10:14:11,048 Loaded tool data table
'bowtie_indexes'
galaxy.tools.data DEBUG 2014-03-04 10:14:11,051 Loading another instance of
data table 'bowtie_indexes', attempting to merge content.
galaxy.tools.parameters.dynamic_options WARNING 2014-03-04 10:14:33,335 Data
table named 'bowtie2_indexes' is required by tool but not configured

And ls -l /global/referenceData/databases/bowtie2/hg19 is
drwxr-xr-x 2 ralla cgrl      2048 Sep 10 11:14 .
drwxr-xr-x 3 ralla cgrl      2048 Sep 10 11:06 ..
-rw-rw-r-- 1 ralla cgrl 960018873 May  2  2012 hg19.1.bt2
-rw-rw-r-- 1 ralla cgrl 716863572 May  2  2012 hg19.2.bt2
-rw-rw-r-- 1 ralla cgrl      3833 May  2  2012 hg19.3.bt2
-rw-rw-r-- 1 ralla cgrl 716863565 May  2  2012 hg19.4.bt2
-rw-rw-r-- 1 ralla cgrl 960018873 May  2  2012 hg19.rev.1.bt2
-rw-rw-r-- 1 ralla cgrl 716863572 May  2  2012 hg19.rev.2.bt2

and ls -l /global/referenceData/databases/bowtie/hg19 is
drwxr-xr-x 2 ralla cgrl      2048 Mar  4 10:07 .
drwxr-xr-x 3 ralla cgrl      2048 Nov 30  2012 ..
-rw-rw-r-- 1 ralla cgrl 821725563 Nov 13  2009 hg19.1.ebwt
-rw-rw-r-- 1 ralla cgrl 357667968 Nov 13  2009 hg19.2.ebwt
-rw-rw-r-- 1 ralla cgrl      3284 Nov 13  2009 hg19.3.ebwt
-rw-rw-r-- 1 ralla cgrl 715335926 Nov 13  2009 hg19.4.ebwt
lrwxrwxrwx 1 ralla cgrl        45 Aug 21  2013 hg19.fa ->
/global/referenceData/genomes/hs/hg19/hg19.fa
-rw-rw-r-- 1 ralla cgrl 821725563 Nov 13  2009 hg19.rev.1.ebwt
-rw-rw-r-- 1 ralla cgrl 357667968 Nov 13  2009 hg19.rev.2.ebwt

Thanks for looking into this.
Ravi.
On Mar 4, 2014, at 9:13 AM, Björn Grüning <[hidden email]> wrote:

Hi Ravi,

can you attach the loc file, do you see anything in the Galaxy log files
about bowtie2, try grepping for "bowtie2".

Cheers,
Bjoern

Am 04.03.2014 18:07, schrieb Ravi Alla:

Hi Bjoern,
Thank you for your clarifications. I have tried installing bowtie2 using
both variations, once with both wrapper and dependencies and once with only
wrapper (since I have dependencies installed on the system). In either case
even after editing the .loc file I cannot see the indices in galaxy. I tried
the same edits to .loc file with bwa and the indices show right up, but not
with bowtie2 and even bowtie for that matter. I really don't know what to do
about this.
Thanks
Ravi
On Mar 4, 2014, at 3:06 AM, Björn Grüning <[hidden email]> wrote:

Hi Ravi,

Hi guys,
I am new to galaxy and am in the process of setting it up on a cluster with
some help. I am trying to install bowtie2 to my local galaxy through the
toolshed. I have a few questions.

- Which bowtie2 should I install? When I search for bowtie2 a bunch of
results come up?


Only two, or? Please make sure you are using the main toolshed. bowtie2 and
package_bowtie2_2_1_0 are connected together. One is the binary and the
other contains the wrapper. Use devteam respositories is always a good
choice.

- What are repository dependencies? Why would I need these if bowtie2 is
already installed on my system?


repository dependecies containing all dependencies of your wrappers, that
can be some binaries, but also R, Perl, python libraries that are needed to
execute your wrappers. We recommend you to use the dependencies because then
you have full control over all your used versions (wrapper, dependencies,
tool) and that enables reproducibility. To put in other words: With the
toolshed you can have different tool and wrappers versions with different
dependencies at the same time.

- I have been installing bowtie2 by unchecking the repository dependencies
and tool dependencies boxes. Is this correct?


It will work, if you have a system installed version. But you need to take
care about binary updates by your own. And reproducibility of your results
is not guaranteed.

- Finally when I install bowtie2 this way it creates 2 bowtie2_indices.loc
files, one in galaxy/tool-data and the other in
shed_tools/toolshed.g2.gx.psu.edu/dev/bowtie2, which one do I need to edit
to point to my index files? No matter which one I change I can't seem to see
the indices.


That is strange. Are you sure you don't have a spelling mistake in it. It
should be the bowtie2_indices.los I think. I will CC Greg, he is one of the
Tool Shed Developers and should know more about it.
Please do not forget to restart your Galaxy instance once you have update
the *loc files.

I hope someone on here can help me out.


Hope it helped a little bit,
Bjoern

Thanks a lot
Ravi.
___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
 http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
 http://galaxyproject.org/search/mailinglists/







___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
 http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
 http://galaxyproject.org/search/mailinglists/


--
Jennifer Hillman-Jackson
http://galaxyproject.org



___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
 http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
 http://galaxyproject.org/search/mailinglists/


___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/