How to test my functional tests?

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

How to test my functional tests?

jeltje
Hi,

I'm trying to find out what the best way is to see if my functional tests work as expected, and I'm also wondering when these tests are (supposed to be) run. Here are the details:

Following instructions at https://wiki.galaxyproject.org/Admin/Tools/AddToolTutorial, I created a little toolExample tool under galaxy-dist/tools/myTools. I also created test input and output in a test-data directory and added a test to the toolExample.xml file:
  <tests>
    <test>
      <param name="input" value="testinput.fa"/>
      <output name="out_file1" file="testoutput.txt"/>
    </test>
  </tests>

And yes, the files are named correctly:
tools/myTools/test-data/testinput.fa
tools/myTools/test-data/testoutput.txt

I added the new tool to config/tools_conf.xml AND to config/tools_conf.xml.sample.
Question #1: Why is a .sample file used by run_functional_tests.sh? I thought with .sample extensions were meant as examples for creating your own versions, I'm really surprised that any script would want to access these.
I restarted galaxy to verify that the new tool showed up, then checked that I could test it:

./run_functional_tests.sh -list | grep fa_gc_content_1
It shows up in the list of tools. So then I tested it:

./run_functional_tests.sh -id fa_gc_content_1

This gives
IOError: [Errno 2] No such file or directory: '/inside/home/jeltje/exp/varscan2/programs/galaxy-dist/test-data/testinput.fa'

So it looks like at some point the testinput should have been copied to galaxy-dist/test-data but wasn't.
Question # 2: Is there some setting that makes this test input (and output) get copied to the correct location, or do I have to do this manually?

I also tried testing by uploading the tool to the testtoolshed (as jtest1). I deliberatly put the wrong value in the testoutput.txt file because I'd like to see a test fail before I'll accept that it succeeds. It appears the testinput does not get tested on upload, because I got no automated test results, and no error messages anywhere. Is that expected behavior? I know the tools get tested every so often, I'm just surprised it doesn't happen on upload.
The tool could then also be easily installed to my galaxy distribution, without errors in my log or on the browser pages.
Question #3: At what point in creating or downloading a repository are these tests run?

And lastly, #4: is either of these the best way to test my tests, or did I overlook something?

Thank you!

-Jeltje

___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
Reply | Threaded
Open this post in threaded view
|

Re: How to test my functional tests?

Peter Cock
On Thu, Jan 8, 2015 at 3:27 AM, Jeltje van Baren
<[hidden email]> wrote:

> Hi,
>
> I'm trying to find out what the best way is to see if my functional tests
> work as expected, and I'm also wondering when these tests are (supposed to
> be) run. Here are the details:
>
> Following instructions at
> https://wiki.galaxyproject.org/Admin/Tools/AddToolTutorial, I created a
> little toolExample tool under galaxy-dist/tools/myTools. I also created test
> input and output in a test-data directory and added a test to the
> toolExample.xml file:
>   <tests>
>     <test>
>       <param name="input" value="testinput.fa"/>
>       <output name="out_file1" file="testoutput.txt"/>
>     </test>
>   </tests>
>
> And yes, the files are named correctly:
> tools/myTools/test-data/testinput.fa
> tools/myTools/test-data/testoutput.txt

If you are manually installing the tool directly under the folder
galaxy-dist/tools/myTools then you should put your test
files under galaxy-dist/test-data (this is how all the tools
used to work back before the ToolShed).

If you want to do the tool installation via a ToolShed,
things are different (more complicated for the tool
development side, but it becomes easier for the end
user to install things).

> I added the new tool to config/tools_conf.xml AND to
> config/tools_conf.xml.sample.
> Question #1: Why is a .sample file used by run_functional_tests.sh? I
> thought with .sample extensions were meant as examples for creating your own
> versions, I'm really surprised that any script would want to access these.

Historically (for reasons unclear to me), the tools for testing
were listed in galaxy-dist/tools_conf.xml.sample however
that was changed (about a year ago? I forget now) to simply
use galaxy-dist/tools_conf.xml (the master tool listing as used
before the ToolShed existed).

You might be using different settings in your main ini file,
but on our setup it is galaxy-dist/tools_conf.xml and not
galaxy-dist/config/tools_conf.xml (which seems to be in
use on your system given the new tool showed up on
restarting Galaxy):

> I restarted galaxy to verify that the new tool showed up,
> then checked that I could test it:
>
> ./run_functional_tests.sh -list | grep fa_gc_content_1
> It shows up in the list of tools. So then I tested it:
>
> ./run_functional_tests.sh -id fa_gc_content_1

Good - although run_functional_tests.sh is now obsolete
and just a backward compatible alias for run_tests.sh.

If you read this in some (out of date) documentation,
please say where so we can update it.

> This gives
> IOError: [Errno 2] No such file or directory:
> '/inside/home/jeltje/exp/varscan2/programs/galaxy-dist/test-data/testinput.fa'
>
> So it looks like at some point the testinput should have been copied to
> galaxy-dist/test-data but wasn't.

No, as noted above, you have to put the test files there
by hand (as long as you are deliberately doing all your
tool installation/development avoiding the ToolShed,
which is quite a common approach if you have a
separate Galaxy instance for development).

> Question # 2: Is there some setting that makes this test input (and output)
> get copied to the correct location, or do I have to do this manually?

No, unless you go down the ToolShed route, do this manually.

> I also tried testing by uploading the tool to the testtoolshed (as jtest1).
> I deliberatly put the wrong value in the testoutput.txt file because I'd
> like to see a test fail before I'll accept that it succeeds. It appears the
> testinput does not get tested on upload, because I got no automated test
> results, and no error messages anywhere. Is that expected behavior? I know
> the tools get tested every so often, I'm just surprised it doesn't happen on
> upload.
> The tool could then also be easily installed to my galaxy distribution,
> without errors in my log or on the browser pages.

The main and Test Tool Shed should now be running the tests
every 48 hours or so - this service had been unavailable over
the Christmas/New Year period but should be OK now.

> Question #3: At what point in creating or downloading a repository
> are these tests run?

Never, assuming I understand your question.

If you mean installing a tool from the ToolShed into a Galaxy
instance via the Galaxy administrator options, then I think
after installation any tests should be run automatically.

> And lastly, #4: is either of these the best way to test my tests, or did I
> overlook something?
>
> Thank you!
>
> -Jeltje

While developing a new tool (or improving one), I personally
use a development Galaxy server (not the one our users run
all their real jobs on), and install the tools manually (as above),
and then test via run_tests.sh -id my_tool_id.

This is complemented by a TravisCI setup, see:
http://blastedbio.blogspot.jp/2013/09/using-travis-ci-for-testing-galaxy-tools.html

If that works, I upload the tool to the TestToolShed, and wait
for those test results. If that all works and I am ready to make
a public release, then I upload to the main ToolShed.

I talked about this (in the context of the NCBI BLAST+
wrappers) at the GCC2014 conference, slides here:
http://www.slideshare.net/pjacock/galaxy-blast-gcc2014

There is even a video of my talk (and most of the others)
on the Galaxy website here:
https://wiki.galaxyproject.org/Events/GCC2014/Abstracts

I hope that helped. There is a work in progress
"Best Practices" being worked on here:
https://wiki.galaxyproject.org/Tools/BestPractices

Regards,

Peter
___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
Reply | Threaded
Open this post in threaded view
|

Re: How to test my functional tests?

John Chilton-4
In addition to the two paths Peter laid out for testing tools:

1 - Embedding them into a Galaxy instance, registering the tools in
$GALAXY_ROOT/config/tool_conf.xml, and populating
$GALAXY_ROOT/test-data and Galaxy's index data as needed, and then
running ./run_tests.sh -id <tool_id>
2 - Publishing them to the test toolshed along with dependencies and
waiting for the tool shed nightly framework to register test results.

There is a newer, light-weight alternative where you can use your
local system dependencies but keep the tools separated out into their
own repository and not worry about how Galaxy is configured during
testing. This is using the new tool Planemo - there is a quick start
here (http://planemo.readthedocs.org/en/latest/readme.html#quick-start).

Installing Planemo (assuming you have Linuxbrew or Homebrew) is as easy

% brew tap galaxyproject/tap
% brew install planemo

Then one can run tool tests using

% planemo t --galaxy_root=/path/to/galaxy example_tool.xml

Or if you don't have a Galaxy on the system

% planemo t --install_galaxy example_tool.xml

Planemo runs Galaxy with dynamic settings so there is no need to
modify Galaxy's tool_conf.xml or need to to know the id of the tool.
Planemo also produces prettier, more informative reports of tool test
results (e.g. http://bit.ly/planemotestreportexample) and also offers
linting functionality so you can catch more bugs quickly before you
even need to run tests.

More information at:

http://planemo.readthedocs.org/
https://github.com/galaxyproject/planemo

Hope this helps,
-John









On Thu, Jan 8, 2015 at 12:03 AM, Peter Cock <[hidden email]> wrote:

> On Thu, Jan 8, 2015 at 3:27 AM, Jeltje van Baren
> <[hidden email]> wrote:
>> Hi,
>>
>> I'm trying to find out what the best way is to see if my functional tests
>> work as expected, and I'm also wondering when these tests are (supposed to
>> be) run. Here are the details:
>>
>> Following instructions at
>> https://wiki.galaxyproject.org/Admin/Tools/AddToolTutorial, I created a
>> little toolExample tool under galaxy-dist/tools/myTools. I also created test
>> input and output in a test-data directory and added a test to the
>> toolExample.xml file:
>>   <tests>
>>     <test>
>>       <param name="input" value="testinput.fa"/>
>>       <output name="out_file1" file="testoutput.txt"/>
>>     </test>
>>   </tests>
>>
>> And yes, the files are named correctly:
>> tools/myTools/test-data/testinput.fa
>> tools/myTools/test-data/testoutput.txt
>
> If you are manually installing the tool directly under the folder
> galaxy-dist/tools/myTools then you should put your test
> files under galaxy-dist/test-data (this is how all the tools
> used to work back before the ToolShed).
>
> If you want to do the tool installation via a ToolShed,
> things are different (more complicated for the tool
> development side, but it becomes easier for the end
> user to install things).
>
>> I added the new tool to config/tools_conf.xml AND to
>> config/tools_conf.xml.sample.
>> Question #1: Why is a .sample file used by run_functional_tests.sh? I
>> thought with .sample extensions were meant as examples for creating your own
>> versions, I'm really surprised that any script would want to access these.
>
> Historically (for reasons unclear to me), the tools for testing
> were listed in galaxy-dist/tools_conf.xml.sample however
> that was changed (about a year ago? I forget now) to simply
> use galaxy-dist/tools_conf.xml (the master tool listing as used
> before the ToolShed existed).
>
> You might be using different settings in your main ini file,
> but on our setup it is galaxy-dist/tools_conf.xml and not
> galaxy-dist/config/tools_conf.xml (which seems to be in
> use on your system given the new tool showed up on
> restarting Galaxy):
>
>> I restarted galaxy to verify that the new tool showed up,
>> then checked that I could test it:
>>
>> ./run_functional_tests.sh -list | grep fa_gc_content_1
>> It shows up in the list of tools. So then I tested it:
>>
>> ./run_functional_tests.sh -id fa_gc_content_1
>
> Good - although run_functional_tests.sh is now obsolete
> and just a backward compatible alias for run_tests.sh.
>
> If you read this in some (out of date) documentation,
> please say where so we can update it.
>
>> This gives
>> IOError: [Errno 2] No such file or directory:
>> '/inside/home/jeltje/exp/varscan2/programs/galaxy-dist/test-data/testinput.fa'
>>
>> So it looks like at some point the testinput should have been copied to
>> galaxy-dist/test-data but wasn't.
>
> No, as noted above, you have to put the test files there
> by hand (as long as you are deliberately doing all your
> tool installation/development avoiding the ToolShed,
> which is quite a common approach if you have a
> separate Galaxy instance for development).
>
>> Question # 2: Is there some setting that makes this test input (and output)
>> get copied to the correct location, or do I have to do this manually?
>
> No, unless you go down the ToolShed route, do this manually.
>
>> I also tried testing by uploading the tool to the testtoolshed (as jtest1).
>> I deliberatly put the wrong value in the testoutput.txt file because I'd
>> like to see a test fail before I'll accept that it succeeds. It appears the
>> testinput does not get tested on upload, because I got no automated test
>> results, and no error messages anywhere. Is that expected behavior? I know
>> the tools get tested every so often, I'm just surprised it doesn't happen on
>> upload.
>> The tool could then also be easily installed to my galaxy distribution,
>> without errors in my log or on the browser pages.
>
> The main and Test Tool Shed should now be running the tests
> every 48 hours or so - this service had been unavailable over
> the Christmas/New Year period but should be OK now.
>
>> Question #3: At what point in creating or downloading a repository
>> are these tests run?
>
> Never, assuming I understand your question.
>
> If you mean installing a tool from the ToolShed into a Galaxy
> instance via the Galaxy administrator options, then I think
> after installation any tests should be run automatically.
>
>> And lastly, #4: is either of these the best way to test my tests, or did I
>> overlook something?
>>
>> Thank you!
>>
>> -Jeltje
>
> While developing a new tool (or improving one), I personally
> use a development Galaxy server (not the one our users run
> all their real jobs on), and install the tools manually (as above),
> and then test via run_tests.sh -id my_tool_id.
>
> This is complemented by a TravisCI setup, see:
> http://blastedbio.blogspot.jp/2013/09/using-travis-ci-for-testing-galaxy-tools.html
>
> If that works, I upload the tool to the TestToolShed, and wait
> for those test results. If that all works and I am ready to make
> a public release, then I upload to the main ToolShed.
>
> I talked about this (in the context of the NCBI BLAST+
> wrappers) at the GCC2014 conference, slides here:
> http://www.slideshare.net/pjacock/galaxy-blast-gcc2014
>
> There is even a video of my talk (and most of the others)
> on the Galaxy website here:
> https://wiki.galaxyproject.org/Events/GCC2014/Abstracts
>
> I hope that helped. There is a work in progress
> "Best Practices" being worked on here:
> https://wiki.galaxyproject.org/Tools/BestPractices
>
> Regards,
>
> Peter
> ___________________________________________________________
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>   https://lists.galaxyproject.org/
>
> To search Galaxy mailing lists use the unified search at:
>   http://galaxyproject.org/search/mailinglists/
___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
Reply | Threaded
Open this post in threaded view
|

Re: How to test my functional tests?

jeltje
In reply to this post by Peter Cock
Thanks for the quick responses John and Peter! I feel fortified to carry on.

-Jeltje

On Wed, Jan 7, 2015 at 9:03 PM, Peter Cock <[hidden email]> wrote:
On Thu, Jan 8, 2015 at 3:27 AM, Jeltje van Baren
<[hidden email]> wrote:
> Hi,
>
> I'm trying to find out what the best way is to see if my functional tests
> work as expected, and I'm also wondering when these tests are (supposed to
> be) run. Here are the details:
>
> Following instructions at
> https://wiki.galaxyproject.org/Admin/Tools/AddToolTutorial, I created a
> little toolExample tool under galaxy-dist/tools/myTools. I also created test
> input and output in a test-data directory and added a test to the
> toolExample.xml file:
>   <tests>
>     <test>
>       <param name="input" value="testinput.fa"/>
>       <output name="out_file1" file="testoutput.txt"/>
>     </test>
>   </tests>
>
> And yes, the files are named correctly:
> tools/myTools/test-data/testinput.fa
> tools/myTools/test-data/testoutput.txt

If you are manually installing the tool directly under the folder
galaxy-dist/tools/myTools then you should put your test
files under galaxy-dist/test-data (this is how all the tools
used to work back before the ToolShed).

If you want to do the tool installation via a ToolShed,
things are different (more complicated for the tool
development side, but it becomes easier for the end
user to install things).

> I added the new tool to config/tools_conf.xml AND to
> config/tools_conf.xml.sample.
> Question #1: Why is a .sample file used by run_functional_tests.sh? I
> thought with .sample extensions were meant as examples for creating your own
> versions, I'm really surprised that any script would want to access these.

Historically (for reasons unclear to me), the tools for testing
were listed in galaxy-dist/tools_conf.xml.sample however
that was changed (about a year ago? I forget now) to simply
use galaxy-dist/tools_conf.xml (the master tool listing as used
before the ToolShed existed).

You might be using different settings in your main ini file,
but on our setup it is galaxy-dist/tools_conf.xml and not
galaxy-dist/config/tools_conf.xml (which seems to be in
use on your system given the new tool showed up on
restarting Galaxy):

> I restarted galaxy to verify that the new tool showed up,
> then checked that I could test it:
>
> ./run_functional_tests.sh -list | grep fa_gc_content_1
> It shows up in the list of tools. So then I tested it:
>
> ./run_functional_tests.sh -id fa_gc_content_1

Good - although run_functional_tests.sh is now obsolete
and just a backward compatible alias for run_tests.sh.

If you read this in some (out of date) documentation,
please say where so we can update it.

> This gives
> IOError: [Errno 2] No such file or directory:
> '/inside/home/jeltje/exp/varscan2/programs/galaxy-dist/test-data/testinput.fa'
>
> So it looks like at some point the testinput should have been copied to
> galaxy-dist/test-data but wasn't.

No, as noted above, you have to put the test files there
by hand (as long as you are deliberately doing all your
tool installation/development avoiding the ToolShed,
which is quite a common approach if you have a
separate Galaxy instance for development).

> Question # 2: Is there some setting that makes this test input (and output)
> get copied to the correct location, or do I have to do this manually?

No, unless you go down the ToolShed route, do this manually.

> I also tried testing by uploading the tool to the testtoolshed (as jtest1).
> I deliberatly put the wrong value in the testoutput.txt file because I'd
> like to see a test fail before I'll accept that it succeeds. It appears the
> testinput does not get tested on upload, because I got no automated test
> results, and no error messages anywhere. Is that expected behavior? I know
> the tools get tested every so often, I'm just surprised it doesn't happen on
> upload.
> The tool could then also be easily installed to my galaxy distribution,
> without errors in my log or on the browser pages.

The main and Test Tool Shed should now be running the tests
every 48 hours or so - this service had been unavailable over
the Christmas/New Year period but should be OK now.

> Question #3: At what point in creating or downloading a repository
> are these tests run?

Never, assuming I understand your question.

If you mean installing a tool from the ToolShed into a Galaxy
instance via the Galaxy administrator options, then I think
after installation any tests should be run automatically.

> And lastly, #4: is either of these the best way to test my tests, or did I
> overlook something?
>
> Thank you!
>
> -Jeltje

While developing a new tool (or improving one), I personally
use a development Galaxy server (not the one our users run
all their real jobs on), and install the tools manually (as above),
and then test via run_tests.sh -id my_tool_id.

This is complemented by a TravisCI setup, see:
http://blastedbio.blogspot.jp/2013/09/using-travis-ci-for-testing-galaxy-tools.html

If that works, I upload the tool to the TestToolShed, and wait
for those test results. If that all works and I am ready to make
a public release, then I upload to the main ToolShed.

I talked about this (in the context of the NCBI BLAST+
wrappers) at the GCC2014 conference, slides here:
http://www.slideshare.net/pjacock/galaxy-blast-gcc2014

There is even a video of my talk (and most of the others)
on the Galaxy website here:
https://wiki.galaxyproject.org/Events/GCC2014/Abstracts

I hope that helped. There is a work in progress
"Best Practices" being worked on here:
https://wiki.galaxyproject.org/Tools/BestPractices

Regards,

Peter


___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/