API v/s twill based testing

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

API v/s twill based testing

saketkc
I recently updated tests for one of my wrappers and came across this strange behaviour:

The twill based testing reports a failure: https://travis-ci.org/saketkc/galaxy_tools/jobs/29956682#L1463

whereas, the API based testing shows success: https://travis-ci.org/saketkc/galaxy_tools/jobs/29956683

Unfortunately I cannot run these tests locally since I am behind a system proxy [Refer: http://dev.list.galaxyproject.org/Functional-Tests-and-ftype-td4664233.html] and have to rely on travis..

The place where twii tests fails shows that  it is trying to compare the diff between 'chasm_output_genes.tabular'  and 'chasm_output_variants.tabular' instead of  'chasm_output_genes.tabular'. [See : https://travis-ci.org/saketkc/galaxy_tools/jobs/29956682#L1469]

I tried running my tools locally and I did not come across any case where the 'variants' output gets replaced by the 'genes' output, thus possibly ruling out unexpected behavior from the tool's server end.

Is this a possible bug or am I missing something?

Saket

___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
Reply | Threaded
Open this post in threaded view
|

Re: API v/s twill based testing

Peter Cock
Hi Saket,

>From memory the Twill tests are fragile with the output file order in the XML.

John was discussing switching the default from the Twill to API backend,
not sure when that is happening though...

Peter

On Tue, Jul 15, 2014 at 9:31 AM, Saket Choudhary <[hidden email]> wrote:

> I recently updated tests for one of my wrappers and came across this strange
> behaviour:
>
> The twill based testing reports a failure:
> https://travis-ci.org/saketkc/galaxy_tools/jobs/29956682#L1463
>
> whereas, the API based testing shows success:
> https://travis-ci.org/saketkc/galaxy_tools/jobs/29956683
>
> Unfortunately I cannot run these tests locally since I am behind a system
> proxy [Refer:
> http://dev.list.galaxyproject.org/Functional-Tests-and-ftype-td4664233.html]
> and have to rely on travis..
>
> The place where twii tests fails shows that  it is trying to compare the
> diff between 'chasm_output_genes.tabular'  and
> 'chasm_output_variants.tabular' instead of  'chasm_output_genes.tabular'.
> [See : https://travis-ci.org/saketkc/galaxy_tools/jobs/29956682#L1469]
>
> I tried running my tools locally and I did not come across any case where
> the 'variants' output gets replaced by the 'genes' output, thus possibly
> ruling out unexpected behavior from the tool's server end.
>
> Is this a possible bug or am I missing something?
>
> Saket
>
> ___________________________________________________________
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>   http://lists.bx.psu.edu/
>
> To search Galaxy mailing lists use the unified search at:
>   http://galaxyproject.org/search/mailinglists/
___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
Reply | Threaded
Open this post in threaded view
|

Re: API v/s twill based testing

saketkc
Thanks Peter, I guess I should then rely on API based tests.


On 15 July 2014 14:18, Peter Cock <[hidden email]> wrote:
Hi Saket,

>From memory the Twill tests are fragile with the output file order in the XML.

John was discussing switching the default from the Twill to API backend,
not sure when that is happening though...

Peter

On Tue, Jul 15, 2014 at 9:31 AM, Saket Choudhary <[hidden email]> wrote:
> I recently updated tests for one of my wrappers and came across this strange
> behaviour:
>
> The twill based testing reports a failure:
> https://travis-ci.org/saketkc/galaxy_tools/jobs/29956682#L1463
>
> whereas, the API based testing shows success:
> https://travis-ci.org/saketkc/galaxy_tools/jobs/29956683
>
> Unfortunately I cannot run these tests locally since I am behind a system
> proxy [Refer:
> http://dev.list.galaxyproject.org/Functional-Tests-and-ftype-td4664233.html]
> and have to rely on travis..
>
> The place where twii tests fails shows that  it is trying to compare the
> diff between 'chasm_output_genes.tabular'  and
> 'chasm_output_variants.tabular' instead of  'chasm_output_genes.tabular'.
> [See : https://travis-ci.org/saketkc/galaxy_tools/jobs/29956682#L1469]
>
> I tried running my tools locally and I did not come across any case where
> the 'variants' output gets replaced by the 'genes' output, thus possibly
> ruling out unexpected behavior from the tool's server end.
>
> Is this a possible bug or am I missing something?
>
> Saket
>
> ___________________________________________________________
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>   http://lists.bx.psu.edu/
>
> To search Galaxy mailing lists use the unified search at:
>   http://galaxyproject.org/search/mailinglists/


___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
Reply | Threaded
Open this post in threaded view
|

Re: API v/s twill based testing

Peter Cock
On Wed, Jul 16, 2014 at 7:44 PM, Saket Choudhary <[hidden email]> wrote:
> Thanks Peter, I guess I should then rely on API based tests.
>

If it is just the order, make sure the order of the output files in the <test>
is consistent with that in the <outputs> and it make be OK with Twill...
I wonder if I filed a Trello card on this, or just an email?

Peter
___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
Reply | Threaded
Open this post in threaded view
|

Re: API v/s twill based testing

John Chilton-4
I think the confusion over two different test methods has reached a
high level and Dave and I agreed previously that we should try to make
the API driven version the default during this release cycle - but it
is not our call :).

Greg - is there anything else you feel that needs to happen to at
least switch the test tool shed over and see how things go? Once the
main tool shed is ready to target API testing we can swap over the
default and issues like this and Aaron's earlier this week with
collections will hopefully not cause confusion.

-John


On Wed, Jul 16, 2014 at 2:13 PM, Peter Cock <[hidden email]> wrote:
> On Wed, Jul 16, 2014 at 7:44 PM, Saket Choudhary <[hidden email]> wrote:
>> Thanks Peter, I guess I should then rely on API based tests.
>>
>
> If it is just the order, make sure the order of the output files in the <test>
> is consistent with that in the <outputs> and it make be OK with Twill...
> I wonder if I filed a Trello card on this, or just an email?
>
> Peter
___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
Reply | Threaded
Open this post in threaded view
|

Re: API v/s twill based testing

Dave Bouvier
John, Peter,

The buildbot builders are already using the api interactor for both
functional tests and the install and test framework.

   --Dave B.

On Fri 18 Jul 2014 11:34:01 AM EDT, John Chilton wrote:

> I think the confusion over two different test methods has reached a
> high level and Dave and I agreed previously that we should try to make
> the API driven version the default during this release cycle - but it
> is not our call :).
>
> Greg - is there anything else you feel that needs to happen to at
> least switch the test tool shed over and see how things go? Once the
> main tool shed is ready to target API testing we can swap over the
> default and issues like this and Aaron's earlier this week with
> collections will hopefully not cause confusion.
>
> -John
>
>
> On Wed, Jul 16, 2014 at 2:13 PM, Peter Cock <[hidden email]> wrote:
>> On Wed, Jul 16, 2014 at 7:44 PM, Saket Choudhary <[hidden email]> wrote:
>>> Thanks Peter, I guess I should then rely on API based tests.
>>>
>>
>> If it is just the order, make sure the order of the output files in the <test>
>> is consistent with that in the <outputs> and it make be OK with Twill...
>> I wonder if I filed a Trello card on this, or just an email?
>>
>> Peter
> ___________________________________________________________
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>    http://lists.bx.psu.edu/
>
> To search Galaxy mailing lists use the unified search at:
>    http://galaxyproject.org/search/mailinglists/
___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
Reply | Threaded
Open this post in threaded view
|

Re: API v/s twill based testing

Peter Cock
On Fri, Jul 18, 2014 at 5:14 PM, Dave Bouvier <[hidden email]> wrote:
> John, Peter,
>
> The buildbot builders are already using the api interactor for both
> functional tests and the install and test framework.
>
>   --Dave B.

Great news.

When did that happen? Did it cause any regressions (and
can/did you flag those to the repository authors to alert them)?

Assuming that change over went smoothly, is the plan for
changing the default test back-end in the master branch of
galaxy-central (and thus eventually galaxy-stable) for those
running tool tests locally?

Thanks,

Peter
___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/