Workflows with Conditionals and Loops

classic Classic list List threaded Threaded
9 messages Options
| Threaded
Open this post in threaded view
|

Workflows with Conditionals and Loops

Michael E. Cotterell
Is there a way to have conditionals and loops for the various steps in a Galaxy workflow?

For example, a conditional might be used to branch to various steps based on the previous steps. Also, a step may need to be repeated until it produces a particular output.

Here is an actual use case: We are invoking a web service in one of our workflows. The first step returns a unique identifier that you need to use later in order to get the actual results (since the web service operation is asynchronous). Later, using another operation, you can check on the status of the job being performed. Sometimes, this operation/step will inform us that the results are not ready. We would like to be able to loop on that second operation/step until it says the job is done, and then move on to a third operation/step to actually get the results.

Any ideas?

Sincerely,
Michael E. Cotterell

Ph.D. Student in Computer Science, University of Georgia
Instructor of Record, Graduate RA & TA, University of Georgia
Department Liaison, CS Graduate Student Association, University of Georgia
[hidden email] (mailto:[hidden email])
[hidden email] (mailto:[hidden email])
[hidden email] (mailto:[hidden email])
http://michaelcotterell.com/


___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
| Threaded
Open this post in threaded view
|

Re: Workflows with Conditionals and Loops

John Chilton-4
Hey Michael,

I know there is a lot of interest in this, but these sorts of things
are not possible with Galaxy workflows at this time. If you want the
traceibility and UI for viewing results provided by Galaxy - you could
consider driving tools externally via the API - at that point though
the utility of running these jobs through Galaxy has dropped
considerably.

There is a Trello card I would encourage you to vote for -
https://trello.com/c/rwSeAwos

I have included a link to this e-mail as a reference for specific use
cases to consider if or when this idea gets worked on.

Sorry,
-John


On Wed, Mar 5, 2014 at 9:05 PM, Michael E. Cotterell
<[hidden email]> wrote:

> Is there a way to have conditionals and loops for the various steps in a Galaxy workflow?
>
> For example, a conditional might be used to branch to various steps based on the previous steps. Also, a step may need to be repeated until it produces a particular output.
>
> Here is an actual use case: We are invoking a web service in one of our workflows. The first step returns a unique identifier that you need to use later in order to get the actual results (since the web service operation is asynchronous). Later, using another operation, you can check on the status of the job being performed. Sometimes, this operation/step will inform us that the results are not ready. We would like to be able to loop on that second operation/step until it says the job is done, and then move on to a third operation/step to actually get the results.
>
> Any ideas?
>
> Sincerely,
> Michael E. Cotterell
>
> Ph.D. Student in Computer Science, University of Georgia
> Instructor of Record, Graduate RA & TA, University of Georgia
> Department Liaison, CS Graduate Student Association, University of Georgia
> [hidden email] (mailto:[hidden email])
> [hidden email] (mailto:[hidden email])
> [hidden email] (mailto:[hidden email])
> http://michaelcotterell.com/
>
>
> ___________________________________________________________
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>   http://lists.bx.psu.edu/
>
> To search Galaxy mailing lists use the unified search at:
>   http://galaxyproject.org/search/mailinglists/

___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
| Threaded
Open this post in threaded view
|

Re: Workflows with Conditionals and Loops

Eric Rasche
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hi Michael,

I, too, would love to have conditionals and loops, however may want to
consider the following approach:

Instead of splitting the web service submission, check for results, and
download of results into three steps, make it a single script.

I believe this would solve your problem completely. It largely simplify
your use case, as a single script could

- - submit and remember a job's ID
- - check periodically using that language's web bindings
- - collect the results when done

This would mean it's a long-running task, which is perfectly suited to
use in galaxy.

Cheers,
Eric

On 03/06/2014 07:51 AM, John Chilton wrote:

> Hey Michael,
>
> I know there is a lot of interest in this, but these sorts of things
> are not possible with Galaxy workflows at this time. If you want the
> traceibility and UI for viewing results provided by Galaxy - you could
> consider driving tools externally via the API - at that point though
> the utility of running these jobs through Galaxy has dropped
> considerably.
>
> There is a Trello card I would encourage you to vote for -
> https://trello.com/c/rwSeAwos
>
> I have included a link to this e-mail as a reference for specific use
> cases to consider if or when this idea gets worked on.
>
> Sorry,
> -John
>
>
> On Wed, Mar 5, 2014 at 9:05 PM, Michael E. Cotterell
> <[hidden email]> wrote:
>> Is there a way to have conditionals and loops for the various steps in a Galaxy workflow?
>>
>> For example, a conditional might be used to branch to various steps based on the previous steps. Also, a step may need to be repeated until it produces a particular output.
>>
>> Here is an actual use case: We are invoking a web service in one of our workflows. The first step returns a unique identifier that you need to use later in order to get the actual results (since the web service operation is asynchronous). Later, using another operation, you can check on the status of the job being performed. Sometimes, this operation/step will inform us that the results are not ready. We would like to be able to loop on that second operation/step until it says the job is done, and then move on to a third operation/step to actually get the results.
>>
>> Any ideas?
>>
>> Sincerely,
>> Michael E. Cotterell
>>
>> Ph.D. Student in Computer Science, University of Georgia
>> Instructor of Record, Graduate RA & TA, University of Georgia
>> Department Liaison, CS Graduate Student Association, University of Georgia
>> [hidden email] (mailto:[hidden email])
>> [hidden email] (mailto:[hidden email])
>> [hidden email] (mailto:[hidden email])
>> http://michaelcotterell.com/
>>
>>
>> ___________________________________________________________
>> Please keep all replies on the list by using "reply all"
>> in your mail client.  To manage your subscriptions to this
>> and other Galaxy lists, please use the interface at:
>>   http://lists.bx.psu.edu/
>>
>> To search Galaxy mailing lists use the unified search at:
>>   http://galaxyproject.org/search/mailinglists/
>
> ___________________________________________________________
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>   http://lists.bx.psu.edu/
>
> To search Galaxy mailing lists use the unified search at:
>   http://galaxyproject.org/search/mailinglists/
>

- --
Eric Rasche
Programmer II
Center for Phage Technology
Texas A&M University
College Station, TX 77843
404-692-2048
[hidden email]
[hidden email]
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.17 (GNU/Linux)

iQIcBAEBAgAGBQJTGIFFAAoJEMqDXdrsMcpV6hsQAKGv/W0dtZXAiuf855D2PU8k
cbyMxliigbvlg0ieZNb0ZFAsx5LHRuek8sU0HMf0v14wxgs6vlwDtp143x5R0MIY
zjhNQkj+C9IXyq/qjrqT8IObbMUGgPFJD/hzrP9YCT7gUKE9Dy1fxN2aznUJgJJU
fmMD48E0/+7/y4dwdRaZ/LQv9jAKHKg1S6df6rYYWa1/hZtWrAGYrycygcrnt0aW
Ai0iE6IXDTbPYZSjNcWLMPJ7FPU/MGdIF1Lia1PB9GLAqwY5Iln6WJ6OUdahNQzw
Yx+0Te80r6wZVKHb5Bx8bszOEfPzxHcR0XABGn6r6zREoFD1kTQqugWyn0FL/v0C
zibHMbFje+LDgUU47QoyNRGXcM6RoiVJGo7KTcDbP5HEWqQ7ozmZSN2ECeX1s1gf
mzE9sJadLH3P8+vA6wO+xKtPpEzvC8VJHFROZNPJNVpVFAKU1RBRSo8qD1ZDZtTx
LnwXmx0lK0spaJXXGg8g8tO23ilwy0JXRqxt56oPp7MlUtDJ7eDyQtmdQlPlNY4v
+eqeJlGoRBlT3AS7CMfY4PWhrg3WZ0dLS+w/bG7Pg4uOj+iZ1qVEHDS4ugwqszNK
eksGuXb/Uc9GGwB1xUssF9XMwf6u/xK9OqCuW8rgzFhm85BhqEJFqiJO408Ibx/S
HPnswg0jhak5c8DKlajS
=TirM
-----END PGP SIGNATURE-----
___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
| Threaded
Open this post in threaded view
|

Re: Workflows with Conditionals and Loops

Michael E. Cotterell
While I agree that would work, the tool I’m working with generates tools for web operations in a generic fashion. That is, you provide it a WDSL and a list of operations you want from that WSDL, and then tool XML files are generated for each of those operations.  

Sincerely,
Michael E. Cotterell

Ph.D. Student in Computer Science, University of Georgia
Instructor of Record, Graduate RA & TA, University of Georgia
Department Liaison, CS Graduate Student Association, University of Georgia
[hidden email] (mailto:[hidden email])
[hidden email] (mailto:[hidden email])
[hidden email] (mailto:[hidden email])
http://michaelcotterell.com/


On Thursday, March 6, 2014 at 9:08 AM, Eric Rasche wrote:

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>  
> Hi Michael,
>  
> I, too, would love to have conditionals and loops, however may want to
> consider the following approach:
>  
> Instead of splitting the web service submission, check for results, and
> download of results into three steps, make it a single script.
>  
> I believe this would solve your problem completely. It largely simplify
> your use case, as a single script could
>  
> - - submit and remember a job's ID
> - - check periodically using that language's web bindings
> - - collect the results when done
>  
> This would mean it's a long-running task, which is perfectly suited to
> use in galaxy.
>  
> Cheers,
> Eric
>  
> On 03/06/2014 07:51 AM, John Chilton wrote:
> > Hey Michael,
> >  
> > I know there is a lot of interest in this, but these sorts of things
> > are not possible with Galaxy workflows at this time. If you want the
> > traceibility and UI for viewing results provided by Galaxy - you could
> > consider driving tools externally via the API - at that point though
> > the utility of running these jobs through Galaxy has dropped
> > considerably.
> >  
> > There is a Trello card I would encourage you to vote for -
> > https://trello.com/c/rwSeAwos
> >  
> > I have included a link to this e-mail as a reference for specific use
> > cases to consider if or when this idea gets worked on.
> >  
> > Sorry,
> > -John
> >  
> >  
> > On Wed, Mar 5, 2014 at 9:05 PM, Michael E. Cotterell
> > <[hidden email] (mailto:[hidden email])> wrote:
> > > Is there a way to have conditionals and loops for the various steps in a Galaxy workflow?
> > >  
> > > For example, a conditional might be used to branch to various steps based on the previous steps. Also, a step may need to be repeated until it produces a particular output.
> > >  
> > > Here is an actual use case: We are invoking a web service in one of our workflows. The first step returns a unique identifier that you need to use later in order to get the actual results (since the web service operation is asynchronous). Later, using another operation, you can check on the status of the job being performed. Sometimes, this operation/step will inform us that the results are not ready. We would like to be able to loop on that second operation/step until it says the job is done, and then move on to a third operation/step to actually get the results.
> > >  
> > > Any ideas?
> > >  
> > > Sincerely,
> > > Michael E. Cotterell
> > >  
> > > Ph.D. Student in Computer Science, University of Georgia
> > > Instructor of Record, Graduate RA & TA, University of Georgia
> > > Department Liaison, CS Graduate Student Association, University of Georgia
> > > [hidden email] (mailto:[hidden email])
> > > [hidden email] (mailto:[hidden email])
> > > [hidden email] (mailto:[hidden email])
> > > http://michaelcotterell.com/
> > >  
> > >  
> > > ___________________________________________________________
> > > Please keep all replies on the list by using "reply all"
> > > in your mail client. To manage your subscriptions to this
> > > and other Galaxy lists, please use the interface at:
> > > http://lists.bx.psu.edu/
> > >  
> > > To search Galaxy mailing lists use the unified search at:
> > > http://galaxyproject.org/search/mailinglists/
> >  
> >  
> >  
> > ___________________________________________________________
> > Please keep all replies on the list by using "reply all"
> > in your mail client. To manage your subscriptions to this
> > and other Galaxy lists, please use the interface at:
> > http://lists.bx.psu.edu/
> >  
> > To search Galaxy mailing lists use the unified search at:
> > http://galaxyproject.org/search/mailinglists/
>  
>  
>  
> - --  
> Eric Rasche
> Programmer II
> Center for Phage Technology
> Texas A&M University
> College Station, TX 77843
> 404-692-2048
> [hidden email] (mailto:[hidden email])
> [hidden email] (mailto:[hidden email])
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v2.0.17 (GNU/Linux)
>  
> iQIcBAEBAgAGBQJTGIFFAAoJEMqDXdrsMcpV6hsQAKGv/W0dtZXAiuf855D2PU8k
> cbyMxliigbvlg0ieZNb0ZFAsx5LHRuek8sU0HMf0v14wxgs6vlwDtp143x5R0MIY
> zjhNQkj+C9IXyq/qjrqT8IObbMUGgPFJD/hzrP9YCT7gUKE9Dy1fxN2aznUJgJJU
> fmMD48E0/+7/y4dwdRaZ/LQv9jAKHKg1S6df6rYYWa1/hZtWrAGYrycygcrnt0aW
> Ai0iE6IXDTbPYZSjNcWLMPJ7FPU/MGdIF1Lia1PB9GLAqwY5Iln6WJ6OUdahNQzw
> Yx+0Te80r6wZVKHb5Bx8bszOEfPzxHcR0XABGn6r6zREoFD1kTQqugWyn0FL/v0C
> zibHMbFje+LDgUU47QoyNRGXcM6RoiVJGo7KTcDbP5HEWqQ7ozmZSN2ECeX1s1gf
> mzE9sJadLH3P8+vA6wO+xKtPpEzvC8VJHFROZNPJNVpVFAKU1RBRSo8qD1ZDZtTx
> LnwXmx0lK0spaJXXGg8g8tO23ilwy0JXRqxt56oPp7MlUtDJ7eDyQtmdQlPlNY4v
> +eqeJlGoRBlT3AS7CMfY4PWhrg3WZ0dLS+w/bG7Pg4uOj+iZ1qVEHDS4ugwqszNK
> eksGuXb/Uc9GGwB1xUssF9XMwf6u/xK9OqCuW8rgzFhm85BhqEJFqiJO408Ibx/S
> HPnswg0jhak5c8DKlajS
> =TirM
> -----END PGP SIGNATURE-----
> ___________________________________________________________
> Please keep all replies on the list by using "reply all"
> in your mail client. To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
> http://lists.bx.psu.edu/
>  
> To search Galaxy mailing lists use the unified search at:
> http://galaxyproject.org/search/mailinglists/




___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
| Threaded
Open this post in threaded view
|

Re: Workflows with Conditionals and Loops

Peter Cock
On Thu, Mar 6, 2014 at 2:12 PM, Michael E. Cotterell
<[hidden email]> wrote:
> While I agree that would work, the tool I'm working with generates
> tools for web operations in a generic fashion. That is, you provide
> it a WDSL and a list of operations you want from that WSDL, and
> then tool XML files are generated for each of those operations.
>

So could your "WDSL Galaxy Tool Factory" also produce a wrapper
script to go with the Galaxy Tool XML, where the wrapper script
handles polling the service with the unique identifier assigned by
the service?

Peter
___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
| Threaded
Open this post in threaded view
|

Re: Workflows with Conditionals and Loops

Michael E. Cotterell
The tool is generic. This means that it is supposed to work (and does work) with any WSDL. In the described use case, one of the operations/“generated tools" already serves as a polling tool: it takes a unique id and requests a result from the web service, then the web service either returns the results or returns a message indicating that the results are not ready. Conditionals/Loops would be a perfect way to solve this.


If we added a polling tool like you described, then that tool would need to Web service specific since not all web service adhere to the same naming conventions and level of asynchrony.  

Sincerely,
Michael E. Cotterell

Ph.D. Student in Computer Science, University of Georgia
Instructor of Record, Graduate RA & TA, University of Georgia
Department Liaison, CS Graduate Student Association, University of Georgia
[hidden email] (mailto:[hidden email])
[hidden email] (mailto:[hidden email])
[hidden email] (mailto:[hidden email])
http://michaelcotterell.com/


On Thursday, March 6, 2014 at 9:17 AM, Peter Cock wrote:

> On Thu, Mar 6, 2014 at 2:12 PM, Michael E. Cotterell
> <[hidden email] (mailto:[hidden email])> wrote:
> > While I agree that would work, the tool I'm working with generates
> > tools for web operations in a generic fashion. That is, you provide
> > it a WDSL and a list of operations you want from that WSDL, and
> > then tool XML files are generated for each of those operations.
>  
>  
>  
> So could your "WDSL Galaxy Tool Factory" also produce a wrapper
> script to go with the Galaxy Tool XML, where the wrapper script
> handles polling the service with the unique identifier assigned by
> the service?
>  
> Peter  



___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
| Threaded
Open this post in threaded view
|

Re: Workflows with Conditionals and Loops

Eric Rasche
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

We're not suggesting adding a "polling tool"; instead the suggestion was
that your "WSDL tool generator" generate the XML tool definition and a
wrapper script.

That wrapper script would handle whatever WSDL operation it was created
for, polling, and collection of results (ALL IN ONE). Since polling is a
WSDL specific operation as you say, this would solve that, as every
generated tool would have code to submit+poll+collect.


Conditionals and loops would generate /many/ "empty" jobs saying "your
results are not done yet", which would not lend themselves to inclusion
in workflows. galaxy workflow loops are probably not the best way to
solve this particular problem.

Additionally, If you write your WSDL tool generator in such a way that
the generated tools handles the entire process: from submission to
retrieval of results, then they could be used in workflows and used with
the existing infrastructure, and jobs down the line could depend on the
output of your generated tools.

Cheers,
Eric

On 03/06/2014 08:25 AM, Michael E. Cotterell wrote:

> The tool is generic. This means that it is supposed to work (and does work) with any WSDL. In the described use case, one of the operations/“generated tools" already serves as a polling tool: it takes a unique id and requests a result from the web service, then the web service either returns the results or returns a message indicating that the results are not ready. Conditionals/Loops would be a perfect way to solve this.
>
>
> If we added a polling tool like you described, then that tool would need to Web service specific since not all web service adhere to the same naming conventions and level of asynchrony.  
>
> Sincerely,
> Michael E. Cotterell
>
> Ph.D. Student in Computer Science, University of Georgia
> Instructor of Record, Graduate RA & TA, University of Georgia
> Department Liaison, CS Graduate Student Association, University of Georgia
> [hidden email] (mailto:[hidden email])
> [hidden email] (mailto:[hidden email])
> [hidden email] (mailto:[hidden email])
> http://michaelcotterell.com/
>
>
> On Thursday, March 6, 2014 at 9:17 AM, Peter Cock wrote:
>
>> On Thu, Mar 6, 2014 at 2:12 PM, Michael E. Cotterell
>> <[hidden email] (mailto:[hidden email])> wrote:
>>> While I agree that would work, the tool I'm working with generates
>>> tools for web operations in a generic fashion. That is, you provide
>>> it a WDSL and a list of operations you want from that WSDL, and
>>> then tool XML files are generated for each of those operations.
>>  
>>  
>>  
>> So could your "WDSL Galaxy Tool Factory" also produce a wrapper
>> script to go with the Galaxy Tool XML, where the wrapper script
>> handles polling the service with the unique identifier assigned by
>> the service?
>>  
>> Peter  
>
>

- --
Eric Rasche
Programmer II
Center for Phage Technology
Texas A&M University
College Station, TX 77843
404-692-2048
[hidden email]
[hidden email]
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.17 (GNU/Linux)

iQIcBAEBAgAGBQJTGI2vAAoJEMqDXdrsMcpVMsAQALuigNEIfG0qqR2pxULOEK39
GYkJ9q8RFRgmQAr5STM97XoeZD+mYyqzjrsdB8XL6LldkN0+XN+Aak8I4KbQlidu
cWnZtjeY46DAxQdOcEzHCfL0apjgeOb+uCcxAtewVUBC4QMU11GkoL+54ZKoNWVV
Vitt8N0hM4WJE+MukwZ9tCjjvpW7sVM1TUfx0bBb47qyx+sxH14JiaICBYpp6jDP
Y3Kx/ncMwr62O2pMDuoP6YHn+ozKRwKa85uHEKx9mv/aQ1/0/F2Y3sd3hoj2fv4k
wxw+7GY05CBUat6d4OH3bq8K6C2Z7Ye/nQc1FClM4ssjfXUEulXgzbOwwxmoyPqw
fQPGIvWxr2YrewYsTScIO2m7kg7Y70d2gW+jRHFZJd9Yiz90ze5et5veqJ9vw7sl
v71jZkkANF8ONdHE4cIkEYBbsNuhTJvSaYOvt0YAhMe18vvBe/AwQ4AwWKcsRYhn
CnilCjICOcwJFDrRsVtAIIU/PmEl9w5tztRozHe+RtMNt4PFyn7qVdlgsFQogEQm
FWhIpTAjioyc/46kIg5vqcA0BR8KJNllCOiFDDI7ZGJJAzykPjEeZshbQcDdA0Do
Axfrr1ata/iHZsIuXsMQrVfiY8lr5XPimdXAdMPxdt+jThUP803lwDIgu9gD7wWm
LBL7haO0S65UDiZ4Be8m
=SvuS
-----END PGP SIGNATURE-----
___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
| Threaded
Open this post in threaded view
|

Re: Workflows with Conditionals and Loops

Michael E. Cotterell
I’m sorry for the confusion. These really are three different operations that handled by three different web service operations with the web service. Other WSDLs might have similar operations, but the process might be different. There is no generic way to do what you just described.  

Sincerely,
Michael E. Cotterell

Ph.D. Student in Computer Science, University of Georgia
Instructor of Record, Graduate RA & TA, University of Georgia
Department Liaison, CS Graduate Student Association, University of Georgia
[hidden email] (mailto:[hidden email])
[hidden email] (mailto:[hidden email])
[hidden email] (mailto:[hidden email])
http://michaelcotterell.com/


On Thursday, March 6, 2014 at 10:01 AM, Eric Rasche wrote:

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>  
> We're not suggesting adding a "polling tool"; instead the suggestion was
> that your "WSDL tool generator" generate the XML tool definition and a
> wrapper script.
>  
> That wrapper script would handle whatever WSDL operation it was created
> for, polling, and collection of results (ALL IN ONE). Since polling is a
> WSDL specific operation as you say, this would solve that, as every
> generated tool would have code to submit+poll+collect.
>  
>  
> Conditionals and loops would generate /many/ "empty" jobs saying "your
> results are not done yet", which would not lend themselves to inclusion
> in workflows. galaxy workflow loops are probably not the best way to
> solve this particular problem.
>  
> Additionally, If you write your WSDL tool generator in such a way that
> the generated tools handles the entire process: from submission to
> retrieval of results, then they could be used in workflows and used with
> the existing infrastructure, and jobs down the line could depend on the
> output of your generated tools.
>  
> Cheers,
> Eric
>  
> On 03/06/2014 08:25 AM, Michael E. Cotterell wrote:
> > The tool is generic. This means that it is supposed to work (and does work) with any WSDL. In the described use case, one of the operations/“generated tools" already serves as a polling tool: it takes a unique id and requests a result from the web service, then the web service either returns the results or returns a message indicating that the results are not ready. Conditionals/Loops would be a perfect way to solve this.
> >  
> >  
> > If we added a polling tool like you described, then that tool would need to Web service specific since not all web service adhere to the same naming conventions and level of asynchrony.  
> >  
> > Sincerely,
> > Michael E. Cotterell
> >  
> > Ph.D. Student in Computer Science, University of Georgia
> > Instructor of Record, Graduate RA & TA, University of Georgia
> > Department Liaison, CS Graduate Student Association, University of Georgia
> > [hidden email] (mailto:[hidden email])
> > [hidden email] (mailto:[hidden email])
> > [hidden email] (mailto:[hidden email])
> > http://michaelcotterell.com/
> >  
> >  
> > On Thursday, March 6, 2014 at 9:17 AM, Peter Cock wrote:
> >  
> > > On Thu, Mar 6, 2014 at 2:12 PM, Michael E. Cotterell
> > > <[hidden email] (mailto:[hidden email])> wrote:
> > > > While I agree that would work, the tool I'm working with generates
> > > > tools for web operations in a generic fashion. That is, you provide
> > > > it a WDSL and a list of operations you want from that WSDL, and
> > > > then tool XML files are generated for each of those operations.
> > >  
> > >  
> > >  
> > >  
> > >  
> > > So could your "WDSL Galaxy Tool Factory" also produce a wrapper
> > > script to go with the Galaxy Tool XML, where the wrapper script
> > > handles polling the service with the unique identifier assigned by
> > > the service?
> > >  
> > > Peter  
>  
>  
> - --  
> Eric Rasche
> Programmer II
> Center for Phage Technology
> Texas A&M University
> College Station, TX 77843
> 404-692-2048
> [hidden email] (mailto:[hidden email])
> [hidden email] (mailto:[hidden email])
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v2.0.17 (GNU/Linux)
>  
> iQIcBAEBAgAGBQJTGI2vAAoJEMqDXdrsMcpVMsAQALuigNEIfG0qqR2pxULOEK39
> GYkJ9q8RFRgmQAr5STM97XoeZD+mYyqzjrsdB8XL6LldkN0+XN+Aak8I4KbQlidu
> cWnZtjeY46DAxQdOcEzHCfL0apjgeOb+uCcxAtewVUBC4QMU11GkoL+54ZKoNWVV
> Vitt8N0hM4WJE+MukwZ9tCjjvpW7sVM1TUfx0bBb47qyx+sxH14JiaICBYpp6jDP
> Y3Kx/ncMwr62O2pMDuoP6YHn+ozKRwKa85uHEKx9mv/aQ1/0/F2Y3sd3hoj2fv4k
> wxw+7GY05CBUat6d4OH3bq8K6C2Z7Ye/nQc1FClM4ssjfXUEulXgzbOwwxmoyPqw
> fQPGIvWxr2YrewYsTScIO2m7kg7Y70d2gW+jRHFZJd9Yiz90ze5et5veqJ9vw7sl
> v71jZkkANF8ONdHE4cIkEYBbsNuhTJvSaYOvt0YAhMe18vvBe/AwQ4AwWKcsRYhn
> CnilCjICOcwJFDrRsVtAIIU/PmEl9w5tztRozHe+RtMNt4PFyn7qVdlgsFQogEQm
> FWhIpTAjioyc/46kIg5vqcA0BR8KJNllCOiFDDI7ZGJJAzykPjEeZshbQcDdA0Do
> Axfrr1ata/iHZsIuXsMQrVfiY8lr5XPimdXAdMPxdt+jThUP803lwDIgu9gD7wWm
> LBL7haO0S65UDiZ4Be8m
> =SvuS
> -----END PGP SIGNATURE-----
> ___________________________________________________________
> Please keep all replies on the list by using "reply all"
> in your mail client. To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
> http://lists.bx.psu.edu/
>  
> To search Galaxy mailing lists use the unified search at:
> http://galaxyproject.org/search/mailinglists/




___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
| Threaded
Open this post in threaded view
|

Re: Workflows with Conditionals and Loops

Aaron Petkau
In reply to this post by Michael E. Cotterell
I remember this being mentioned in a project called Tavaxy http://www.biomedcentral.com/1471-2105/13/77, but it's a separate piece of software (using Galaxy) and I don't know if it's been updated in a while so it may not be as useful for you.  Not sure if anybody else had any experience with it?

It would be pretty cool to have conditionals and loops within workflows though.

Aaron


On Wed, Mar 5, 2014 at 9:05 PM, Michael E. Cotterell <[hidden email]> wrote:
Is there a way to have conditionals and loops for the various steps in a Galaxy workflow?

For example, a conditional might be used to branch to various steps based on the previous steps. Also, a step may need to be repeated until it produces a particular output.

Here is an actual use case: We are invoking a web service in one of our workflows. The first step returns a unique identifier that you need to use later in order to get the actual results (since the web service operation is asynchronous). Later, using another operation, you can check on the status of the job being performed. Sometimes, this operation/step will inform us that the results are not ready. We would like to be able to loop on that second operation/step until it says the job is done, and then move on to a third operation/step to actually get the results.

Any ideas?

Sincerely,
Michael E. Cotterell

Ph.D. Student in Computer Science, University of Georgia
Instructor of Record, Graduate RA & TA, University of Georgia
Department Liaison, CS Graduate Student Association, University of Georgia
[hidden email] (mailto:[hidden email])
[hidden email] (mailto:[hidden email])
[hidden email] (mailto:[hidden email])
http://michaelcotterell.com/


___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/


___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/