workflow startup issues

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

workflow startup issues

Geert Vandeweyer
Hi,

After the latest upgrade (april), I'm having issues when starting
(batch) workfows. Sometimes, it seems that step dependencies are not
stored correctly, leading to some steps being started too soon, without
prior datasets being completed. Specifically, for some workflows, some
of the later steps start immediately. The issue seems to be random: in
batch workflow submission, some run flawlessly, others show the issue,
at different steps. I use one handler in the job_conf to exclude race
conditions and shared variable clashes.

I suspect there is an issue with fetching or storing the steps to/from
the database (a remote mysql database in my case). On the mysql-side
there are no errors in the log files, and the galaxy-logs don't show
clear issues either.

Does the job new job handler check for failed database connections? If
not, could somebody point me to the code that performs these steps, I'd
like to add a check & retry routine to see if this improves the situation.

best,

Geert

--

Geert Vandeweyer, Ph.D.
Department of Medical Genetics
University of Antwerp
Prins Boudewijnlaan 43
2650 Edegem
Belgium
Tel: +32 (0)3 275 97 56
E-mail: [hidden email]
http://ua.ac.be/cognitivegenetics
http://www.linkedin.com/pub/geert-vandeweyer/26/457/726

___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
Reply | Threaded
Open this post in threaded view
|

Re: workflow startup issues

Dannon Baker-2
Geert,

This should be fixed in the up-to-date stable branch as far as I know.  Can you tell me what 'hg summary' states?

-Dannon


On Mon, May 20, 2013 at 4:43 PM, Geert Vandeweyer <[hidden email]> wrote:
Hi,

After the latest upgrade (april), I'm having issues when starting (batch) workfows. Sometimes, it seems that step dependencies are not stored correctly, leading to some steps being started too soon, without prior datasets being completed. Specifically, for some workflows, some of the later steps start immediately. The issue seems to be random: in batch workflow submission, some run flawlessly, others show the issue, at different steps. I use one handler in the job_conf to exclude race conditions and shared variable clashes.

I suspect there is an issue with fetching or storing the steps to/from the database (a remote mysql database in my case). On the mysql-side there are no errors in the log files, and the galaxy-logs don't show clear issues either.

Does the job new job handler check for failed database connections? If not, could somebody point me to the code that performs these steps, I'd like to add a check & retry routine to see if this improves the situation.

best,

Geert

--

Geert Vandeweyer, Ph.D.
Department of Medical Genetics
University of Antwerp
Prins Boudewijnlaan 43
2650 Edegem
Belgium
Tel: +32 (0)3 275 97 56
E-mail: [hidden email]
http://ua.ac.be/cognitivegenetics
http://www.linkedin.com/pub/geert-vandeweyer/26/457/726

___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
 http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
 http://galaxyproject.org/search/mailinglists/


___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
Reply | Threaded
Open this post in threaded view
|

Re: workflow startup issues

Geert Vandeweyer
hg summary:

parent: 9320:47ddf167c9f1 tip
 Use Galaxy's ErrorMiddleware since Paste's doesn't return start_response.  Fixes downloading tarballs from the Tool Shed when use_debug = false.
branch: stable
commit: 18 modified, 178 unknown
update: (current)


Geert

On 05/21/2013 04:59 PM, Dannon Baker wrote:
Geert,

This should be fixed in the up-to-date stable branch as far as I know.  Can you tell me what 'hg summary' states?

-Dannon


On Mon, May 20, 2013 at 4:43 PM, Geert Vandeweyer <[hidden email]> wrote:
Hi,

After the latest upgrade (april), I'm having issues when starting (batch) workfows. Sometimes, it seems that step dependencies are not stored correctly, leading to some steps being started too soon, without prior datasets being completed. Specifically, for some workflows, some of the later steps start immediately. The issue seems to be random: in batch workflow submission, some run flawlessly, others show the issue, at different steps. I use one handler in the job_conf to exclude race conditions and shared variable clashes.

I suspect there is an issue with fetching or storing the steps to/from the database (a remote mysql database in my case). On the mysql-side there are no errors in the log files, and the galaxy-logs don't show clear issues either.

Does the job new job handler check for failed database connections? If not, could somebody point me to the code that performs these steps, I'd like to add a check & retry routine to see if this improves the situation.

best,

Geert

--

Geert Vandeweyer, Ph.D.
Department of Medical Genetics
University of Antwerp
Prins Boudewijnlaan 43
2650 Edegem
Belgium
Tel: +32 (0)3 275 97 56
E-mail: [hidden email]
http://ua.ac.be/cognitivegenetics
http://www.linkedin.com/pub/geert-vandeweyer/26/457/726

___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
 http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
 http://galaxyproject.org/search/mailinglists/



-- 

Geert Vandeweyer, Ph.D.
Department of Medical Genetics
University of Antwerp
Prins Boudewijnlaan 43
2650 Edegem
Belgium
Tel: +32 (0)3 275 97 56
E-mail: [hidden email]
http://ua.ac.be/cognitivegenetics
http://www.linkedin.com/pub/geert-vandeweyer/26/457/726 

___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
Reply | Threaded
Open this post in threaded view
|

Re: workflow startup issues

Dannon Baker-2
Ok, so you're running the old tip of -dist.  There are a few new updates you can retrieve using `hg pull -u -b stable <a href="http://bitbucket.org/galaxy/galaxy-central`">http://bitbucket.org/galaxy/galaxy-central`.

-Dannon


On Tue, May 21, 2013 at 12:01 PM, Geert Vandeweyer <[hidden email]> wrote:
hg summary:

parent: 9320:47ddf167c9f1 tip
 Use Galaxy's ErrorMiddleware since Paste's doesn't return start_response.  Fixes downloading tarballs from the Tool Shed when use_debug = false.
branch: stable
commit: 18 modified, 178 unknown
update: (current)


Geert


On 05/21/2013 04:59 PM, Dannon Baker wrote:
Geert,

This should be fixed in the up-to-date stable branch as far as I know.  Can you tell me what 'hg summary' states?

-Dannon


On Mon, May 20, 2013 at 4:43 PM, Geert Vandeweyer <[hidden email]> wrote:
Hi,

After the latest upgrade (april), I'm having issues when starting (batch) workfows. Sometimes, it seems that step dependencies are not stored correctly, leading to some steps being started too soon, without prior datasets being completed. Specifically, for some workflows, some of the later steps start immediately. The issue seems to be random: in batch workflow submission, some run flawlessly, others show the issue, at different steps. I use one handler in the job_conf to exclude race conditions and shared variable clashes.

I suspect there is an issue with fetching or storing the steps to/from the database (a remote mysql database in my case). On the mysql-side there are no errors in the log files, and the galaxy-logs don't show clear issues either.

Does the job new job handler check for failed database connections? If not, could somebody point me to the code that performs these steps, I'd like to add a check & retry routine to see if this improves the situation.

best,

Geert

--

Geert Vandeweyer, Ph.D.
Department of Medical Genetics
University of Antwerp
Prins Boudewijnlaan 43
2650 Edegem
Belgium
Tel: +32 (0)3 275 97 56
E-mail: [hidden email]
http://ua.ac.be/cognitivegenetics
http://www.linkedin.com/pub/geert-vandeweyer/26/457/726

___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
 http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
 http://galaxyproject.org/search/mailinglists/



-- 

Geert Vandeweyer, Ph.D.
Department of Medical Genetics
University of Antwerp
Prins Boudewijnlaan 43
2650 Edegem
Belgium
Tel: +32 (0)3 275 97 56
E-mail: [hidden email]
http://ua.ac.be/cognitivegenetics
http://www.linkedin.com/pub/geert-vandeweyer/26/457/726 


___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
Reply | Threaded
Open this post in threaded view
|

Re: workflow startup issues

Geert Vandeweyer
Hi Dannon,

I pulled the new updates and restarted galaxy. However, the issue is not resolved. Is there other information you might need?

hg summary:

parent: 9332:0f51ae4f1c84 tip
 Unicode fixes for workflow names in Tool Menu.
branch: stable

Best,

Geert


On 05/21/2013 06:08 PM, Dannon Baker wrote:
Ok, so you're running the old tip of -dist.  There are a few new updates you can retrieve using `hg pull -u -b stable http://bitbucket.org/galaxy/galaxy-central`.

-Dannon


On Tue, May 21, 2013 at 12:01 PM, Geert Vandeweyer <[hidden email]> wrote:
hg summary:

parent: 9320:47ddf167c9f1 tip
 Use Galaxy's ErrorMiddleware since Paste's doesn't return start_response.  Fixes downloading tarballs from the Tool Shed when use_debug = false.
branch: stable
commit: 18 modified, 178 unknown
update: (current)


Geert


On 05/21/2013 04:59 PM, Dannon Baker wrote:
Geert,

This should be fixed in the up-to-date stable branch as far as I know.  Can you tell me what 'hg summary' states?

-Dannon


On Mon, May 20, 2013 at 4:43 PM, Geert Vandeweyer <[hidden email]> wrote:
Hi,

After the latest upgrade (april), I'm having issues when starting (batch) workfows. Sometimes, it seems that step dependencies are not stored correctly, leading to some steps being started too soon, without prior datasets being completed. Specifically, for some workflows, some of the later steps start immediately. The issue seems to be random: in batch workflow submission, some run flawlessly, others show the issue, at different steps. I use one handler in the job_conf to exclude race conditions and shared variable clashes.

I suspect there is an issue with fetching or storing the steps to/from the database (a remote mysql database in my case). On the mysql-side there are no errors in the log files, and the galaxy-logs don't show clear issues either.

Does the job new job handler check for failed database connections? If not, could somebody point me to the code that performs these steps, I'd like to add a check & retry routine to see if this improves the situation.

best,

Geert

--

Geert Vandeweyer, Ph.D.
Department of Medical Genetics
University of Antwerp
Prins Boudewijnlaan 43
2650 Edegem
Belgium
Tel: +32 (0)3 275 97 56
E-mail: [hidden email]
http://ua.ac.be/cognitivegenetics
http://www.linkedin.com/pub/geert-vandeweyer/26/457/726

___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
 http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
 http://galaxyproject.org/search/mailinglists/



-- 

Geert Vandeweyer, Ph.D.
Department of Medical Genetics
University of Antwerp
Prins Boudewijnlaan 43
2650 Edegem
Belgium
Tel: +32 (0)3 275 97 56
E-mail: [hidden email]
http://ua.ac.be/cognitivegenetics
http://www.linkedin.com/pub/geert-vandeweyer/26/457/726 



-- 

Geert Vandeweyer, Ph.D.
Department of Medical Genetics
University of Antwerp
Prins Boudewijnlaan 43
2650 Edegem
Belgium
Tel: +32 (0)3 275 97 56
E-mail: [hidden email]
http://ua.ac.be/cognitivegenetics
http://www.linkedin.com/pub/geert-vandeweyer/26/457/726 

___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
Reply | Threaded
Open this post in threaded view
|

Re: workflow startup issues

Geert Vandeweyer
Aside: If I add steps to the beginning of the workfow, the step that runs too early, typically occurs earlier in the flow. Can this indicate a race condition where not all dependencies are available (stored?) yet in the database at the time of querying for job eligible for queueing?

Best,

Geert

On 05/21/2013 11:06 PM, Geert Vandeweyer wrote:
Hi Dannon,

I pulled the new updates and restarted galaxy. However, the issue is not resolved. Is there other information you might need?

hg summary:

parent: 9332:0f51ae4f1c84 tip
 Unicode fixes for workflow names in Tool Menu.
branch: stable

Best,

Geert


On 05/21/2013 06:08 PM, Dannon Baker wrote:
Ok, so you're running the old tip of -dist.  There are a few new updates you can retrieve using `hg pull -u -b stable http://bitbucket.org/galaxy/galaxy-central`.

-Dannon


On Tue, May 21, 2013 at 12:01 PM, Geert Vandeweyer <[hidden email]> wrote:
hg summary:

parent: 9320:47ddf167c9f1 tip
 Use Galaxy's ErrorMiddleware since Paste's doesn't return start_response.  Fixes downloading tarballs from the Tool Shed when use_debug = false.
branch: stable
commit: 18 modified, 178 unknown
update: (current)


Geert


On 05/21/2013 04:59 PM, Dannon Baker wrote:
Geert,

This should be fixed in the up-to-date stable branch as far as I know.  Can you tell me what 'hg summary' states?

-Dannon


On Mon, May 20, 2013 at 4:43 PM, Geert Vandeweyer <[hidden email]> wrote:
Hi,

After the latest upgrade (april), I'm having issues when starting (batch) workfows. Sometimes, it seems that step dependencies are not stored correctly, leading to some steps being started too soon, without prior datasets being completed. Specifically, for some workflows, some of the later steps start immediately. The issue seems to be random: in batch workflow submission, some run flawlessly, others show the issue, at different steps. I use one handler in the job_conf to exclude race conditions and shared variable clashes.

I suspect there is an issue with fetching or storing the steps to/from the database (a remote mysql database in my case). On the mysql-side there are no errors in the log files, and the galaxy-logs don't show clear issues either.

Does the job new job handler check for failed database connections? If not, could somebody point me to the code that performs these steps, I'd like to add a check & retry routine to see if this improves the situation.

best,

Geert

--

Geert Vandeweyer, Ph.D.
Department of Medical Genetics
University of Antwerp
Prins Boudewijnlaan 43
2650 Edegem
Belgium
Tel: +32 (0)3 275 97 56
E-mail: [hidden email]
http://ua.ac.be/cognitivegenetics
http://www.linkedin.com/pub/geert-vandeweyer/26/457/726

___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
 http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
 http://galaxyproject.org/search/mailinglists/



-- 

Geert Vandeweyer, Ph.D.
Department of Medical Genetics
University of Antwerp
Prins Boudewijnlaan 43
2650 Edegem
Belgium
Tel: +32 (0)3 275 97 56
E-mail: [hidden email]
http://ua.ac.be/cognitivegenetics
http://www.linkedin.com/pub/geert-vandeweyer/26/457/726 



-- 

Geert Vandeweyer, Ph.D.
Department of Medical Genetics
University of Antwerp
Prins Boudewijnlaan 43
2650 Edegem
Belgium
Tel: +32 (0)3 275 97 56
E-mail: [hidden email]
http://ua.ac.be/cognitivegenetics
http://www.linkedin.com/pub/geert-vandeweyer/26/457/726 


___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/


-- 

Geert Vandeweyer, Ph.D.
Department of Medical Genetics
University of Antwerp
Prins Boudewijnlaan 43
2650 Edegem
Belgium
Tel: +32 (0)3 275 97 56
E-mail: [hidden email]
http://ua.ac.be/cognitivegenetics
http://www.linkedin.com/pub/geert-vandeweyer/26/457/726 

___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/