Galaxy unhandled exception checking jobs error

classic Classic list List threaded Threaded
5 messages Options
| Threaded
Open this post in threaded view
|

Galaxy unhandled exception checking jobs error

Ravi Alla
Hi guys,
I keep getting an error everytime I start up the galaxy server. I am guessing this has to do with a job that galaxy is trying to resume and cannot find.

galaxy.jobs.runners ERROR 2014-04-02 09:31:10,889 Unhandled exception checking active jobs
Traceback (most recent call last):
  File "/srv/www/galaxy/source/galaxy-dist/lib/galaxy/jobs/runners/__init__.py", line 366, in monitor
    self.check_watched_items()
  File "/srv/www/galaxy/source/galaxy-dist/lib/galaxy/jobs/runners/pbs.py", line 363, in check_watched_items
    ( failures, statuses ) = self.check_all_jobs()
  File "/srv/www/galaxy/source/galaxy-dist/lib/galaxy/jobs/runners/pbs.py", line 452, in check_all_jobs
    c = pbs.pbs_connect( pbs_server_name )
TypeError: in method 'pbs_connect', argument 1 of type 'char *'

Because of this error I cannot get any other jobs to run either. They just sit queued on the cluster.
Any ideas?
Thanks

___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
| Threaded
Open this post in threaded view
|

Re: Galaxy unhandled exception checking jobs error

John Chilton-4
This looks very to similar to an issue discussed here:
http://dev.list.galaxyproject.org/pbs-runner-deserializes-server-names-as-unicode-tt4663616.html.

The default branch of Galaxy contains changeset that should address
this issue - https://bitbucket.org/galaxy/galaxy-central/commits/af5577a24c155fa04aa607ff2fec283634df2fb0.

Hope this helps.

-John

On Wed, Apr 2, 2014 at 11:35 AM, Ravi Alla <[hidden email]> wrote:

> Hi guys,
> I keep getting an error everytime I start up the galaxy server. I am
> guessing this has to do with a job that galaxy is trying to resume and
> cannot find.
>
> galaxy.jobs.runners ERROR 2014-04-02 09:31:10,889 Unhandled exception
> checking active jobs
> Traceback (most recent call last):
>   File
> "/srv/www/galaxy/source/galaxy-dist/lib/galaxy/jobs/runners/__init__.py",
> line 366, in monitor
>     self.check_watched_items()
>   File "/srv/www/galaxy/source/galaxy-dist/lib/galaxy/jobs/runners/pbs.py",
> line 363, in check_watched_items
>     ( failures, statuses ) = self.check_all_jobs()
>   File "/srv/www/galaxy/source/galaxy-dist/lib/galaxy/jobs/runners/pbs.py",
> line 452, in check_all_jobs
>     c = pbs.pbs_connect( pbs_server_name )
> TypeError: in method 'pbs_connect', argument 1 of type 'char *'
>
> Because of this error I cannot get any other jobs to run either. They just
> sit queued on the cluster.
> Any ideas?
> Thanks
>
> ___________________________________________________________
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>   http://lists.bx.psu.edu/
>
> To search Galaxy mailing lists use the unified search at:
>   http://galaxyproject.org/search/mailinglists/
___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
| Threaded
Open this post in threaded view
|

Re: Galaxy unhandled exception checking jobs error

Ravi Alla
John,
Thanks for this. I am new to managing galaxy, but how do I go about updating my galaxy to reflect these changes? I found directions on https://wiki.galaxyproject.org/Admin/GetGalaxy#Keep_your_code_up_to_date about pulling changes from the bitbucket repository. Does this preserve my previous galaxy settings?
Do I have to back up before I do this? I spent considerable time to get galaxy to work on the cluster and don't want to ruin it.
Thank you
Ravi
On Apr 2, 2014, at 9:58 AM, John Chilton <[hidden email]> wrote:

> This looks very to similar to an issue discussed here:
> http://dev.list.galaxyproject.org/pbs-runner-deserializes-server-names-as-unicode-tt4663616.html.
>
> The default branch of Galaxy contains changeset that should address
> this issue - https://bitbucket.org/galaxy/galaxy-central/commits/af5577a24c155fa04aa607ff2fec283634df2fb0.
>
> Hope this helps.
>
> -John
>
> On Wed, Apr 2, 2014 at 11:35 AM, Ravi Alla <[hidden email]> wrote:
>> Hi guys,
>> I keep getting an error everytime I start up the galaxy server. I am
>> guessing this has to do with a job that galaxy is trying to resume and
>> cannot find.
>>
>> galaxy.jobs.runners ERROR 2014-04-02 09:31:10,889 Unhandled exception
>> checking active jobs
>> Traceback (most recent call last):
>>  File
>> "/srv/www/galaxy/source/galaxy-dist/lib/galaxy/jobs/runners/__init__.py",
>> line 366, in monitor
>>    self.check_watched_items()
>>  File "/srv/www/galaxy/source/galaxy-dist/lib/galaxy/jobs/runners/pbs.py",
>> line 363, in check_watched_items
>>    ( failures, statuses ) = self.check_all_jobs()
>>  File "/srv/www/galaxy/source/galaxy-dist/lib/galaxy/jobs/runners/pbs.py",
>> line 452, in check_all_jobs
>>    c = pbs.pbs_connect( pbs_server_name )
>> TypeError: in method 'pbs_connect', argument 1 of type 'char *'
>>
>> Because of this error I cannot get any other jobs to run either. They just
>> sit queued on the cluster.
>> Any ideas?
>> Thanks
>>
>> ___________________________________________________________
>> Please keep all replies on the list by using "reply all"
>> in your mail client.  To manage your subscriptions to this
>> and other Galaxy lists, please use the interface at:
>>  http://lists.bx.psu.edu/
>>
>> To search Galaxy mailing lists use the unified search at:
>>  http://galaxyproject.org/search/mailinglists/


___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
| Threaded
Open this post in threaded view
|

Re: Galaxy unhandled exception checking jobs error

John Chilton-4
This change is very small and was not committed to the stable branch
of galaxy-central so I would just modify the your Galaxy's copy of the
pbs runner file directly until the next release:

% wget https://bitbucket.org/galaxy/galaxy-central/commits/af5577a24c155fa04aa607ff2fec283634df2fb0/raw
-O /tmp/pbs.patch
% hg import --no-commit /tmp/pbs.patch

When you go to update Galaxy next it will probably warn you that this
file has been modified - at that time you can just run the following
command to cleanup your Galaxy instance:

% hg revert lib/galaxy/jobs/runners/pbs.py

Hope this helps. The Galaxy team is actively discussing alternative
ways to distribute fixes between large releases - hopefully I can stop
e-mailing out random mercurial commands at some point :).

Back to your broader question however, the Galaxy release and update
process is such that you really shouldn't lose your configuration as a
result of updating Galaxy. Galaxy distributes sample tool_conf.xml,
universe_wsgi.ini, etc... files but the actual configuration files
themselves are not tracked in the Galaxy central repositories so these
files should be unaffected by updates. Deviations from this ideal
should be rare and I believe will be spelled out the dev news for
releases as they occur. Hope this helps.

-John



On Wed, Apr 2, 2014 at 12:27 PM, Ravi Alla <[hidden email]> wrote:

> John,
> Thanks for this. I am new to managing galaxy, but how do I go about updating my galaxy to reflect these changes? I found directions on https://wiki.galaxyproject.org/Admin/GetGalaxy#Keep_your_code_up_to_date about pulling changes from the bitbucket repository. Does this preserve my previous galaxy settings?
> Do I have to back up before I do this? I spent considerable time to get galaxy to work on the cluster and don't want to ruin it.
> Thank you
> Ravi
> On Apr 2, 2014, at 9:58 AM, John Chilton <[hidden email]> wrote:
>
>> This looks very to similar to an issue discussed here:
>> http://dev.list.galaxyproject.org/pbs-runner-deserializes-server-names-as-unicode-tt4663616.html.
>>
>> The default branch of Galaxy contains changeset that should address
>> this issue - https://bitbucket.org/galaxy/galaxy-central/commits/af5577a24c155fa04aa607ff2fec283634df2fb0.
>>
>> Hope this helps.
>>
>> -John
>>
>> On Wed, Apr 2, 2014 at 11:35 AM, Ravi Alla <[hidden email]> wrote:
>>> Hi guys,
>>> I keep getting an error everytime I start up the galaxy server. I am
>>> guessing this has to do with a job that galaxy is trying to resume and
>>> cannot find.
>>>
>>> galaxy.jobs.runners ERROR 2014-04-02 09:31:10,889 Unhandled exception
>>> checking active jobs
>>> Traceback (most recent call last):
>>>  File
>>> "/srv/www/galaxy/source/galaxy-dist/lib/galaxy/jobs/runners/__init__.py",
>>> line 366, in monitor
>>>    self.check_watched_items()
>>>  File "/srv/www/galaxy/source/galaxy-dist/lib/galaxy/jobs/runners/pbs.py",
>>> line 363, in check_watched_items
>>>    ( failures, statuses ) = self.check_all_jobs()
>>>  File "/srv/www/galaxy/source/galaxy-dist/lib/galaxy/jobs/runners/pbs.py",
>>> line 452, in check_all_jobs
>>>    c = pbs.pbs_connect( pbs_server_name )
>>> TypeError: in method 'pbs_connect', argument 1 of type 'char *'
>>>
>>> Because of this error I cannot get any other jobs to run either. They just
>>> sit queued on the cluster.
>>> Any ideas?
>>> Thanks
>>>
>>> ___________________________________________________________
>>> Please keep all replies on the list by using "reply all"
>>> in your mail client.  To manage your subscriptions to this
>>> and other Galaxy lists, please use the interface at:
>>>  http://lists.bx.psu.edu/
>>>
>>> To search Galaxy mailing lists use the unified search at:
>>>  http://galaxyproject.org/search/mailinglists/
>

___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
| Threaded
Open this post in threaded view
|

Re: Galaxy unhandled exception checking jobs error

Ravi Alla
Hi John,
Thank you so much for this detailed explanation. Before I saw your email I went ahead and manually made changes in the pbs.py script. I didn't follow your mercurial commands and hope that is not a problem down the road (I guess I could use the hg revert option then).This seemed to have solved the problems I was seeing before.
Cheers
Ravi
On Apr 2, 2014, at 7:47 PM, John Chilton <[hidden email]> wrote:

> This change is very small and was not committed to the stable branch
> of galaxy-central so I would just modify the your Galaxy's copy of the
> pbs runner file directly until the next release:
>
> % wget https://bitbucket.org/galaxy/galaxy-central/commits/af5577a24c155fa04aa607ff2fec283634df2fb0/raw
> -O /tmp/pbs.patch
> % hg import --no-commit /tmp/pbs.patch
>
> When you go to update Galaxy next it will probably warn you that this
> file has been modified - at that time you can just run the following
> command to cleanup your Galaxy instance:
>
> % hg revert lib/galaxy/jobs/runners/pbs.py
>
> Hope this helps. The Galaxy team is actively discussing alternative
> ways to distribute fixes between large releases - hopefully I can stop
> e-mailing out random mercurial commands at some point :).
>
> Back to your broader question however, the Galaxy release and update
> process is such that you really shouldn't lose your configuration as a
> result of updating Galaxy. Galaxy distributes sample tool_conf.xml,
> universe_wsgi.ini, etc... files but the actual configuration files
> themselves are not tracked in the Galaxy central repositories so these
> files should be unaffected by updates. Deviations from this ideal
> should be rare and I believe will be spelled out the dev news for
> releases as they occur. Hope this helps.
>
> -John
>
>
>
> On Wed, Apr 2, 2014 at 12:27 PM, Ravi Alla <[hidden email]> wrote:
>> John,
>> Thanks for this. I am new to managing galaxy, but how do I go about updating my galaxy to reflect these changes? I found directions on https://wiki.galaxyproject.org/Admin/GetGalaxy#Keep_your_code_up_to_date about pulling changes from the bitbucket repository. Does this preserve my previous galaxy settings?
>> Do I have to back up before I do this? I spent considerable time to get galaxy to work on the cluster and don't want to ruin it.
>> Thank you
>> Ravi
>> On Apr 2, 2014, at 9:58 AM, John Chilton <[hidden email]> wrote:
>>
>>> This looks very to similar to an issue discussed here:
>>> http://dev.list.galaxyproject.org/pbs-runner-deserializes-server-names-as-unicode-tt4663616.html.
>>>
>>> The default branch of Galaxy contains changeset that should address
>>> this issue - https://bitbucket.org/galaxy/galaxy-central/commits/af5577a24c155fa04aa607ff2fec283634df2fb0.
>>>
>>> Hope this helps.
>>>
>>> -John
>>>
>>> On Wed, Apr 2, 2014 at 11:35 AM, Ravi Alla <[hidden email]> wrote:
>>>> Hi guys,
>>>> I keep getting an error everytime I start up the galaxy server. I am
>>>> guessing this has to do with a job that galaxy is trying to resume and
>>>> cannot find.
>>>>
>>>> galaxy.jobs.runners ERROR 2014-04-02 09:31:10,889 Unhandled exception
>>>> checking active jobs
>>>> Traceback (most recent call last):
>>>> File
>>>> "/srv/www/galaxy/source/galaxy-dist/lib/galaxy/jobs/runners/__init__.py",
>>>> line 366, in monitor
>>>>   self.check_watched_items()
>>>> File "/srv/www/galaxy/source/galaxy-dist/lib/galaxy/jobs/runners/pbs.py",
>>>> line 363, in check_watched_items
>>>>   ( failures, statuses ) = self.check_all_jobs()
>>>> File "/srv/www/galaxy/source/galaxy-dist/lib/galaxy/jobs/runners/pbs.py",
>>>> line 452, in check_all_jobs
>>>>   c = pbs.pbs_connect( pbs_server_name )
>>>> TypeError: in method 'pbs_connect', argument 1 of type 'char *'
>>>>
>>>> Because of this error I cannot get any other jobs to run either. They just
>>>> sit queued on the cluster.
>>>> Any ideas?
>>>> Thanks
>>>>
>>>> ___________________________________________________________
>>>> Please keep all replies on the list by using "reply all"
>>>> in your mail client.  To manage your subscriptions to this
>>>> and other Galaxy lists, please use the interface at:
>>>> http://lists.bx.psu.edu/
>>>>
>>>> To search Galaxy mailing lists use the unified search at:
>>>> http://galaxyproject.org/search/mailinglists/
>>


___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/