New Remote Job Runner

classic Classic list List threaded Threaded
2 messages Options
| Threaded
Open this post in threaded view
|

New Remote Job Runner

John Chilton
Hello,

     We are looking at integrating some tools into Galaxy for a domain
where many of the existing programs are Windows only. My desire is to
write tool files and have the certain jobs run remotely on a Windows
host (our Galaxy servers are Linux). It wasn't clear how to do this with
the existing job runners since they all seem to require a shared file
system and absolute file paths that are the same on the Galaxy server as
they are on the remote execution hosts.

     I developed a new job runner that allows for the remote execution
of jobs that also rewrites paths and stages files as part of the job
execution. There is a server application that needs to be running on the
remote machine (*nix or Windows). This application is likely easier to
setup/configure than SGE/PBS/etc... because it is a simple Python paste
application, with just a few configuration options. There may be valid
use cases for this approach for non-Windows applications as well - if
you would like a lighter weight server component on the remote system,
if shared file systems are impossible/impractical, etc....

     The path rewrites are not going to cover every possible way an
absolute path can sneak into the execution, so this is only useful for a
subset of tools. But it does handle normal file inputs and outputs,
config files, and scripts in the tool definition directory. I welcome
input/advice on this approach or the implementation. If there is any way
I can improve the code base to the point where it can be included in
galaxy-central I would be more than happy to make the desired changes.
If not, it is out there for people to use if they have a similar use case.

     The server application as well as a patch to add the runner/client
to a galaxy-central installation can be found at:

https://bitbucket.org/jmchilton/lwr/src

Thanks for your time,
John

------------------------------------------------
John Chilton
Software Developer
University of Minnesota Supercomputing Institute
Office: 612-625-0917
Cell: 612-226-9223
E-Mail: [hidden email]

| Threaded
Open this post in threaded view
|

Re: New Remote Job Runner

Nate Coraor (nate@bx.psu.edu)
Hi John,

This is fantastic, thanks for writing this.  I've committed the
Galaxy-side code in 5050:74048a9ab689, so the patch should no longer be
required.  I'll try to get something placed on the Cluster page in the
wiki, too.

Thanks,
--nate

John Chilton wrote:

> Hello,
>
>     We are looking at integrating some tools into Galaxy for a
> domain where many of the existing programs are Windows only. My
> desire is to write tool files and have the certain jobs run remotely
> on a Windows host (our Galaxy servers are Linux). It wasn't clear
> how to do this with the existing job runners since they all seem to
> require a shared file system and absolute file paths that are the
> same on the Galaxy server as they are on the remote execution hosts.
>
>     I developed a new job runner that allows for the remote
> execution of jobs that also rewrites paths and stages files as part
> of the job execution. There is a server application that needs to be
> running on the remote machine (*nix or Windows). This application is
> likely easier to setup/configure than SGE/PBS/etc... because it is a
> simple Python paste application, with just a few configuration
> options. There may be valid use cases for this approach for
> non-Windows applications as well - if you would like a lighter
> weight server component on the remote system, if shared file systems
> are impossible/impractical, etc....
>
>     The path rewrites are not going to cover every possible way an
> absolute path can sneak into the execution, so this is only useful
> for a subset of tools. But it does handle normal file inputs and
> outputs, config files, and scripts in the tool definition directory.
> I welcome input/advice on this approach or the implementation. If
> there is any way I can improve the code base to the point where it
> can be included in galaxy-central I would be more than happy to make
> the desired changes. If not, it is out there for people to use if
> they have a similar use case.
>
>     The server application as well as a patch to add the
> runner/client to a galaxy-central installation can be found at:
>
> https://bitbucket.org/jmchilton/lwr/src
>
> Thanks for your time,
> John
>
> ------------------------------------------------
> John Chilton
> Software Developer
> University of Minnesota Supercomputing Institute
> Office: 612-625-0917
> Cell: 612-226-9223
> E-Mail: [hidden email]
> _______________________________________________
> galaxy-dev mailing list
> [hidden email]
> http://lists.bx.psu.edu/listinfo/galaxy-dev