Singularity Container forks ?

classic Classic list List threaded Threaded
7 messages Options
| Threaded
Open this post in threaded view
|

Singularity Container forks ?

Nikos Nikoloutsakos
Hello,

Are there any galaxy forks using Singularity images instead of Docker ?
Is it trivial enough to replace "docker run" with "singularity exec" ?

Thank you,
Nikos


--
Nikos Nikoloutsakos
Greek Research and Technology Network (GRNET)
☎ +30 210 7471127
[hidden email]
  hpc.grnet.gr

___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/
| Threaded
Open this post in threaded view
|

Re: Singularity Container forks ?

John Chilton-4
Sorry for the late response - but earlier this week Björn Grüning and I added various support to Galaxy's development branch for Singularity.

The following pull request added Singularity support in job running (https://github.com/galaxyproject/galaxy/pull/4175) - here job destinations may describe the paths to Singularity containers. It may also work with Singularity 2.3 to just specify Docker containers and let it auto-convert - I haven't tested that.

For "best practice" tools - these are tools that have only requirements that can resolve via Conda in the best practice Conda channels we can do even more - we can build Singularity containers and use them automatically - either ahead of time or build on the demand during job execution.

Here is the PR that added the on-demand building support for such Singularity containers to Galaxy https://github.com/galaxyproject/galaxy/pull/4185 and here is the tooling PR (https://github.com/galaxyproject/galaxy-lib/pull/64) and cached container PR (https://github.com/galaxyproject/galaxy/pull/4179) support. If you or anyone has a chance to try it out and beta test - let the list know how it goes. I plan to talk a little about this work and related Docker and Conda work at the GCC so I hope to see you there!

-John



On Tue, May 2, 2017 at 8:51 AM, Nikos Nikoloutsakos <[hidden email]> wrote:
Hello,

Are there any galaxy forks using Singularity images instead of Docker ?
Is it trivial enough to replace "docker run" with "singularity exec" ?

Thank you,
Nikos


--
Nikos Nikoloutsakos
Greek Research and Technology Network (GRNET)
☎ <a href="tel:%2B30%20210%207471127" value="+302107471127" target="_blank">+30 210 7471127
[hidden email]
 hpc.grnet.gr

___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
 https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
 http://galaxyproject.org/search/


___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/
| Threaded
Open this post in threaded view
|

Re: Singularity Container forks ?

Nikos Nikoloutsakos

Hi,

It's seems like you got me ! yey ! :D  I was already working on my fork for singularity ,
since we don't need any sudo rights for the galaxy user "setup manually by the admin and the paths hard-coded for each tool "

I will test it out and let you know !

Best regards from Greece,
Nikos



On 14/06/2017 07:46 μμ, John Chilton wrote:
Sorry for the late response - but earlier this week Björn Grüning and I added various support to Galaxy's development branch for Singularity.

The following pull request added Singularity support in job running (https://github.com/galaxyproject/galaxy/pull/4175) - here job destinations may describe the paths to Singularity containers. It may also work with Singularity 2.3 to just specify Docker containers and let it auto-convert - I haven't tested that.

For "best practice" tools - these are tools that have only requirements that can resolve via Conda in the best practice Conda channels we can do even more - we can build Singularity containers and use them automatically - either ahead of time or build on the demand during job execution.

Here is the PR that added the on-demand building support for such Singularity containers to Galaxy https://github.com/galaxyproject/galaxy/pull/4185 and here is the tooling PR (https://github.com/galaxyproject/galaxy-lib/pull/64) and cached container PR (https://github.com/galaxyproject/galaxy/pull/4179) support. If you or anyone has a chance to try it out and beta test - let the list know how it goes. I plan to talk a little about this work and related Docker and Conda work at the GCC so I hope to see you there!

-John



On Tue, May 2, 2017 at 8:51 AM, Nikos Nikoloutsakos <[hidden email]> wrote:
Hello,

Are there any galaxy forks using Singularity images instead of Docker ?
Is it trivial enough to replace "docker run" with "singularity exec" ?

Thank you,
Nikos


--
Nikos Nikoloutsakos
Greek Research and Technology Network (GRNET)
☎ <a href="tel:%2B30%20210%207471127" value="+302107471127" target="_blank" moz-do-not-send="true">+30 210 7471127
[hidden email]
 hpc.grnet.gr

___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
 https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
 http://galaxyproject.org/search/


-- 
Nikos Nikoloutsakos
Greek Research and Technology Network (GRNET)
☎ +30 210 7471127
✉ [hidden email]
 hpc.grnet.gr

___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/
| Threaded
Open this post in threaded view
|

Re: Singularity Container forks ?

Nikos Nikoloutsakos
In reply to this post by John Chilton-4

hi,

i have question when submitting jobs as real user

defaults = "$galaxy_root:default_ro,$tool_directory:default_ro"

singularity gives a WARNING since the home dir is already mounted "WARNING: Not mounting requested bind point (already mounted in container): default_ro"

can this be avoided ?


Thanks,




On 14/06/2017 07:46 μμ, John Chilton wrote:
Sorry for the late response - but earlier this week Björn Grüning and I added various support to Galaxy's development branch for Singularity.

The following pull request added Singularity support in job running (https://github.com/galaxyproject/galaxy/pull/4175) - here job destinations may describe the paths to Singularity containers. It may also work with Singularity 2.3 to just specify Docker containers and let it auto-convert - I haven't tested that.

For "best practice" tools - these are tools that have only requirements that can resolve via Conda in the best practice Conda channels we can do even more - we can build Singularity containers and use them automatically - either ahead of time or build on the demand during job execution.

Here is the PR that added the on-demand building support for such Singularity containers to Galaxy https://github.com/galaxyproject/galaxy/pull/4185 and here is the tooling PR (https://github.com/galaxyproject/galaxy-lib/pull/64) and cached container PR (https://github.com/galaxyproject/galaxy/pull/4179) support. If you or anyone has a chance to try it out and beta test - let the list know how it goes. I plan to talk a little about this work and related Docker and Conda work at the GCC so I hope to see you there!

-John



On Tue, May 2, 2017 at 8:51 AM, Nikos Nikoloutsakos <[hidden email]> wrote:
Hello,

Are there any galaxy forks using Singularity images instead of Docker ?
Is it trivial enough to replace "docker run" with "singularity exec" ?

Thank you,
Nikos


--
Nikos Nikoloutsakos
Greek Research and Technology Network (GRNET)
☎ <a href="tel:%2B30%20210%207471127" value="+302107471127" target="_blank" moz-do-not-send="true">+30 210 7471127
[hidden email]
 hpc.grnet.gr

___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
 https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
 http://galaxyproject.org/search/


-- 
Nikos Nikoloutsakos
Greek Research and Technology Network (GRNET)
☎ +30 210 7471127
✉ [hidden email]
 hpc.grnet.gr

___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/
| Threaded
Open this post in threaded view
|

Re: Singularity Container forks ?

Björn Grüning-3
Hi Nikos,

we are discussion various enhancement with the Singularity developers,
one is to suppress warnings.

Can you try this for the time being? This is working fine for me.

$tool_directory:rw,$working_directory:rw,$default_file_path:rw

Looking more closely to your error message it seems to be that
Singularity is interpreting the default_ro, which should not happen.
This should be replace by either ro or rw. Where do you specify this string?

Cheers,
Bjoern


> hi,
>
> i have question when submitting jobs as real user
>
> defaults = "$galaxy_root:default_ro,$tool_directory:default_ro"
>
> singularity gives a WARNING since the home dir is already mounted
> "WARNING: Not mounting requested bind point (already mounted in
> container): default_ro"
>
> can this be avoided ?
>
>
> Thanks,
>
>
>
>
> On 14/06/2017 07:46 μμ, John Chilton wrote:
>> Sorry for the late response - but earlier this week Björn Grüning and
>> I added various support to Galaxy's development branch for Singularity.
>>
>> The following pull request added Singularity support in job running
>> (https://github.com/galaxyproject/galaxy/pull/4175) - here job
>> destinations may describe the paths to Singularity containers. It may
>> also work with Singularity 2.3 to just specify Docker containers and
>> let it auto-convert - I haven't tested that.
>>
>> For "best practice" tools - these are tools that have only
>> requirements that can resolve via Conda in the best practice Conda
>> channels we can do even more - we can build Singularity containers and
>> use them automatically - either ahead of time or build on the demand
>> during job execution.
>>
>> Here is the PR that added the on-demand building support for such
>> Singularity containers to Galaxy
>> https://github.com/galaxyproject/galaxy/pull/4185 and here is the
>> tooling PR (https://github.com/galaxyproject/galaxy-lib/pull/64) and
>> cached container PR
>> (https://github.com/galaxyproject/galaxy/pull/4179) support. If you or
>> anyone has a chance to try it out and beta test - let the list know
>> how it goes. I plan to talk a little about this work and related
>> Docker and Conda work at the GCC so I hope to see you there!
>>
>> -John
>>
>>
>>
>> On Tue, May 2, 2017 at 8:51 AM, Nikos Nikoloutsakos
>> <[hidden email] <mailto:[hidden email]>> wrote:
>>
>>     Hello,
>>
>>     Are there any galaxy forks using Singularity images instead of
>>     Docker ?
>>     Is it trivial enough to replace "docker run" with "singularity exec" ?
>>
>>     Thank you,
>>     Nikos
>>
>>
>>     --
>>     Nikos Nikoloutsakos
>>     Greek Research and Technology Network (GRNET)
>>     ☎ +30 210 7471127 <tel:%2B30%20210%207471127>
>>     ✉ [hidden email] <mailto:[hidden email]>
>>      hpc.grnet.gr <http://hpc.grnet.gr>
>>
>>     ___________________________________________________________
>>     Please keep all replies on the list by using "reply all"
>>     in your mail client.  To manage your subscriptions to this
>>     and other Galaxy lists, please use the interface at:
>>      https://lists.galaxyproject.org/ <https://lists.galaxyproject.org/>
>>
>>     To search Galaxy mailing lists use the unified search at:
>>      http://galaxyproject.org/search/ <http://galaxyproject.org/search/>
>>
>>
>
> --
> Nikos Nikoloutsakos
> Greek Research and Technology Network (GRNET)
> ☎ +30 210 7471127
> ✉ [hidden email]
>  hpc.grnet.gr
>
>
>
> ___________________________________________________________
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>   https://lists.galaxyproject.org/
>
> To search Galaxy mailing lists use the unified search at:
>   http://galaxyproject.org/search/
>
___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/
| Threaded
Open this post in threaded view
|

Re: Singularity Container forks ?

Nikos Nikoloutsakos
Hi Bjoern,
i'm not setting the <param id="singularity_volumes">

it seems the default_ro is not interpreting the preprocess_volumes in class SingularityContainers

volumes_raw = self._expand_volume_str(self.destination_info.get("singularity_volumes", "$defaults"))
volumes = docker_util.DockerVolume.volumes_from_str(volumes_raw)
#preprocessed_volumes_str = preprocess_volumes(volumes_raw, self.container_type)
#volumes = docker_util.DockerVolume.volumes_from_str(preprocessed_volumes_str)




Cheers,
Nikos

On 16/06/2017 07:09 μμ, Björn Grüning wrote:
Hi Nikos,

we are discussion various enhancement with the Singularity developers,
one is to suppress warnings.

Can you try this for the time being? This is working fine for me.

$tool_directory:rw,$working_directory:rw,$default_file_path:rw

Looking more closely to your error message it seems to be that
Singularity is interpreting the default_ro, which should not happen.
This should be replace by either ro or rw. Where do you specify this string?

Cheers,
Bjoern


hi,

i have question when submitting jobs as real user

defaults = "$galaxy_root:default_ro,$tool_directory:default_ro"

singularity gives a WARNING since the home dir is already mounted
"WARNING: Not mounting requested bind point (already mounted in
container): default_ro"

can this be avoided ?


Thanks,




On 14/06/2017 07:46 μμ, John Chilton wrote:
Sorry for the late response - but earlier this week Björn Grüning and
I added various support to Galaxy's development branch for Singularity.

The following pull request added Singularity support in job running
(https://github.com/galaxyproject/galaxy/pull/4175) - here job
destinations may describe the paths to Singularity containers. It may
also work with Singularity 2.3 to just specify Docker containers and
let it auto-convert - I haven't tested that.

For "best practice" tools - these are tools that have only
requirements that can resolve via Conda in the best practice Conda
channels we can do even more - we can build Singularity containers and
use them automatically - either ahead of time or build on the demand
during job execution.

Here is the PR that added the on-demand building support for such
Singularity containers to Galaxy
https://github.com/galaxyproject/galaxy/pull/4185 and here is the
tooling PR (https://github.com/galaxyproject/galaxy-lib/pull/64) and
cached container PR
(https://github.com/galaxyproject/galaxy/pull/4179) support. If you or
anyone has a chance to try it out and beta test - let the list know
how it goes. I plan to talk a little about this work and related
Docker and Conda work at the GCC so I hope to see you there!

-John



On Tue, May 2, 2017 at 8:51 AM, Nikos Nikoloutsakos
<[hidden email] [hidden email]> wrote:

    Hello,

    Are there any galaxy forks using Singularity images instead of
    Docker ?
    Is it trivial enough to replace "docker run" with "singularity exec" ?

    Thank you,
    Nikos


    -- 
    Nikos Nikoloutsakos
    Greek Research and Technology Network (GRNET)
    ☎ +30 210 7471127 <tel:%2B30%20210%207471127>
    ✉ [hidden email] [hidden email]
     hpc.grnet.gr <http://hpc.grnet.gr>

    ___________________________________________________________
    Please keep all replies on the list by using "reply all"
    in your mail client.  To manage your subscriptions to this
    and other Galaxy lists, please use the interface at:
     https://lists.galaxyproject.org/ <https://lists.galaxyproject.org/>

    To search Galaxy mailing lists use the unified search at:
     http://galaxyproject.org/search/ <http://galaxyproject.org/search/>


-- 
Nikos Nikoloutsakos
Greek Research and Technology Network (GRNET)
☎ +30 210 7471127
✉ [hidden email]
 hpc.grnet.gr



___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/


-- 
Nikos Nikoloutsakos
Greek Research and Technology Network (GRNET)
☎ +30 210 7471127
✉ [hidden email]
 hpc.grnet.gr

___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/
| Threaded
Open this post in threaded view
|

Re: Singularity Container forks ?

John Chilton-4
Yup - I messed that up - sorry.

Can you try the fix in this PR (https://github.com/galaxyproject/galaxy/pull/4208) and comment on the PR to let me know if they fix the problem?

-John

On Mon, Jun 19, 2017 at 10:24 AM, Nikos Nikoloutsakos <[hidden email]> wrote:
Hi Bjoern,
i'm not setting the <param id="singularity_volumes">

it seems the default_ro is not interpreting the preprocess_volumes in class SingularityContainers

volumes_raw = self._expand_volume_str(self.destination_info.get("singularity_volumes", "$defaults"))
volumes = docker_util.DockerVolume.volumes_from_str(volumes_raw)
#preprocessed_volumes_str = preprocess_volumes(volumes_raw, self.container_type)
#volumes = docker_util.DockerVolume.volumes_from_str(preprocessed_volumes_str)




Cheers,
Nikos


On 16/06/2017 07:09 μμ, Björn Grüning wrote:
Hi Nikos,

we are discussion various enhancement with the Singularity developers,
one is to suppress warnings.

Can you try this for the time being? This is working fine for me.

$tool_directory:rw,$working_directory:rw,$default_file_path:rw

Looking more closely to your error message it seems to be that
Singularity is interpreting the default_ro, which should not happen.
This should be replace by either ro or rw. Where do you specify this string?

Cheers,
Bjoern


hi,

i have question when submitting jobs as real user

defaults = "$galaxy_root:default_ro,$tool_directory:default_ro"

singularity gives a WARNING since the home dir is already mounted
"WARNING: Not mounting requested bind point (already mounted in
container): default_ro"

can this be avoided ?


Thanks,




On 14/06/2017 07:46 μμ, John Chilton wrote:
Sorry for the late response - but earlier this week Björn Grüning and
I added various support to Galaxy's development branch for Singularity.

The following pull request added Singularity support in job running
(https://github.com/galaxyproject/galaxy/pull/4175) - here job
destinations may describe the paths to Singularity containers. It may
also work with Singularity 2.3 to just specify Docker containers and
let it auto-convert - I haven't tested that.

For "best practice" tools - these are tools that have only
requirements that can resolve via Conda in the best practice Conda
channels we can do even more - we can build Singularity containers and
use them automatically - either ahead of time or build on the demand
during job execution.

Here is the PR that added the on-demand building support for such
Singularity containers to Galaxy
https://github.com/galaxyproject/galaxy/pull/4185 and here is the
tooling PR (https://github.com/galaxyproject/galaxy-lib/pull/64) and
cached container PR
(https://github.com/galaxyproject/galaxy/pull/4179) support. If you or
anyone has a chance to try it out and beta test - let the list know
how it goes. I plan to talk a little about this work and related
Docker and Conda work at the GCC so I hope to see you there!

-John



On Tue, May 2, 2017 at 8:51 AM, Nikos Nikoloutsakos
<[hidden email] [hidden email]> wrote:

    Hello,

    Are there any galaxy forks using Singularity images instead of
    Docker ?
    Is it trivial enough to replace "docker run" with "singularity exec" ?

    Thank you,
    Nikos


    -- 
    Nikos Nikoloutsakos
    Greek Research and Technology Network (GRNET)
    ☎ <a href="tel:+30%2021%200747%201127" value="+302107471127" target="_blank">+30 210 7471127 <tel:%2B30%20210%207471127>
    ✉ [hidden email] [hidden email]
     hpc.grnet.gr <http://hpc.grnet.gr>

    ___________________________________________________________
    Please keep all replies on the list by using "reply all"
    in your mail client.  To manage your subscriptions to this
    and other Galaxy lists, please use the interface at:
     https://lists.galaxyproject.org/ <https://lists.galaxyproject.org/>

    To search Galaxy mailing lists use the unified search at:
     http://galaxyproject.org/search/ <http://galaxyproject.org/search/>


-- 
Nikos Nikoloutsakos
Greek Research and Technology Network (GRNET)
☎ <a href="tel:+30%2021%200747%201127" value="+302107471127" target="_blank">+30 210 7471127
✉ [hidden email]
 hpc.grnet.gr



___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/


-- 
Nikos Nikoloutsakos
Greek Research and Technology Network (GRNET)
☎ <a href="tel:+30%2021%200747%201127" value="+302107471127" target="_blank">+30 210 7471127
✉ [hidden email]
 hpc.grnet.gr


___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/