Packaging tools with language specific dependencies

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

Packaging tools with language specific dependencies

Renato Alves
Hi everyone,

Is there any standard or commonly used way to package tools that have
language specific dependencies.

I know that with Python libraries one can use setup_virtualenv and with
Java jars the JAVA_JAR_LIB strategy is used.
Is there anything equivalent for R, Perl and Ruby libraries?

Thanks
Renato




___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

signature.asc (254 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Packaging tools with language specific dependencies

Björn Grüning-3
Hi Renato,

Am 30.07.2014 um 12:21 schrieb Renato Alves:
> Hi everyone,
>
> Is there any standard or commonly used way to package tools that have
> language specific dependencies.
>
> I know that with Python libraries one can use setup_virtualenv and with
> Java jars the JAVA_JAR_LIB strategy is used.
> Is there anything equivalent for R, Perl and Ruby libraries?

please have a look at:
https://github.com/bgruening/galaxytools/tree/master/test_repositories

If you have any questions I'm happy to help you!
Bjoern

> Thanks
> Renato
>
>
>
>
>
> ___________________________________________________________
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>    http://lists.bx.psu.edu/
>
> To search Galaxy mailing lists use the unified search at:
>    http://galaxyproject.org/search/mailinglists/
>
___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
Reply | Threaded
Open this post in threaded view
|

Re: Packaging tools with language specific dependencies

Renato Alves
Hi Björn,

Thanks, this is exactly what I was looking for.

Cheers,
Renato

Quoting Björn Grüning on 30-07-2014 12:04:

> Hi Renato,
>
> Am 30.07.2014 um 12:21 schrieb Renato Alves:
>> Hi everyone,
>>
>> Is there any standard or commonly used way to package tools that have
>> language specific dependencies.
>>
>> I know that with Python libraries one can use setup_virtualenv and with
>> Java jars the JAVA_JAR_LIB strategy is used.
>> Is there anything equivalent for R, Perl and Ruby libraries?
>
> please have a look at:
> https://github.com/bgruening/galaxytools/tree/master/test_repositories
>
> If you have any questions I'm happy to help you!
> Bjoern
>
>> Thanks
>> Renato
>>
>>
>>
>>
>>
>> ___________________________________________________________
>> Please keep all replies on the list by using "reply all"
>> in your mail client.  To manage your subscriptions to this
>> and other Galaxy lists, please use the interface at:
>>    http://lists.bx.psu.edu/
>>
>> To search Galaxy mailing lists use the unified search at:
>>    http://galaxyproject.org/search/mailinglists/
>>

___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

signature.asc (254 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Packaging tools with language specific dependencies

Renato Alves
In reply to this post by Björn Grüning-3
Hi BJörn,

We have run into some problems with the perl environment.
This might also affect other tools that rely on a shebang with absolute
paths to the interpreter. Details below.

During the installation of the perl package (we tried package_perl_5_18
from main and testing toolshed), the cpanm script gets installed but
when executed it is invalid/not found.

The issue seems to be the length of the shebang in the script which is
limited to 80 characters. In cpanm it's:

!#/galaxy_dist/tool_dependencies/perl/5.18.1/bgruening/package_perl_5_18/e89824189ec6/bin/perl

That is 92 characters long. When launching the script it gets truncated
at "...perl_5_18/e898241", causing the command to fail with a "bad
interpreter" error.

I was able to workaround this problem by editing the script manually and
replacing the long shebang by: #!/usr/bin/env perl


As it stands it seems relying on the shebang to use the correct
interpreter is a problem. Repositories with long (character wise) names,
versions or owners will be more likely to suffer from this. So will
installations that are not close to the system root.

Renato

Quoting Björn Grüning on 30-07-2014 12:04:

> Hi Renato,
>
> Am 30.07.2014 um 12:21 schrieb Renato Alves:
>> Hi everyone,
>>
>> Is there any standard or commonly used way to package tools that have
>> language specific dependencies.
>>
>> I know that with Python libraries one can use setup_virtualenv and with
>> Java jars the JAVA_JAR_LIB strategy is used.
>> Is there anything equivalent for R, Perl and Ruby libraries?
>
> please have a look at:
> https://github.com/bgruening/galaxytools/tree/master/test_repositories
>
> If you have any questions I'm happy to help you!
> Bjoern
>
>> Thanks
>> Renato
>>
>>
>>
>>
>>
>> ___________________________________________________________
>> Please keep all replies on the list by using "reply all"
>> in your mail client.  To manage your subscriptions to this
>> and other Galaxy lists, please use the interface at:
>>    http://lists.bx.psu.edu/
>>
>> To search Galaxy mailing lists use the unified search at:
>>    http://galaxyproject.org/search/mailinglists/
>>

___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

signature.asc (254 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Packaging tools with language specific dependencies

Björn Grüning-3
Hi Renato,

cpanm is not the only problem it seems to be an general perl 'feature'.
How annoying :(

As I figured from:
https://metacpan.org/source/RJBS/perl-5.20.0/Configure#L9162

we can use:-Dstartperl='#!/usr/bin/env perl' to change them via install.
Now every perl script from the main perl has the correct shebang.

Unfortunately, every other package installed with cpanm or Makefile.PL
will have this annoying long shebang.

I'm not sure there is a switch in cpan/cpanm to do that. Under cpan
there are a few repositories, called change_shebang... so I assume we
are not the only ones that have this problem.

So I ended up with:

<action type="shell_command">sed -i
's|#!$INSTALL_DIR/bin/|#!/usr/bin/env |' $INSTALL_DIR/bin/*</action>

I have uploaded it to the testtoolshed and to galaxytools.
If this is working for you, also with depending repositories, I will try
to add this to into the Tool Shed core to make this sed command implicit
for all perl tools.

Cheers,
Bjoern



Am 01.08.2014 um 16:06 schrieb Renato Alves:

> Hi BJörn,
>
> We have run into some problems with the perl environment.
> This might also affect other tools that rely on a shebang with absolute
> paths to the interpreter. Details below.
>
> During the installation of the perl package (we tried package_perl_5_18
> from main and testing toolshed), the cpanm script gets installed but
> when executed it is invalid/not found.
>
> The issue seems to be the length of the shebang in the script which is
> limited to 80 characters. In cpanm it's:
>
> !#/galaxy_dist/tool_dependencies/perl/5.18.1/bgruening/package_perl_5_18/e89824189ec6/bin/perl
>
> That is 92 characters long. When launching the script it gets truncated
> at "...perl_5_18/e898241", causing the command to fail with a "bad
> interpreter" error.
>
> I was able to workaround this problem by editing the script manually and
> replacing the long shebang by: #!/usr/bin/env perl
>
>
> As it stands it seems relying on the shebang to use the correct
> interpreter is a problem. Repositories with long (character wise) names,
> versions or owners will be more likely to suffer from this. So will
> installations that are not close to the system root.
>
> Renato
>
> Quoting Björn Grüning on 30-07-2014 12:04:
>> Hi Renato,
>>
>> Am 30.07.2014 um 12:21 schrieb Renato Alves:
>>> Hi everyone,
>>>
>>> Is there any standard or commonly used way to package tools that have
>>> language specific dependencies.
>>>
>>> I know that with Python libraries one can use setup_virtualenv and with
>>> Java jars the JAVA_JAR_LIB strategy is used.
>>> Is there anything equivalent for R, Perl and Ruby libraries?
>>
>> please have a look at:
>> https://github.com/bgruening/galaxytools/tree/master/test_repositories
>>
>> If you have any questions I'm happy to help you!
>> Bjoern
>>
>>> Thanks
>>> Renato
>>>
>>>
>>>
>>>
>>>
>>> ___________________________________________________________
>>> Please keep all replies on the list by using "reply all"
>>> in your mail client.  To manage your subscriptions to this
>>> and other Galaxy lists, please use the interface at:
>>>     http://lists.bx.psu.edu/
>>>
>>> To search Galaxy mailing lists use the unified search at:
>>>     http://galaxyproject.org/search/mailinglists/
>>>
>
___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
Reply | Threaded
Open this post in threaded view
|

Re: Packaging tools with language specific dependencies

Renato Alves
Hi Bjoern,

Thanks for the info and the fix.

So far we had no issues but it's not extensive testing. I'll let you
know if it fails for some reason.

In addition to the shebang change we also started using "perl script"
instead of simply "script". Since "perl" is looked up in PATH, all else
works as expected.
It does get a bit tricky if you need to guess where the script is
located. "$(which script)" works but it's hacky. Still, it's the only
way you can "fix" the shebang without changing it before.

PS: There's shutil.which() but only python 3.3 onwards.

Cheers,
Renato

Quoting Björn Grüning on 02-08-2014 15:26:

> Hi Renato,
>
> cpanm is not the only problem it seems to be an general perl 'feature'.
> How annoying :(
>
> As I figured from:
> https://metacpan.org/source/RJBS/perl-5.20.0/Configure#L9162
>
> we can use:-Dstartperl='#!/usr/bin/env perl' to change them via install.
> Now every perl script from the main perl has the correct shebang.
>
> Unfortunately, every other package installed with cpanm or Makefile.PL
> will have this annoying long shebang.
>
> I'm not sure there is a switch in cpan/cpanm to do that. Under cpan
> there are a few repositories, called change_shebang... so I assume we
> are not the only ones that have this problem.
>
> So I ended up with:
>
> <action type="shell_command">sed -i
> 's|#!$INSTALL_DIR/bin/|#!/usr/bin/env |' $INSTALL_DIR/bin/*</action>
>
> I have uploaded it to the testtoolshed and to galaxytools.
> If this is working for you, also with depending repositories, I will try
> to add this to into the Tool Shed core to make this sed command implicit
> for all perl tools.
>
> Cheers,
> Bjoern
>
>
>
> Am 01.08.2014 um 16:06 schrieb Renato Alves:
>> Hi BJörn,
>>
>> We have run into some problems with the perl environment.
>> This might also affect other tools that rely on a shebang with absolute
>> paths to the interpreter. Details below.
>>
>> During the installation of the perl package (we tried package_perl_5_18
>> from main and testing toolshed), the cpanm script gets installed but
>> when executed it is invalid/not found.
>>
>> The issue seems to be the length of the shebang in the script which is
>> limited to 80 characters. In cpanm it's:
>>
>> !#/galaxy_dist/tool_dependencies/perl/5.18.1/bgruening/package_perl_5_18/e89824189ec6/bin/perl
>>
>>
>> That is 92 characters long. When launching the script it gets truncated
>> at "...perl_5_18/e898241", causing the command to fail with a "bad
>> interpreter" error.
>>
>> I was able to workaround this problem by editing the script manually and
>> replacing the long shebang by: #!/usr/bin/env perl
>>
>>
>> As it stands it seems relying on the shebang to use the correct
>> interpreter is a problem. Repositories with long (character wise) names,
>> versions or owners will be more likely to suffer from this. So will
>> installations that are not close to the system root.
>>
>> Renato
>>
>> Quoting Björn Grüning on 30-07-2014 12:04:
>>> Hi Renato,
>>>
>>> Am 30.07.2014 um 12:21 schrieb Renato Alves:
>>>> Hi everyone,
>>>>
>>>> Is there any standard or commonly used way to package tools that have
>>>> language specific dependencies.
>>>>
>>>> I know that with Python libraries one can use setup_virtualenv and with
>>>> Java jars the JAVA_JAR_LIB strategy is used.
>>>> Is there anything equivalent for R, Perl and Ruby libraries?
>>>
>>> please have a look at:
>>> https://github.com/bgruening/galaxytools/tree/master/test_repositories
>>>
>>> If you have any questions I'm happy to help you!
>>> Bjoern
>>>
>>>> Thanks
>>>> Renato
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> ___________________________________________________________
>>>> Please keep all replies on the list by using "reply all"
>>>> in your mail client.  To manage your subscriptions to this
>>>> and other Galaxy lists, please use the interface at:
>>>>     http://lists.bx.psu.edu/
>>>>
>>>> To search Galaxy mailing lists use the unified search at:
>>>>     http://galaxyproject.org/search/mailinglists/
>>>>
>>

___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

signature.asc (254 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Packaging tools with language specific dependencies

Björn Grüning-3
Hi Renato,

Am 05.08.2014 um 17:34 schrieb Renato Alves:

> Hi Bjoern,
>
> Thanks for the info and the fix.
>
> So far we had no issues but it's not extensive testing. I'll let you
> know if it fails for some reason.
>
> In addition to the shebang change we also started using "perl script"
> instead of simply "script". Since "perl" is looked up in PATH, all else
> works as expected.

Can you try to explain this a bit more?

> It does get a bit tricky if you need to guess where the script is
> located. "$(which script)" works but it's hacky. Still, it's the only
> way you can "fix" the shebang without changing it before.
>
> PS: There's shutil.which() but only python 3.3 onwards.

Anything I can do to support you? I'm right that the shebang
modifications work for you?

Do you think that should go into Galaxy Core, so that every perl script
under /bin/ will get a simple shebang?

Thanks,
Bjoern

> Cheers,
> Renato
>
> Quoting Björn Grüning on 02-08-2014 15:26:
>> Hi Renato,
>>
>> cpanm is not the only problem it seems to be an general perl 'feature'.
>> How annoying :(
>>
>> As I figured from:
>> https://metacpan.org/source/RJBS/perl-5.20.0/Configure#L9162
>>
>> we can use:-Dstartperl='#!/usr/bin/env perl' to change them via install.
>> Now every perl script from the main perl has the correct shebang.
>>
>> Unfortunately, every other package installed with cpanm or Makefile.PL
>> will have this annoying long shebang.
>>
>> I'm not sure there is a switch in cpan/cpanm to do that. Under cpan
>> there are a few repositories, called change_shebang... so I assume we
>> are not the only ones that have this problem.
>>
>> So I ended up with:
>>
>> <action type="shell_command">sed -i
>> 's|#!$INSTALL_DIR/bin/|#!/usr/bin/env |' $INSTALL_DIR/bin/*</action>
>>
>> I have uploaded it to the testtoolshed and to galaxytools.
>> If this is working for you, also with depending repositories, I will try
>> to add this to into the Tool Shed core to make this sed command implicit
>> for all perl tools.
>>
>> Cheers,
>> Bjoern
>>
>>
>>
>> Am 01.08.2014 um 16:06 schrieb Renato Alves:
>>> Hi BJörn,
>>>
>>> We have run into some problems with the perl environment.
>>> This might also affect other tools that rely on a shebang with absolute
>>> paths to the interpreter. Details below.
>>>
>>> During the installation of the perl package (we tried package_perl_5_18
>>> from main and testing toolshed), the cpanm script gets installed but
>>> when executed it is invalid/not found.
>>>
>>> The issue seems to be the length of the shebang in the script which is
>>> limited to 80 characters. In cpanm it's:
>>>
>>> !#/galaxy_dist/tool_dependencies/perl/5.18.1/bgruening/package_perl_5_18/e89824189ec6/bin/perl
>>>
>>>
>>> That is 92 characters long. When launching the script it gets truncated
>>> at "...perl_5_18/e898241", causing the command to fail with a "bad
>>> interpreter" error.
>>>
>>> I was able to workaround this problem by editing the script manually and
>>> replacing the long shebang by: #!/usr/bin/env perl
>>>
>>>
>>> As it stands it seems relying on the shebang to use the correct
>>> interpreter is a problem. Repositories with long (character wise) names,
>>> versions or owners will be more likely to suffer from this. So will
>>> installations that are not close to the system root.
>>>
>>> Renato
>>>
>>> Quoting Björn Grüning on 30-07-2014 12:04:
>>>> Hi Renato,
>>>>
>>>> Am 30.07.2014 um 12:21 schrieb Renato Alves:
>>>>> Hi everyone,
>>>>>
>>>>> Is there any standard or commonly used way to package tools that have
>>>>> language specific dependencies.
>>>>>
>>>>> I know that with Python libraries one can use setup_virtualenv and with
>>>>> Java jars the JAVA_JAR_LIB strategy is used.
>>>>> Is there anything equivalent for R, Perl and Ruby libraries?
>>>>
>>>> please have a look at:
>>>> https://github.com/bgruening/galaxytools/tree/master/test_repositories
>>>>
>>>> If you have any questions I'm happy to help you!
>>>> Bjoern
>>>>
>>>>> Thanks
>>>>> Renato
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> ___________________________________________________________
>>>>> Please keep all replies on the list by using "reply all"
>>>>> in your mail client.  To manage your subscriptions to this
>>>>> and other Galaxy lists, please use the interface at:
>>>>>      http://lists.bx.psu.edu/
>>>>>
>>>>> To search Galaxy mailing lists use the unified search at:
>>>>>      http://galaxyproject.org/search/mailinglists/
>>>>>
>>>
>
___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/