java options

classic Classic list List threaded Threaded
5 messages Options
| Threaded
Open this post in threaded view
|

java options

Matthias Bernt
Dear list,

I'm struggling to set java options for galaxy tools. Currently I use
`<env file="job_setup_script.sh"/> in my job_conf.xml and in the script
I explored two ways to set java options:

`_JAVA_OPTIONS="-Xmx5G -Xms1G -Djava.io.tmpdir=/work/songalax/tmp"`

But then java prints "Picked up _JAVA_OPTIONS: ..." to stderr which is
be interpreted as error by some tools. If I'm correct this is currently
the default .. I have seen this happening, but can't remember the name
of the tool.

So I switched to this 'trick':

`alias java='java -Xmx5G -Xms1G -Djava.io.tmpdir=/work/songalax/tmp'`

but the problem is that the tool script is called and not sourced and
therefore aliases are not used (could this be changed?). Furthermore,
for tools which explicitly set the java options my settings would be
ignored anyway (an example is the MSGFPlusAdapter which calles `java
-Xmx3500m` .. so my -Xmx is overwritten .. I passing GALAXY_MEMORY_MB to
the corresponding parameter might be a solution for this tool).

Any thought or suggestions on how to set the parameters in a production
environment are very welcome.

Cheers,
Matthias




--

-------------------------------------------
Matthias Bernt
Bioinformatics Service
Molekulare Systembiologie (MOLSYB)
Helmholtz-Zentrum für Umweltforschung GmbH - UFZ/
Helmholtz Centre for Environmental Research GmbH - UFZ
Permoserstraße 15, 04318 Leipzig, Germany
Phone +49 341 235 482296,
[hidden email], www.ufz.de

Sitz der Gesellschaft/Registered Office: Leipzig
Registergericht/Registration Office: Amtsgericht Leipzig
Handelsregister Nr./Trade Register Nr.: B 4703
Vorsitzender des Aufsichtsrats/Chairman of the Supervisory Board:
MinDirig Wilfried Kraus
Wissenschaftlicher Geschäftsführer/Scientific Managing Director:
Prof. Dr. Dr. h.c. Georg Teutsch
Administrative Geschäftsführerin/ Administrative Managing Director:
Prof. Dr. Heike Graßmann
-------------------------------------------
___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/
| Threaded
Open this post in threaded view
|

Re: java options

Marius van den Beek
Hi Matthias,

if a tool prints something to the stderr that is indeed considered to indicate a failure, and it's the default mostly because
we don't want to break legacy tools that make this assumption. If you include a profile version in your tool 
or change your command section to <command detect_errors="exit_code"></commands> this will more reasonably determine
if a tool failed based on the exit code.
To me this sounds like something that really should be fixed in the tool as presumably this affects every deployer.

Best,
Marius

On Mon, 27 Aug 2018 at 11:52, Matthias Bernt <[hidden email]> wrote:
Dear list,

I'm struggling to set java options for galaxy tools. Currently I use
`<env file="job_setup_script.sh"/> in my job_conf.xml and in the script
I explored two ways to set java options:

`_JAVA_OPTIONS="-Xmx5G -Xms1G -Djava.io.tmpdir=/work/songalax/tmp"`

But then java prints "Picked up _JAVA_OPTIONS: ..." to stderr which is
be interpreted as error by some tools. If I'm correct this is currently
the default .. I have seen this happening, but can't remember the name
of the tool.

So I switched to this 'trick':

`alias java='java -Xmx5G -Xms1G -Djava.io.tmpdir=/work/songalax/tmp'`

but the problem is that the tool script is called and not sourced and
therefore aliases are not used (could this be changed?). Furthermore,
for tools which explicitly set the java options my settings would be
ignored anyway (an example is the MSGFPlusAdapter which calles `java
-Xmx3500m` .. so my -Xmx is overwritten .. I passing GALAXY_MEMORY_MB to
the corresponding parameter might be a solution for this tool).

Any thought or suggestions on how to set the parameters in a production
environment are very welcome.

Cheers,
Matthias




--

-------------------------------------------
Matthias Bernt
Bioinformatics Service
Molekulare Systembiologie (MOLSYB)
Helmholtz-Zentrum für Umweltforschung GmbH - UFZ/
Helmholtz Centre for Environmental Research GmbH - UFZ
Permoserstraße 15, 04318 Leipzig, Germany
Phone +49 341 235 482296,
[hidden email], www.ufz.de

Sitz der Gesellschaft/Registered Office: Leipzig
Registergericht/Registration Office: Amtsgericht Leipzig
Handelsregister Nr./Trade Register Nr.: B 4703
Vorsitzender des Aufsichtsrats/Chairman of the Supervisory Board:
MinDirig Wilfried Kraus
Wissenschaftlicher Geschäftsführer/Scientific Managing Director:
Prof. Dr. Dr. h.c. Georg Teutsch
Administrative Geschäftsführerin/ Administrative Managing Director:
Prof. Dr. Heike Graßmann
-------------------------------------------
___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/

___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/
| Threaded
Open this post in threaded view
|

Re: java options

Matthias Bernt
Hi Marius,

Thanks for your comments. That's what I thought.

 > To me this sounds like something that really should be fixed in the
 > tool as presumably this affects every deployer.

Hmm. The 'tool' would be java. As far as I have seen java is
never/seldom in the tool dependencies. Hence, the system's java
installation is used for which we do not know if it is Sun/OpenJDK or
which version it is (in addition the installation will likely not change
on production systems for a long time). So I guess fixing this in java
would not help here in reasonable time anyway.

So, it seems that the best option is to:
- switching back to _JAVA_OPTIONS (which seems to be implemented also on
usegalaxy.org)
- handling the output in each java tool

Are there any guidelines for wrapping java based applications (or other
programming languages)?

- add java to dependencies? I think this should be done, or?
- if yes: is there a favored conda package?

Cheers,
Matthias

P.S. I just added java to my list of despicable programming languages.

On 27.08.2018 12:00, Marius van den Beek wrote:

> Hi Matthias,
>
> if a tool prints something to the stderr that is indeed considered to
> indicate a failure, and it's the default mostly because
> we don't want to break legacy tools that make this assumption. If you
> include a profile version in your tool
> or change your command section to <command
> detect_errors="exit_code"></commands> this will more reasonably determine
> if a tool failed based on the exit code.
>
> Best,
> Marius
>
> On Mon, 27 Aug 2018 at 11:52, Matthias Bernt <[hidden email]
> <mailto:[hidden email]>> wrote:
>
>     Dear list,
>
>     I'm struggling to set java options for galaxy tools. Currently I use
>     `<env file="job_setup_script.sh"/> in my job_conf.xml and in the script
>     I explored two ways to set java options:
>
>     `_JAVA_OPTIONS="-Xmx5G -Xms1G -Djava.io.tmpdir=/work/songalax/tmp"`
>
>     But then java prints "Picked up _JAVA_OPTIONS: ..." to stderr which is
>     be interpreted as error by some tools. If I'm correct this is currently
>     the default .. I have seen this happening, but can't remember the name
>     of the tool.
>
>     So I switched to this 'trick':
>
>     `alias java='java -Xmx5G -Xms1G -Djava.io.tmpdir=/work/songalax/tmp'`
>
>     but the problem is that the tool script is called and not sourced and
>     therefore aliases are not used (could this be changed?). Furthermore,
>     for tools which explicitly set the java options my settings would be
>     ignored anyway (an example is the MSGFPlusAdapter which calles `java
>     -Xmx3500m` .. so my -Xmx is overwritten .. I passing
>     GALAXY_MEMORY_MB to
>     the corresponding parameter might be a solution for this tool).
>
>     Any thought or suggestions on how to set the parameters in a production
>     environment are very welcome.
>
>     Cheers,
>     Matthias
>
>
>
>
>     --
>
>     -------------------------------------------
>     Matthias Bernt
>     Bioinformatics Service
>     Molekulare Systembiologie (MOLSYB)
>     Helmholtz-Zentrum für Umweltforschung GmbH - UFZ/
>     Helmholtz Centre for Environmental Research GmbH - UFZ
>     Permoserstraße 15, 04318 Leipzig, Germany
>     Phone +49 341 235 482296,
>     [hidden email] <mailto:[hidden email]>, www.ufz.de <http://www.ufz.de>
>
>     Sitz der Gesellschaft/Registered Office: Leipzig
>     Registergericht/Registration Office: Amtsgericht Leipzig
>     Handelsregister Nr./Trade Register Nr.: B 4703
>     Vorsitzender des Aufsichtsrats/Chairman of the Supervisory Board:
>     MinDirig Wilfried Kraus
>     Wissenschaftlicher Geschäftsführer/Scientific Managing Director:
>     Prof. Dr. Dr. h.c. Georg Teutsch
>     Administrative Geschäftsführerin/ Administrative Managing Director:
>     Prof. Dr. Heike Graßmann
>     -------------------------------------------
>     ___________________________________________________________
>     Please keep all replies on the list by using "reply all"
>     in your mail client.  To manage your subscriptions to this
>     and other Galaxy lists, please use the interface at:
>     https://lists.galaxyproject.org/
>
>     To search Galaxy mailing lists use the unified search at:
>     http://galaxyproject.org/search/
>

--

-------------------------------------------
Matthias Bernt
Bioinformatics Service
Molekulare Systembiologie (MOLSYB)
Helmholtz-Zentrum für Umweltforschung GmbH - UFZ/
Helmholtz Centre for Environmental Research GmbH - UFZ
Permoserstraße 15, 04318 Leipzig, Germany
Phone +49 341 235 482296,
[hidden email], www.ufz.de

Sitz der Gesellschaft/Registered Office: Leipzig
Registergericht/Registration Office: Amtsgericht Leipzig
Handelsregister Nr./Trade Register Nr.: B 4703
Vorsitzender des Aufsichtsrats/Chairman of the Supervisory Board:
MinDirig Wilfried Kraus
Wissenschaftlicher Geschäftsführer/Scientific Managing Director:
Prof. Dr. Dr. h.c. Georg Teutsch
Administrative Geschäftsführerin/ Administrative Managing Director:
Prof. Dr. Heike Graßmann
-------------------------------------------
___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/
| Threaded
Open this post in threaded view
|

Re: java options

Björn Grüning-3
Hi Matthias,

> Hi Marius,
>
> Thanks for your comments. That's what I thought.
>
>  > To me this sounds like something that really should be fixed in the
>  > tool as presumably this affects every deployer.
>
> Hmm. The 'tool' would be java. As far as I have seen java is
> never/seldom in the tool dependencies. Hence, the system's java
> installation is used for which we do not know if it is Sun/OpenJDK or
> which version it is (in addition the installation will likely not change
> on production systems for a long time). So I guess fixing this in java
> would not help here in reasonable time anyway.
>
> So, it seems that the best option is to:
> - switching back to _JAVA_OPTIONS (which seems to be implemented also on
> usegalaxy.org)
> - handling the output in each java tool

This is also what usegalaxy.eu is using and we have special destinations
for java tools.

> Are there any guidelines for wrapping java based applications (or other
> programming languages)?
>
> - add java to dependencies? I think this should be done, or?

This should be part of the dependencies of your tool already, isn't it?
So MSGFplus should depend on a java version of their choice.

See here:
https://github.com/bioconda/bioconda-recipes/blob/master/recipes/msgf_plus/meta.yaml#L15

> - if yes: is there a favored conda package?

openjdk if needed.


Ciao,
Bjoern

> Cheers,
> Matthias
>
> P.S. I just added java to my list of despicable programming languages.
>
> On 27.08.2018 12:00, Marius van den Beek wrote:
>> Hi Matthias,
>>
>> if a tool prints something to the stderr that is indeed considered to
>> indicate a failure, and it's the default mostly because
>> we don't want to break legacy tools that make this assumption. If you
>> include a profile version in your tool
>> or change your command section to <command
>> detect_errors="exit_code"></commands> this will more reasonably determine
>> if a tool failed based on the exit code.
>>
>> Best,
>> Marius
>>
>> On Mon, 27 Aug 2018 at 11:52, Matthias Bernt <[hidden email]
>> <mailto:[hidden email]>> wrote:
>>
>>     Dear list,
>>
>>     I'm struggling to set java options for galaxy tools. Currently I use
>>     `<env file="job_setup_script.sh"/> in my job_conf.xml and in the
>> script
>>     I explored two ways to set java options:
>>
>>     `_JAVA_OPTIONS="-Xmx5G -Xms1G -Djava.io.tmpdir=/work/songalax/tmp"`
>>
>>     But then java prints "Picked up _JAVA_OPTIONS: ..." to stderr
>> which is
>>     be interpreted as error by some tools. If I'm correct this is
>> currently
>>     the default .. I have seen this happening, but can't remember the
>> name
>>     of the tool.
>>
>>     So I switched to this 'trick':
>>
>>     `alias java='java -Xmx5G -Xms1G -Djava.io.tmpdir=/work/songalax/tmp'`
>>
>>     but the problem is that the tool script is called and not sourced and
>>     therefore aliases are not used (could this be changed?). Furthermore,
>>     for tools which explicitly set the java options my settings would be
>>     ignored anyway (an example is the MSGFPlusAdapter which calles `java
>>     -Xmx3500m` .. so my -Xmx is overwritten .. I passing
>>     GALAXY_MEMORY_MB to
>>     the corresponding parameter might be a solution for this tool).
>>
>>     Any thought or suggestions on how to set the parameters in a
>> production
>>     environment are very welcome.
>>
>>     Cheers,
>>     Matthias
>>
>>
>>
>>
>>     --
>>     -------------------------------------------
>>     Matthias Bernt
>>     Bioinformatics Service
>>     Molekulare Systembiologie (MOLSYB)
>>     Helmholtz-Zentrum für Umweltforschung GmbH - UFZ/
>>     Helmholtz Centre for Environmental Research GmbH - UFZ
>>     Permoserstraße 15, 04318 Leipzig, Germany
>>     Phone +49 341 235 482296,
>>     [hidden email] <mailto:[hidden email]>, www.ufz.de
>> <http://www.ufz.de>
>>
>>     Sitz der Gesellschaft/Registered Office: Leipzig
>>     Registergericht/Registration Office: Amtsgericht Leipzig
>>     Handelsregister Nr./Trade Register Nr.: B 4703
>>     Vorsitzender des Aufsichtsrats/Chairman of the Supervisory Board:
>>     MinDirig Wilfried Kraus
>>     Wissenschaftlicher Geschäftsführer/Scientific Managing Director:
>>     Prof. Dr. Dr. h.c. Georg Teutsch
>>     Administrative Geschäftsführerin/ Administrative Managing Director:
>>     Prof. Dr. Heike Graßmann
>>     -------------------------------------------
>>     ___________________________________________________________
>>     Please keep all replies on the list by using "reply all"
>>     in your mail client.  To manage your subscriptions to this
>>     and other Galaxy lists, please use the interface at:
>>     https://lists.galaxyproject.org/
>>
>>     To search Galaxy mailing lists use the unified search at:
>>     http://galaxyproject.org/search/
>>
>
___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/
| Threaded
Open this post in threaded view
|

Re: java options

Matthias Bernt
Hi Bjoern,

>> So, it seems that the best option is to:
>> - switching back to _JAVA_OPTIONS (which seems to be implemented also
>> on usegalaxy.org)
>> - handling the output in each java tool
>
> This is also what usegalaxy.eu is using and we have special destinations
> for java tools.

Indeed, this seems to be the best option.

>
>> Are there any guidelines for wrapping java based applications (or
>> other programming languages)?
>>
>> - add java to dependencies? I think this should be done, or?
>
> This should be part of the dependencies of your tool already, isn't it?
> So MSGFplus should depend on a java version of their choice.
>
> See here:
> https://github.com/bioconda/bioconda-recipes/blob/master/recipes/msgf_plus/meta.yaml#L15 

I still tend to forget that conda also has dependencies .. of course
this makes more sense than having a galaxy dependency.

>> - if yes: is there a favored conda package?
>
> openjdk if needed.

OK. Thanks also to you.

Best,
Matthias

>
>
> Ciao,
> Bjoern
>
>> Cheers,
>> Matthias
>>
>> P.S. I just added java to my list of despicable programming languages.
>>
>> On 27.08.2018 12:00, Marius van den Beek wrote:
>>> Hi Matthias,
>>>
>>> if a tool prints something to the stderr that is indeed considered to
>>> indicate a failure, and it's the default mostly because
>>> we don't want to break legacy tools that make this assumption. If you
>>> include a profile version in your tool
>>> or change your command section to <command
>>> detect_errors="exit_code"></commands> this will more reasonably
>>> determine
>>> if a tool failed based on the exit code.
>>>
>>> Best,
>>> Marius
>>>
>>> On Mon, 27 Aug 2018 at 11:52, Matthias Bernt <[hidden email]
>>> <mailto:[hidden email]>> wrote:
>>>
>>>     Dear list,
>>>
>>>     I'm struggling to set java options for galaxy tools. Currently I use
>>>     `<env file="job_setup_script.sh"/> in my job_conf.xml and in the
>>> script
>>>     I explored two ways to set java options:
>>>
>>>     `_JAVA_OPTIONS="-Xmx5G -Xms1G -Djava.io.tmpdir=/work/songalax/tmp"`
>>>
>>>     But then java prints "Picked up _JAVA_OPTIONS: ..." to stderr
>>> which is
>>>     be interpreted as error by some tools. If I'm correct this is
>>> currently
>>>     the default .. I have seen this happening, but can't remember the
>>> name
>>>     of the tool.
>>>
>>>     So I switched to this 'trick':
>>>
>>>     `alias java='java -Xmx5G -Xms1G
>>> -Djava.io.tmpdir=/work/songalax/tmp'`
>>>
>>>     but the problem is that the tool script is called and not sourced
>>> and
>>>     therefore aliases are not used (could this be changed?).
>>> Furthermore,
>>>     for tools which explicitly set the java options my settings would be
>>>     ignored anyway (an example is the MSGFPlusAdapter which calles `java
>>>     -Xmx3500m` .. so my -Xmx is overwritten .. I passing
>>>     GALAXY_MEMORY_MB to
>>>     the corresponding parameter might be a solution for this tool).
>>>
>>>     Any thought or suggestions on how to set the parameters in a
>>> production
>>>     environment are very welcome.
>>>
>>>     Cheers,
>>>     Matthias
>>>
>>>
>>>
>>>
>>>     --
>>>     -------------------------------------------
>>>     Matthias Bernt
>>>     Bioinformatics Service
>>>     Molekulare Systembiologie (MOLSYB)
>>>     Helmholtz-Zentrum für Umweltforschung GmbH - UFZ/
>>>     Helmholtz Centre for Environmental Research GmbH - UFZ
>>>     Permoserstraße 15, 04318 Leipzig, Germany
>>>     Phone +49 341 235 482296,
>>>     [hidden email] <mailto:[hidden email]>, www.ufz.de
>>> <http://www.ufz.de>
>>>
>>>     Sitz der Gesellschaft/Registered Office: Leipzig
>>>     Registergericht/Registration Office: Amtsgericht Leipzig
>>>     Handelsregister Nr./Trade Register Nr.: B 4703
>>>     Vorsitzender des Aufsichtsrats/Chairman of the Supervisory Board:
>>>     MinDirig Wilfried Kraus
>>>     Wissenschaftlicher Geschäftsführer/Scientific Managing Director:
>>>     Prof. Dr. Dr. h.c. Georg Teutsch
>>>     Administrative Geschäftsführerin/ Administrative Managing Director:
>>>     Prof. Dr. Heike Graßmann
>>>     -------------------------------------------
>>>     ___________________________________________________________
>>>     Please keep all replies on the list by using "reply all"
>>>     in your mail client.  To manage your subscriptions to this
>>>     and other Galaxy lists, please use the interface at:
>>>     https://lists.galaxyproject.org/
>>>
>>>     To search Galaxy mailing lists use the unified search at:
>>>     http://galaxyproject.org/search/
>>>
>>

--

-------------------------------------------
Matthias Bernt
Bioinformatics Service
Molekulare Systembiologie (MOLSYB)
Helmholtz-Zentrum für Umweltforschung GmbH - UFZ/
Helmholtz Centre for Environmental Research GmbH - UFZ
Permoserstraße 15, 04318 Leipzig, Germany
Phone +49 341 235 482296,
[hidden email], www.ufz.de

Sitz der Gesellschaft/Registered Office: Leipzig
Registergericht/Registration Office: Amtsgericht Leipzig
Handelsregister Nr./Trade Register Nr.: B 4703
Vorsitzender des Aufsichtsrats/Chairman of the Supervisory Board:
MinDirig Wilfried Kraus
Wissenschaftlicher Geschäftsführer/Scientific Managing Director:
Prof. Dr. Dr. h.c. Georg Teutsch
Administrative Geschäftsführerin/ Administrative Managing Director:
Prof. Dr. Heike Graßmann
-------------------------------------------
___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/