Scaling tools: data available?

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

Scaling tools: data available?

Peter van Heusden
Hi there

We're researching the resource (CPU time and memory) requirements for RNA STAR at the moment. Specifically, we'd like to build up a database of input size to resource usage so that we can use this to feed a dynamic destination mapper (currently our STAR configuration uses a thumbsuck for RAM and CPU requirements that is not optimal but works in many cases).

Has anyone collected these metrics before? And more generally, is anyone collecting metrics for a range of bioinformatics tools? A database like this could be very useful!

Thanks!
Peter

___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
Reply | Threaded
Open this post in threaded view
|

Re: Scaling tools: data available?

Eric Rasche-2
Hi peter,

I've been working on this in the past but got caught up in other projects. There's work for a server-side component here (maybe 50% done), the client side stuff could all be done through collectl but I was having issues actually collecting collectl data that had put the project on hold temporarily.

https://github.com/erasche/galactic-radio-telescope

I'd be very, very interested in finishing this project up.

Cheers,
Eric

On 02/21/2016 11:13 PM, Peter van Heusden wrote:
Hi there

We're researching the resource (CPU time and memory) requirements for RNA STAR at the moment. Specifically, we'd like to build up a database of input size to resource usage so that we can use this to feed a dynamic destination mapper (currently our STAR configuration uses a thumbsuck for RAM and CPU requirements that is not optimal but works in many cases).

Has anyone collected these metrics before? And more generally, is anyone collecting metrics for a range of bioinformatics tools? A database like this could be very useful!

Thanks!
Peter


___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

--
--
Eric Rasche
Programmer II

Center for Phage Technology
Rm 312A, BioBio
Texas A&M University
College Station, TX 77843
<a href="tel:404-692-2048">404-692-2048
[hidden email]
Not responding quickly enough?

___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
Reply | Threaded
Open this post in threaded view
|

Re: Scaling tools: data available?

Peter van Heusden
Hi Eric

I'm having a look at the GRT. We're collecting metrics on our local Galaxy server, I'm just trying to understand how you envision these metrics getting into the GRT?

Peter

On 22 February 2016 at 07:31, Eric Rasche <[hidden email]> wrote:
Hi peter,

I've been working on this in the past but got caught up in other projects. There's work for a server-side component here (maybe 50% done), the client side stuff could all be done through collectl but I was having issues actually collecting collectl data that had put the project on hold temporarily.

https://github.com/erasche/galactic-radio-telescope

I'd be very, very interested in finishing this project up.

Cheers,
Eric


On 02/21/2016 11:13 PM, Peter van Heusden wrote:
Hi there

We're researching the resource (CPU time and memory) requirements for RNA STAR at the moment. Specifically, we'd like to build up a database of input size to resource usage so that we can use this to feed a dynamic destination mapper (currently our STAR configuration uses a thumbsuck for RAM and CPU requirements that is not optimal but works in many cases).

Has anyone collected these metrics before? And more generally, is anyone collecting metrics for a range of bioinformatics tools? A database like this could be very useful!

Thanks!
Peter


___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

--
--
Eric Rasche
Programmer II

Center for Phage Technology
Rm 312A, BioBio
Texas A&M University
College Station, TX 77843
<a href="tel:404-692-2048" target="_blank">404-692-2048
[hidden email]
Not responding quickly enough?


___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
Reply | Threaded
Open this post in threaded view
|

Re: Scaling tools: data available?

Eric Rasche-2

Hi Peter,

The vision I had was a cron script admins could install which would send thr past day/week's collectl/memory/parameter/dataset size logs to GRT.

Again, the GRT side stalled when I hit issues with collectl locally. If you have time/energy to work on that we should chat :-) Otherwise I plan to find time for it before GCC, minus the collectl stuff, just capturing runtimes as a start.

Ciao,
Eric

14. mar. 2016 1.09 p.m. skrev "Peter van Heusden" <[hidden email]>:
Hi Eric

I'm having a look at the GRT. We're collecting metrics on our local Galaxy server, I'm just trying to understand how you envision these metrics getting into the GRT?

Peter

On 22 February 2016 at 07:31, Eric Rasche <[hidden email]> wrote:
Hi peter,

I've been working on this in the past but got caught up in other projects. There's work for a server-side component here (maybe 50% done), the client side stuff could all be done through collectl but I was having issues actually collecting collectl data that had put the project on hold temporarily.

https://github.com/erasche/galactic-radio-telescope

I'd be very, very interested in finishing this project up.

Cheers,
Eric


On 02/21/2016 11:13 PM, Peter van Heusden wrote:
Hi there

We're researching the resource (CPU time and memory) requirements for RNA STAR at the moment. Specifically, we'd like to build up a database of input size to resource usage so that we can use this to feed a dynamic destination mapper (currently our STAR configuration uses a thumbsuck for RAM and CPU requirements that is not optimal but works in many cases).

Has anyone collected these metrics before? And more generally, is anyone collecting metrics for a range of bioinformatics tools? A database like this could be very useful!

Thanks!
Peter


___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

--
--
Eric Rasche
Programmer II

Center for Phage Technology
Rm 312A, BioBio
Texas A&M University
College Station, TX 77843
<a href="tel:404-692-2048" target="_blank">404-692-2048
[hidden email]
Not responding quickly enough?


___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/