Project

General

Profile

Evolution #5111

Optimisation de la consommation mémoire

Added by Emmanuel GARETTE almost 7 years ago. Updated over 6 years ago.

Status:
Fermé
Priority:
Normal
Assigned To:
Category:
-
Start date:
03/25/2013
Due date:
04/05/2013
% Done:

100%

Distribution:
EOLE 2.4

Description

Il faudrait voir s'il serait possible de diminuer la consommation mémoire des objets tiramisu (utilisation des slots, ...).

Le script que j'utilise pour mes tests :

#!/usr/bin/env python

from creole.loader import CreoleVarLoader, configeol, eoledirs

all_configs = []

def load_config():
    for j in range(0, 20):
        loader = CreoleVarLoader()
        loader.read_dir(eoledirs, 'creole')
        for i in range(0, 100):
            loader._config = None
            loader.load_values('creole')
            config = loader.get_config(valid_opt_names=False)
            all_configs.append(config)

load_config()

raw_input('quitter')

Valeur constatée pour 20 variantes de 100 serveurs chacun :

USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 9427 95.6 73.6 499872 370840 pts/1 Sl+ 10:04 0:53 python /root/load_tiramisu.py

Répartition de la mémoire pour 1 variante et un serveur :

Partition of a set of 4256 objects. Total size = 892000 bytes.
 Index  Count   %     Size   % Cumulative  % Kind (class / dict of class)
     0    196   5   205408  23    205408  23 dict of tiramisu.option.UnicodeOption
     1    462  11   153168  17    358576  40 dict (no owner)
     2    977  23   115720  13    474296  53 list
     3   1493  35    96864  11    571160  64 str
     4     82   2    85936  10    657096  74 dict of tiramisu.option.ChoiceOption
     5     55   1    78376   9    735472  82 dict of tiramisu.option.OptionDescription
     6     55   1    57640   6    793112  89 dict of tiramisu.config.Config
     7    302   7    22736   3    815848  91 tuple
     8    125   3    20168   2    836016  94 unicode
     9     18   0    18864   2    854880  96 dict of tiramisu.option.IntOption
    10    196   5    12544   1    867424  97 tiramisu.option.UnicodeOption
    11     82   2     5248   1    872672  98 tiramisu.option.ChoiceOption
    12     55   1     3520   0    876192  98 tiramisu.config.Config
    13     55   1     3520   0    879712  99 tiramisu.option.OptionDescription
    14     22   1     2640   0    882352  99 function
    15     17   0     1496   0    883848  99 __builtin__.weakref
    16     18   0     1152   0    885000  99 tiramisu.option.IntOption
    17     20   0     1120   0    886120  99 __builtin__.cell
    18      1   0     1048   0    887168  99 dict of codecs.CodecInfo
    19      1   0     1048   0    888216 100 dict of tiramisu.value.Values
    20      4   0      928   0    889144 100 __builtin__.set
    21      2   0      872   0    890016 100 types.FrameType
    22      2   0      560   0    890576 100 dict of _weakrefset.WeakSet
    23      1   0      280   0    890856 100 dict of tiramisu.setting.Setting
    24      1   0      280   0    891136 100 dict of tiramisu.value.Multi
    25      1   0      152   0    891288 100 tiramisu.value.Multi
    26      6   0      144   0    891432 100 int
    27      2   0      128   0    891560 100 _weakrefset.WeakSet
    28      1   0      128   0    891688 100 types.CodeType
    29      1   0       96   0    891784 100 codecs.CodecInfo
    30      1   0       80   0    891864 100 types.MethodType
    31      1   0       72   0    891936 100 tiramisu.setting.Setting
    32      1   0       64   0    892000 100 tiramisu.value.Values

Related issues

Related to eole-genconfig - Evolution #4984: Choix Technologiques Fermé 03/04/2013 04/19/2013

Associated revisions

Revision d0015378 (diff)
Added by Emmanuel GARETTE almost 7 years ago

Le cache des paths est dans l'OptionDescription
Utilisation des slots pour la Config

Il faut 566Mo de mémoire pour charger 50 variantes de 100 serveurs chacuns (5000 serveurs en tout).

ref #5111

History

#1 Updated by Emmanuel GARETTE almost 7 years ago

Pour 20*100 :

root 11078 50.9 37.5 318004 189176 pts/1 Sl+ 16:20 0:49 python /root/load_tiramisu.py

Pour 50*100 :

root 10976 56.3 87.0 566936 438296 pts/1 Sl+ 16:02 2:11 python /root/load_tiramisu.py

Partition of a set of 4251 objects. Total size = 852808 bytes.
 Index  Count   %     Size   % Cumulative  % Kind (class / dict of class)
     0    196   5   205408  24    205408  24 dict of tiramisu.option.UnicodeOption
     1    516  12   168288  20    373696  44 dict (no owner)
     2    977  23   115720  14    489416  57 list
     3   1489  35    96672  11    586088  69 str
     4     82   2    85936  10    672024  79 dict of tiramisu.option.ChoiceOption
     5     55   1    78376   9    750400  88 dict of tiramisu.option.OptionDescription
     6    302   7    22736   3    773136  91 tuple
     7    125   3    20168   2    793304  93 unicode
     8     18   0    18864   2    812168  95 dict of tiramisu.option.IntOption
     9    196   5    12544   1    824712  97 tiramisu.option.UnicodeOption
    10     55   1     7040   1    831752  98 tiramisu.config.Config
    11     82   2     5248   1    837000  98 tiramisu.option.ChoiceOption
    12     55   1     3520   0    840520  99 tiramisu.option.OptionDescription
    13     22   1     2640   0    843160  99 function
    14     17   0     1496   0    844656  99 __builtin__.weakref
    15     18   0     1152   0    845808  99 tiramisu.option.IntOption
    16     20   0     1120   0    846928  99 __builtin__.cell
    17      1   0     1048   0    847976  99 dict of codecs.CodecInfo
    18      1   0     1048   0    849024 100 dict of tiramisu.value.Values
    19      4   0      928   0    849952 100 __builtin__.set
    20      2   0      872   0    850824 100 types.FrameType
    21      2   0      560   0    851384 100 dict of _weakrefset.WeakSet
    22      1   0      280   0    851664 100 dict of tiramisu.setting.Setting
    23      1   0      280   0    851944 100 dict of tiramisu.value.Multi
    24      1   0      152   0    852096 100 tiramisu.value.Multi
    25      6   0      144   0    852240 100 int
    26      2   0      128   0    852368 100 _weakrefset.WeakSet
    27      1   0      128   0    852496 100 types.CodeType
    28      1   0       96   0    852592 100 codecs.CodecInfo
    29      1   0       80   0    852672 100 types.MethodType
    30      1   0       72   0    852744 100 tiramisu.setting.Setting
    31      1   0       64   0    852808 100 tiramisu.value.Values

#2 Updated by Emmanuel GARETTE almost 7 years ago

Avec la dernière version :

Pour 20*100 :

root 26927 65.5 24.3 251492 122644 pts/2 Sl+ 16:24 0:43 | \_ python /root/load_tiramisu.py

Pour 50*100 :

root 26940 69.4 49.6 378948 250116 pts/2 Sl+ 16:26 2:03 | \_ python /root/load_tiramisu.py

Partition of a set of 3807 objects. Total size = 414752 bytes.
 Index  Count   %     Size   % Cumulative  % Kind (class / dict of class)
     0    404  11   114656  28    114656  28 dict (no owner)
     1   1525  40    99392  24    214048  52 str
     2    966  25    79904  19    293952  71 tuple
     3    361   9    41992  10    335944  81 list
     4    196   5    28224   7    364168  88 tiramisu.option.UnicodeOption
     5    126   3    20240   5    384408  93 unicode
     6     82   2    13120   3    397528  96 tiramisu.option.ChoiceOption
     7     55   1     5720   1    403248  97 tiramisu.option.OptionDescription
     8     23   1     2760   1    406008  98 function
     9     18   0     2592   1    408600  99 tiramisu.option.IntOption
    10     20   1     1120   0    409720  99 __builtin__.cell
    11      1   0     1048   0    410768  99 dict of codecs.CodecInfo
    12      4   0      928   0    411696  99 __builtin__.set
    13      2   0      872   0    412568  99 types.FrameType
    14      7   0      616   0    413184 100 __builtin__.weakref
    15      2   0      560   0    413744 100 dict of _weakrefset.WeakSet
    16      2   0      256   0    414000 100 types.CodeType
    17      1   0      152   0    414152 100 tiramisu.value.Multi
    18      6   0      144   0    414296 100 int
    19      2   0      128   0    414424 100 _weakrefset.WeakSet
    20      1   0       96   0    414520 100 codecs.CodecInfo
    21      1   0       96   0    414616 100 tiramisu.config.Config
    22      1   0       72   0    414688 100 tiramisu.setting.Setting
    23      1   0       64   0    414752 100 tiramisu.value.Values

#3 Updated by Emmanuel GARETTE almost 7 years ago

  • Status changed from Nouveau to Résolu
  • Target version set to Eole 2.4-dev-3

Je peux difficilement faire mieux pour le moment.

#4 Updated by Emmanuel GARETTE almost 7 years ago

  • Due date set to 04/05/2013
  • Start date set to 03/25/2013
  • % Done changed from 0 to 100

#5 Updated by Joël Cuissinat over 6 years ago

  • Status changed from Résolu to Fermé
  • Assigned To set to Emmanuel GARETTE

Also available in: Atom PDF