Project

General

Profile

Tâche #19595

Scénario #20318: Traitement express MEN (19-21)

impossible d'extraire le catalog de son volume 127.0.0.1-dir-catalog-XXXX

Added by Christophe Dezé over 2 years ago. Updated over 2 years ago.

Status:
Fermé
Priority:
Normal
Assigned To:
Start date:
03/10/2017
Due date:
% Done:

100%

Estimated time:
1.00 h
Spent time:
Remaining (hours):
0.0

Description

Je veux faire une restauration complète d'un Horus.
pour une raison que j'ignore, la restauration du catalog bloque si elle ne trouve pas de config.eol dans le volume....

#bacularestore.py --catalog 127.0.0.1-dir
Restauration du catalogue
Pas de fichier /etc/eole/config.eol dans le volume
127.0.0.1-dir-catalog-0011
Impossible de lister le catalogue 127.0.0.1-dir-catalog-0011

vi /usr/share/pyshared/pyeole/bacularestore.py

j'ai commenté
#raise Exception("Impossible de lister le catalogue {0}".format(volume_name))
et le catalog se restaure correctement, et je peux faire une restauration de tout,
bacularestore.py --all

bareos-dir.conf.patch View (313 Bytes) Yoni Baude, 04/27/2017 03:10 PM

bareos-dir.conf.patch View (390 Bytes) Yoni Baude, 04/28/2017 10:13 AM

Associated revisions

Revision d1548285 (diff)
Added by Emmanuel GARETTE over 2 years ago

le catalogue doit être sauvegardé dans une sauvegarde de type FULL (ref #19595)

Revision c815a862 (diff)
Added by Emmanuel GARETTE over 2 years ago

le catalogue bacula doit être sauvegardé dans une sauvegarde de type FULL (ref #19595)

History

#1 Updated by Gérald Schwartzmann over 2 years ago

  • Subject changed from impossible d'extraire le catalog de son volume 27.0.0.1-dir-catalog-XXXX to impossible d'extraire le catalog de son volume 127.0.0.1-dir-catalog-XXXX

#2 Updated by Yoni Baude over 2 years ago

Bonjour,

Sur l'Horus 2.5.2, il faut modifier le fichier /usr/lib/python2.7/dist-packages/pyeole/bareosrestore.py.

#3 Updated by Christophe Dezé over 2 years ago

ligne 208 même ! ;)

#4 Updated by Yoni Baude over 2 years ago

Le problème concerne Bacula et Bareos.

En mode débug du deamon bareos-fd on peut voir le soucis et reproduire le problème via bconsole :

root@horus:~# for i in dir fd sd; do service bareos-$i stop;done
root@horus:~# /usr/sbin/bareos-dir -d 100 -dt >> /var/log/bareos/debug-dir.log 2>&1
root@horus:~# /usr/sbin/bareos-fd -d 100 -dt >> /var/log/bareos/debug-fd.log 2>&1
root@horus:~# /usr/sbin/bareos-sd -d 100 -dt >> /var/log/bareos/debug-sd.log 2>&1

Lorsque la sauvegarde du catalogue est en "Level" "Incr" :

27-avril-2017 11:46:08 horus-fd: fileset.c:251-0 F /etc/eole/bareos.conf
27-avril-2017 11:46:08 horus-fd: fileset.c:251-0 F /etc/eole/config.eol
27-avril-2017 11:46:08 horus-fd: fileset.c:251-0 F /var/lib/bareos/bareos.sql
27-avril-2017 11:46:08 horus-fd: fileset.c:251-0 F /var/lib/bareos/127.0.0.1-dir-JobDefsSauvegarde.bsr
27-avril-2017 11:46:08 horus-fd: fileset.c:251-0 F /var/lib/bareos/127.0.0.1-dir-JobDefsCatalog.bsr
...
27-avril-2017 11:46:08 horus-fd: find.c:156-0 Verify=<V> Accurate=<Cmcs> BaseJob=<Jspug5> flags=<2>
27-avril-2017 11:46:08 horus-fd: bfile.c:1101-0 bopen: fname /var/lib/bareos/bareos.sql, flags 0, mode 0, rdev 0
27-avril-2017 11:46:08 horus-fd: bfile.c:1101-0 bopen: fname /var/lib/bareos/127.0.0.1-dir-JobDefsCatalog.bsr, flags 0, mode 0, rdev 0

Lorsque la sauvegarde du catalogue est en "Level" "Full" :

27-avril-2017 11:47:40 horus-fd: fileset.c:251-0 F /etc/eole/bareos.conf
27-avril-2017 11:47:40 horus-fd: fileset.c:251-0 F /etc/eole/config.eol
27-avril-2017 11:47:40 horus-fd: fileset.c:251-0 F /var/lib/bareos/bareos.sql
27-avril-2017 11:47:40 horus-fd: fileset.c:251-0 F /var/lib/bareos/127.0.0.1-dir-JobDefsSauvegarde.bsr
27-avril-2017 11:47:40 horus-fd: fileset.c:251-0 F /var/lib/bareos/127.0.0.1-dir-JobDefsCatalog.bsr
...
27-avril-2017 11:47:40 horus-fd: find.c:156-0 Verify=<V> Accurate=<Cmcs> BaseJob=<Jspug5> flags=<2>
27-avril-2017 11:47:40 horus-fd: bfile.c:1101-0 bopen: fname /etc/eole/config.eol, flags 0, mode 0, rdev 0
27-avril-2017 11:47:40 horus-fd: bfile.c:1101-0 bopen: fname /var/lib/bareos/bareos.sql, flags 0, mode 0, rdev 0
27-avril-2017 11:47:40 horus-fd: bfile.c:1101-0 bopen: fname /var/lib/bareos/127.0.0.1-dir-JobDefsSauvegarde.bsr, flags 0, mode 0, rdev 0
27-avril-2017 11:47:40 horus-fd: bfile.c:1101-0 bopen: fname /var/lib/bareos/127.0.0.1-dir-JobDefsCatalog.bsr, flags 0, mode 0, rdev 0

Il reste a déterminer pourquoi les "Level" du "BackupCatalog" ne sont pas tous "Full".

Exemple :

Terminated Jobs:
 JobId  Level    Files      Bytes   Status   Finished        Name 
====================================================================
   238  Incr          3    54.76 M  OK -- with warnings  25-Apr-17 00:07 BackupCatalog
   239  Full          0         0   OK       25-Apr-17 00:35 JobSchedulePost
   240  Incr          0         0   OK       26-Apr-17 00:00 JobSchedulePre
   241  Incr        856    6.198 G  OK -- with warnings  26-Apr-17 00:07 JobSauvegarde
   242  Full          4    54.88 M  OK -- with warnings  26-Apr-17 00:07 BackupCatalog
   243  Full          0         0   OK       26-Apr-17 00:35 JobSchedulePost
   244  Incr          0         0   OK       27-Apr-17 00:00 JobSchedulePre
   245  Incr        402    1.684 G  OK -- with warnings  27-Apr-17 00:02 JobSauvegarde
   246  Incr          3    54.92 M  OK -- with warnings  27-Apr-17 00:02 BackupCatalog
   247  Full          0         0   OK       27-Apr-17 00:35 JobSchedulePost

Extrait log bareos-dir

2017-04-26T00:07:32.111694+02:00 horus.RNE.rne.ac-caen.fr bareos-dir: 127.0.0.1-dir JobId 242: Max Volume jobs=1 exceeded. Marking Volume "127.0.0.1-dir-catalog-0046" as Used.
2017-04-26T00:07:32.212296+02:00 horus.RNE.rne.ac-caen.fr bareos-dir: horus-fd JobId 242:      Could not stat "/etc/eole/bareos.conf": ERR=No such file or directory
2017-04-26T00:07:32.522309+02:00 horus.RNE.rne.ac-caen.fr bareos-dir: horus-fd JobId 242: shell command: run ClientAfterJob "sudo /usr/share/eole/sbin/bareosconfig.py --unlock --dir=127.0.0.1-dir --daemon=horus-fd --jobID=242 --backup_ok --jobType=catalogue" 
2017-04-26T00:07:32.522390+02:00 horus.RNE.rne.ac-caen.fr bareos-dir: 127.0.0.1-sd JobId 242: Elapsed time=00:00:01, Transfer rate=54.88 M Bytes/second
2017-04-26T00:07:34.256666+02:00 horus.RNE.rne.ac-caen.fr bareos-dir: 127.0.0.1-dir JobId 242: Bareos 127.0.0.1-dir 14.2.1 (12Sep14):
2017-04-26T00:07:34.256677+02:00 horus.RNE.rne.ac-caen.fr bareos-dir:   Build OS:               x86_64-pc-linux-gnu ubuntu Ubuntu 14.04.5 LTS
2017-04-26T00:07:34.256681+02:00 horus.RNE.rne.ac-caen.fr bareos-dir:   JobId:                  242
2017-04-26T00:07:34.256686+02:00 horus.RNE.rne.ac-caen.fr bareos-dir:   Job:                    BackupCatalog.2017-04-26_00.00.00_03
2017-04-26T00:07:34.256690+02:00 horus.RNE.rne.ac-caen.fr bareos-dir:   Backup Level:           Full (upgraded from Incremental)
2017-04-26T00:07:34.256694+02:00 horus.RNE.rne.ac-caen.fr bareos-dir:   Client:                 "horus-fd" 14.2.1 (12Sep14) x86_64-pc-linux-gnu,ubuntu,Ubuntu 14.04.5 LTS
2017-04-26T00:07:34.256698+02:00 horus.RNE.rne.ac-caen.fr bareos-dir:   FileSet:                "FileSetCatalog" 2017-02-17 11:22:53
2017-04-26T00:07:34.256703+02:00 horus.RNE.rne.ac-caen.fr bareos-dir:   Pool:                   "PoolCatalog" (From Job resource)
2017-04-26T00:07:34.256707+02:00 horus.RNE.rne.ac-caen.fr bareos-dir:   Catalog:                "MyCatalog" (From Client resource)
...

2017-04-27T00:02:52.663232+02:00 horus.RNE.rne.ac-caen.fr bareos-dir: 127.0.0.1-dir JobId 246: Max Volume jobs=1 exceeded. Marking Volume "127.0.0.1-dir-catalog-0046" as Used.
2017-04-27T00:02:52.772164+02:00 horus.RNE.rne.ac-caen.fr bareos-dir: horus-fd JobId 246:      Could not stat "/etc/eole/bareos.conf": ERR=No such file or directory
2017-04-27T00:02:53.106313+02:00 horus.RNE.rne.ac-caen.fr bareos-dir: horus-fd JobId 246: shell command: run ClientAfterJob "sudo /usr/share/eole/sbin/bareosconfig.py --unlock --dir=127.0.0.1-dir --daemon=horus-fd --jobID=246 --backup_ok --jobType=catalogue" 
2017-04-27T00:02:53.106401+02:00 horus.RNE.rne.ac-caen.fr bareos-dir: 127.0.0.1-sd JobId 246: Elapsed time=00:00:01, Transfer rate=54.92 M Bytes/second
2017-04-27T00:02:55.308224+02:00 horus.RNE.rne.ac-caen.fr bareos-dir: 127.0.0.1-dir JobId 246: Bareos 127.0.0.1-dir 14.2.1 (12Sep14):
2017-04-27T00:02:55.308235+02:00 horus.RNE.rne.ac-caen.fr bareos-dir:   Build OS:               x86_64-pc-linux-gnu ubuntu Ubuntu 14.04.5 LTS
2017-04-27T00:02:55.308239+02:00 horus.RNE.rne.ac-caen.fr bareos-dir:   JobId:                  246
2017-04-27T00:02:55.308243+02:00 horus.RNE.rne.ac-caen.fr bareos-dir:   Job:                    BackupCatalog.2017-04-27_00.00.00_40
2017-04-27T00:02:55.308247+02:00 horus.RNE.rne.ac-caen.fr bareos-dir:   Backup Level:           Incremental, since=2017-04-26 00:07:31
2017-04-27T00:02:55.308251+02:00 horus.RNE.rne.ac-caen.fr bareos-dir:   Client:                 "horus-fd" 14.2.1 (12Sep14) x86_64-pc-linux-gnu,ubuntu,Ubuntu 14.04.5 LTS
2017-04-27T00:02:55.308255+02:00 horus.RNE.rne.ac-caen.fr bareos-dir:   FileSet:                "FileSetCatalog" 2017-02-17 11:22:53
2017-04-27T00:02:55.308259+02:00 horus.RNE.rne.ac-caen.fr bareos-dir:   Pool:                   "PoolCatalog" (From Job resource)
2017-04-27T00:02:55.308262+02:00 horus.RNE.rne.ac-caen.fr bareos-dir:   Catalog:                "MyCatalog" (From Client resource)

Cdlt
Yoni

#5 Updated by Yoni Baude over 2 years ago

Il semble que le fait BackupCatalog soit full ou inc dépend d'un délai de temps, j'ai pu remarquer que c'est cyclique :
=> full inc full inc ...

27-avril-2017 00:00:01 127.0.0.1-dir: job.c:195-0 Created job record JobId=13 Name=BackupCatalog.2017-04-27_00.00.01_30 Type=B Level=I
27-avril-2017 00:00:01 127.0.0.1-dir: sql_find.c:132-0 Got start time: 2017-04-26 17:32:44, job: BackupCatalog.2017-04-26_17.32.31_05
27-avril-2017 00:00:01 127.0.0.1-dir: sql_find.c:183-0 Got start time: 2017-04-26 17:32:44
27-avril-2017 00:00:01 127.0.0.1-dir: job.c:970-0 have_full=1 do_full=0 now=1493244001 full_time=1493220764
27-avril-2017 00:00:01 127.0.0.1-dir: job.c:1045-0 Level=I last start time=2017-04-26 17:32:44 job=BackupCatalog.2017-04-26_17.32.31_05
27-avril-2017 00:00:01 127.0.0.1-dir: job.c:1052-0 entering apply_pool_overrides()

A confirmer

#6 Updated by Yoni Baude over 2 years ago

Proposition d'un path pour Baréos pour forcer le BackupCatalog à full :

=> bareos-dir.conf.patch

--- distrib/bareos-dir.conf    2016-10-18 16:47:51.000000000 +0200
+++ modif/bareos-dir.conf    2017-04-27 15:04:36.409650371 +0200
@@ -392,6 +392,7 @@
 Job {
   Name = "BackupCatalog" 
   JobDefs = "JobDefsCatalog" 
+  Level = Full
   FileSet = "FileSetCatalog" 
   Schedule = "ScheduleDefaultPre" 
 #  Prune Jobs = "yes" 

#7 Updated by Yoni Baude over 2 years ago

Le patch proposé bareos-dir.conf.patch concerne le module 2.5.2

#8 Updated by Yoni Baude over 2 years ago

Le patch ne suffit pas, lors d'une programmation de job, le "Level" du volume BackupCatalog reste "Incremental".

#9 Updated by Yoni Baude over 2 years ago

Je précise, lors de la programmation d'une sauvegarde "Incremental", le "Level" du volume BackupCatalog est "Incremental", la valeur par défaut est écrasée.

#10 Updated by Yoni Baude over 2 years ago

Nouveau patch bareos-dir.conf.patch module 2.5.2 pour forcer à Full le "Level" du volume BackupCatalog même lors d'un programmation "Incremental".

--- distrib/bareos-dir.conf    2017-04-03 15:01:24.000000000 +0200
+++ modif/bareos-dir.conf    2017-04-28 10:08:30.257200832 +0200
@@ -392,8 +392,9 @@
 Job {
   Name = "BackupCatalog" 
   JobDefs = "JobDefsCatalog" 
+  Level = Full
   FileSet = "FileSetCatalog" 
-  Schedule = "ScheduleDefaultPre" 
+  Schedule = "ScheduleDefaultPost" 
 #  Prune Jobs = "yes" 
   Pool = "PoolCatalog" 
   Priority = 98

#11 Updated by Yoni Baude over 2 years ago

Erreur de prise en compte de patch sur les fichiers bareos : https://dev-eole.ac-dijon.fr/issues/20336

Modification du fichier /usr/lib/python2.7/dist-packages/pyeole/bareos.py avec la commande :

sed -i -e 's-\/usr\/share\/eole\/creole\/distrib\/-\/var\/lib\/creole\/-g' /usr/lib/python2.7/dist-packages/pyeole/bareos.py

Ou si utilisation d'une variante sur zéphir ajouter le fichier corrigé /usr/lib/python2.7/dist-packages/pyeole/bareos.py en attendant la correction de l'équipe Eole.

#12 Updated by Emmanuel GARETTE over 2 years ago

  • Assigned To set to Emmanuel GARETTE
  • Parent task set to #20318

#13 Updated by Emmanuel GARETTE over 2 years ago

  • Status changed from Nouveau to En cours
  • Estimated time set to 1.00 h
  • Remaining (hours) set to 1.0

#14 Updated by Emmanuel GARETTE over 2 years ago

  • % Done changed from 0 to 80

Correction faite à partir d'une branche 2.5.1 (branche la plus ancienne).

Correction appliqué sur 2.6.1 et master. Etudier pour les autres versions.

#15 Updated by Emmanuel GARETTE over 2 years ago

  • Parent task changed from #20318 to #20101

#16 Updated by Emmanuel GARETTE over 2 years ago

Paquet fait sur 2.4.0, 2.4.1, 2.5.0, 2.5.1, 2.5.2, 2.6.0, 2.6.1 et 2.6.2.

Reste a synchroniser le paquet 2.4.1 en 2.4.2.

#17 Updated by Emmanuel GARETTE over 2 years ago

  • % Done changed from 80 to 90
  • Remaining (hours) changed from 1.0 to 0.5

#18 Updated by Joël Cuissinat over 2 years ago

  • Parent task changed from #20101 to #20318

#19 Updated by Joël Cuissinat over 2 years ago

  • % Done changed from 90 to 100

Copie du paquet eole-bacula de eole-2.4.1-proposed-updates vers eole-2.4.2-proposed-updates.

#20 Updated by Scrum Master over 2 years ago

  • Project changed from Horus to Distribution EOLE
  • Status changed from En cours to Résolu

#21 Updated by Joël Cuissinat over 2 years ago

  • Status changed from Résolu to Fermé
  • Remaining (hours) changed from 0.5 to 0.0
jojo@jojo-MS-B09611:~/git/eole-bareos (master=)$ git branch -r --contains  d1548285 
  origin/2.5.1/master
  origin/2.5.2/master
  origin/2.6.0/master
  origin/2.6.1/master
  origin/HEAD -> origin/master
  origin/dist/eole/2.5.1/master
  origin/dist/eole/2.5.2/master
  origin/dist/eole/2.6.0/master
  origin/dist/eole/2.6.1/master
  origin/dist/eole/2.6.2/master
  origin/master

Also available in: Atom PDF