AirWare
air quality assessment & management
Reference and User Manual
AirWare   On-line Reference Manual
  Release Level 7.0
  Revision Level beta
  Release Date 2015 03

Last modified on:   Thursday, 3-Dec-15 12:33 CET

Backup system

The AirWare backup system provides fully automatic daily incremental dumps (with 1-N days of modification date DELTAS) to any standard FILE (such as on-line disks, removable disks, tape system on the main server or any networked machine.

A dedicated backup server with sufficient disk capacity) which can also be the basic AirWare application server runs a daily, monthly, and manually triggered scripts (driven by cron) that collect all data cross one or more networked servers based on a multi-server configurations file into a tar file, and copies that to the local backup server disk /opt/backup; From there the data are transferred to one or more large data disks (network accessible, nfs mounted on the backup server as /backup1, /backup2, .../backupN, plain or in any RAID configuration desired) where the target repository (/backup[i] disk system) is switched or rotated based on the free disk space of the current target drive. If a threshold free disk space minimum id reached, the drive(s) are switched, and the oldest previous target disk is overwritten. Depending on the available disk space, and the size of the application, this yields incremental backups stored for several months to years.

Backup of DB tables:

uses mysqldump - a database backup program The dump typically contains SQL statements to create the table and populate it.

To reload a dump file, you must have the same privileges needed to create the dump file. Files area just collected via tar, then extracted on the backup-server, so you have there a directory tree.

Backup (target drives and free disk threshold) is configured in the location setup, and several plain ASCII files on the backup servers local directory

    /opt/backup

Backup results/performance and any errors are logged on the Application server's STATUS BOARD, as well as a specific home page of the backup system linked from the Administrate manual of the Application start page. Access to the backup log (daily progress reporting) and the configuration files is from the AirWare server's STATUSBOARD directory.

The backup system home page provides a short summary of the latest backup run:

The links in the tabular summary show:

  • a summary (by top level directory) of the last incremental dump, leading to individual file lists for each server included in the configuration or per directory.
        Directory Listing of /opt/backup/8_6
        aca3g23968 kB24 files
        aca5g874220 kB1959 files
        aca6g11962916 kB19060 files
        aca7g191836 kB 10 files
        ..........................

    • a long-term log of backup runs (daily) that lists possible errors, and the duration of the backup run, total volume, target disk and free disk space.

      The log also shows the current disk for long-term storage, which is rotated (automatically) on a regular basis when a disk-full threshold (configured) is exceeded.

    Configuration

    There are four configuration files, two that enumerate the directories and data bases to be backed up,

    • Config: consists of triplets, with SERVER DIRECTORY DELTA,
      where DELTA denoted the number of days of incremental change that triggers backup 1 mean all files that were changed within the last 24 hours, 2 within the last 48 hours etc.
        File: BACKUPSERVER:/opt/backup/config essg /var/www/html 1 essg /etc 1 zeus /h/kurt 1 zeus /DATA 1 .................................. SERVER DIRECTORY DELTA
  • DB config, that lists server, data base location, and port:
         File:   BACKUPSERVER:/opt/backup/dbconfig
        mp2g /var/lib/mysql 1
        aca3g /var/lib/mysql 1
        aca3g /var/lib/mysql2 1 3308
        aca3g /var/lib/mysql2 1 3309
        #aca4g /var/lib/mysql 1
        #aca4g /var/lib/mysql2 1 3308
        
    and two exclusion lists that enumerate specific directories or file types to be excluded, or data base TABLES to be excluded:
    The configuration file syntax supports comment lines, starting with "#".
    • Exclude list
    • DB exclude list

    Scheduling of backup runs

    The backup script is triggered by the cron system of the BACKUP SERVER

    Scheduling of backup runs

    The backup script is triggered by the cron system of the BACKUPSERVER: edit (as users root) the crontab file with: crontab -e, the backup scripts also perform the regular transfer from the intermediate local storage (on BACKUPSERVER:/backup/opt, to the final long-term storage drives:

      # m h  dom mon dow   command
      0 20 * * * df -k /backup1 > /opt/backup/du1; df -k /backup2 > opt/backup/du2
      0 21 * * * cd /opt/backup; ./main
      

    Adding a new host to the backup list

    • client: BACKUPSERVER (where the backup is run);
    • server: the machine to be backed up.
    Connect to client; as root (or a user in the ADMIN group):

    IF the file /root/.ssh/id-rsa does NOT YET exist:

      type: ssh-keygen -t rsa default directory for keyfiles will be /root/.ssh/ if you do not want to be prompted, leave passphrase blank
    ELSE, or THEN:
      copy the contents of .ssh/id_rsa.pub (there should only be one line) insert on the server in /root/.ssh/authorized_keys

    Manual backup:

      to run extra backups manually: /opt/backup/main

    Retrieving files: RESTORE

    Find the missing or damaged file (name) in the hierarchical backup lists starting from the current backup (STATUSBOARD) or the historical backup list from the backup description home page:

      BACKUPSERVER:/var/www/html/backup.html
    and copy it to the original location as indicated in the backup log files.

    Restoring FILES

    For single files you can try: e.g. for: /var/www/html/RBO/includes/objects_navigator.php Alternatively you can use the command "find", but that can take minutes depending on the backup disk's size

      franz@gis:/backup2$ ls -alt
      */mp2g/var/www/html/RBO/includes/objects_navigator.php
      -rwxrwSrwt 1 root ess 68583 2014-07-08 21:23
      7_8/mp2g/var/www/html/RBO/includes/objects_navigator.php
      -rwxrwSrwt 1 root ess 68476 2014-07-03 21:26
      7_3/mp2g/var/www/html/RBO/includes/objects_navigator.php
      -rwxrwSrwt 1 root ess 68158 2014-06-02 21:24
      6_2/mp2g/var/www/html/RBO/includes/objects_navigator.php
      -rwxrwSrwt 1 root ess 68158 2014-05-23 21:36
      5_23/mp2g/var/www/html/RBO/includes/objects_navigator.php
      
    Then cp or scp (from/to a different server) that file back to be application server; to preserver ownership and modifications dates, use -p flag.

    Restoring DATA BASE DUMPS:

    DB-tables are saved with it's database name prefix: eg. db=AIR,

    • table=AREA_SOURCE,
    • saved backup file: AIR.AREA_SOURCE
    Search the file (created with mysqldump utility) within the backups, sub-directory db, eg. MySQL-table AIR.AREA_SOURCE.
    franz@gis:/backup2$ ls -alt */mp2g/db/AIR.AREA_SOURCE
    -rwxrwSrwt 1 root ess 84042 2014-07-04 21:26 7_4/mp2g/db/AIR.AREA_SOURCE
    -rwxrwSrwt 1 root ess 83083 2014-05-27 21:25 5_27/mp2g/db/AIR.AREA_SOURCE
    -rwxrwSrwt 1 root ess 83021 2014-05-26 21:24 5_26/mp2g/db/AIR.AREA_SOURCE
    -rwxrwSrwt 1 root ess 82792 2014-05-12 21:42 5_12/mp2g/db/AIR.AREA_SOURCE
    -rwxrwSrwt 1 root ess 82704 2014-04-10 21:44 4_10/mp2g/db/AIR.AREA_SOURCE
    
    Then copy the newest (or requested) file to the application server and install via mysqldump utility:
      eg. mysqldump -u user -h host -p password  < AIR.AREA_SOURCE 
      

    Restoring DIRECTORIES or entire disks:

    • Search for the latest zero-level dump as a start, restore;
    • incrementally find and restore individual files and data bases as described above with copies of individual files more recently modified and stored in the incremental dumps.


  • © Copyright 1995-2016 by:   ESS   Environmental Software and Services GmbH AUSTRIA | print page