mirrors/tar
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Properly document incremental dumps

Browse Source
This commit is contained in:
Sergey Poznyakoff
2005-11-06 11:51:12 +00:00
parent 04e2551256
commit d01f65092c
3 changed files with 209 additions and 224 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
doc/snapshot.texi
View File

@@ -5,7 +5,7 @@
@c published by the Free Software Foundation. @c published by the Free Software Foundation.
A @dfn{snapshot file} (or @dfn{directory file}) is created during A @dfn{snapshot file} (or @dfn{directory file}) is created during
incremental backups (@pxref{incremental and listed-incremental}). It incremental backups (@pxref{Incremental Dumps}). It
contains the status of the filesystem at the time of the dump and is contains the status of the filesystem at the time of the dump and is
used to determine which files were modified since the last backup. used to determine which files were modified since the last backup.

421
doc/tar.texi
View File

@@ -240,8 +240,7 @@ Coping with Scarce Resources
Performing Backups and Restoring Files Performing Backups and Restoring Files
* Full Dumps:: Using @command{tar} to Perform Full Dumps * Full Dumps:: Using @command{tar} to Perform Full Dumps
* Inc Dumps:: Using @command{tar} to Perform Incremental Dumps * Incremental Dumps:: Using @command{tar} to Perform Incremental Dumps
* incremental and listed-incremental:: The Incremental Options
* Backup Levels:: Levels of Backups * Backup Levels:: Levels of Backups
* Backup Parameters:: Setting Parameters for Backups and Restoration * Backup Parameters:: Setting Parameters for Backups and Restoration
* Scripted Backups:: Using the Backup Scripts * Scripted Backups:: Using the Backup Scripts
@@ -2203,7 +2202,7 @@ archive, or if they do not already exist in the archive.
Normally when creating an archive, @command{tar} strips an initial Normally when creating an archive, @command{tar} strips an initial
@samp{/} from member names. This option disables that behavior. @samp{/} from member names. This option disables that behavior.
@FIXME-xref{} @xref{absolute}.
@item --after-date @item --after-date
@@ -4592,48 +4591,52 @@ http://www.cs.umd.edu/projects/amanda/amanda.html
ftp://ftp.cs.umd.edu/pub/amanda ftp://ftp.cs.umd.edu/pub/amanda
@end smallexample @end smallexample
@ifclear PUBLISH @FIXME{
Here is a possible plan for a future documentation about the backuping Here is a possible plan for a future documentation about the backuping
scripts which are provided within the @GNUTAR{} scripts which are provided within the @GNUTAR{}
distribution. distribution.
@smallexample @itemize @bullet
.* dumps @item dumps
. + what are dumps @itemize @minus
@item what are dumps
. + different levels of dumps @item different levels of dumps
. - full dump = dump everything @itemize +
. - level 1, level 2 dumps etc, - @item full dump = dump everything
A level n dump dumps everything changed since the last level @item level 1, level 2 dumps etc
n-1 dump (?) A level @var{n} dump dumps everything changed since the last level
@var{n}-1 dump (?)
. + how to use scripts for dumps (ie, the concept) @end itemize
. - scripts to run after editing backup specs (details) @item how to use scripts for dumps (ie, the concept)
@itemize +
. + Backup Specs, what is it. @item scripts to run after editing backup specs (details)
. - how to customize @end itemize
. - actual text of script [/sp/dump/backup-specs] @item Backup Specs, what is it.
@itemize +
. + Problems @item how to customize
. - rsh doesn't work @item actual text of script [/sp/dump/backup-specs]
. - rtape isn't installed @end itemize
. - (others?) @item Problems
@itemize +
. + the --incremental option of tar @item rsh doesn't work
@item rtape isn't installed
. + tapes @item (others?)
. - write protection @end itemize
. - types of media @item the @option{--incremental} option of tar
. : different sizes and types, useful for different things @item tapes
. - files and tape marks @itemize +
@item write protection
@item types of media, different sizes and types, useful for different things
@item files and tape marks
one tape mark between files, two at end. one tape mark between files, two at end.
. - positioning the tape @item positioning the tape
MT writes two at end of write, MT writes two at end of write,
backspaces over one when writing again. backspaces over one when writing again.
@end smallexample @end itemize
@end itemize
@end ifclear @end itemize
}
This chapter documents both the provided shell scripts and @command{tar} This chapter documents both the provided shell scripts and @command{tar}
options which are more specific to usage as a backup tool. options which are more specific to usage as a backup tool.
@@ -4646,8 +4649,7 @@ called @dfn{dumps}.
@menu @menu
* Full Dumps:: Using @command{tar} to Perform Full Dumps * Full Dumps:: Using @command{tar} to Perform Full Dumps
* Inc Dumps:: Using @command{tar} to Perform Incremental Dumps * Incremental Dumps:: Using @command{tar} to Perform Incremental Dumps
* incremental and listed-incremental:: The Incremental Options
* Backup Levels:: Levels of Backups * Backup Levels:: Levels of Backups
* Backup Parameters:: Setting Parameters for Backups and Restoration * Backup Parameters:: Setting Parameters for Backups and Restoration
* Scripted Backups:: Using the Backup Scripts * Scripted Backups:: Using the Backup Scripts
@@ -4681,9 +4683,10 @@ If you want to dump each filesystem separately you will need to use
the @value{op-one-file-system} option to prevent @command{tar} from crossing the @value{op-one-file-system} option to prevent @command{tar} from crossing
filesystem boundaries when storing (sub)directories. filesystem boundaries when storing (sub)directories.
The @value{op-incremental} option is not needed, since this is a complete The @value{op-incremental} (@FIXME-pxref{}) option is not needed,
copy of everything in the filesystem, and a full restore from this since this is a complete copy of everything in the filesystem, and a
backup would only be done onto a completely empty disk. full restore from this backup would only be done onto a completely
empty disk.
Unless you are in a hurry, and trust the @command{tar} program (and your Unless you are in a hurry, and trust the @command{tar} program (and your
tapes), it is a good idea to use the @value{op-verify} option, to make tapes), it is a good idea to use the @value{op-verify} option, to make
@@ -4692,199 +4695,183 @@ also detect cases where the file was modified while (or just after)
it was being archived. Not all media (notably cartridge tapes) are it was being archived. Not all media (notably cartridge tapes) are
capable of being verified, unfortunately. capable of being verified, unfortunately.
@value{op-listed-incremental} take a file name argument always. If the @node Incremental Dumps
file doesn't exist, run a level zero dump, creating the file. If the
file exists, uses that file to see what has changed.
@value{op-incremental} @FIXME{look it up}
@value{op-incremental} handle old @acronym{GNU}-format incremental backup.
This option should only be used when creating an incremental backup of
a filesystem. When the @value{op-incremental} option is used, @command{tar}
writes, at the beginning of the archive, an entry for each of the
directories that will be operated on. The entry for a directory
includes a list of all the files in the directory at the time the
dump was done, and a flag for each file indicating whether the file
is going to be put in the archive. This information is used when
doing a complete incremental restore.
Note that this option causes @command{tar} to create a non-standard
archive that may not be readable by non-@acronym{GNU} versions of the
@command{tar} program.
The @value{op-incremental} option means the archive is an incremental
backup. Its meaning depends on the command that it modifies.
If the @value{op-incremental} option is used with @value{op-list},
@command{tar} will list, for each directory in the archive, the list
of files in that directory at the time the archive was created. This
information is put out in a format that is not easy for humans to
read, but which is unambiguous for a program: each file name is
preceded by either a @samp{Y} if the file is present in the archive,
an @samp{N} if the file is not included in the archive, or a @samp{D}
if the file is a directory (and is included in the archive). Each
file name is terminated by a null character. The last file is
followed by an additional null and a newline to indicate the end of
the data.
If the @value{op-incremental} option is used with @value{op-extract}, then
when the entry for a directory is found, all files that currently
exist in that directory but are not listed in the archive @emph{are
deleted from the directory}.
This behavior is convenient when you are restoring a damaged file
system from a succession of incremental backups: it restores the
entire state of the file system to that which obtained when the backup
was made. If you don't use @value{op-incremental}, the file system will
probably fill up with files that shouldn't exist any more.
@value{op-listed-incremental} handle new @acronym{GNU}-format
incremental backup. This option handles new @acronym{GNU}-format
incremental backup. It has much the same effect as
@value{op-incremental}, but also the time when the dump is done and
the list of directories dumped is written to the given
@var{file}. When restoring, only files newer than the saved time are
restored, and the directory list is used to speed up operations.
@value{op-listed-incremental} acts like @value{op-incremental}, but when
used in conjunction with @value{op-create} will also cause @command{tar} to
use the file @var{file}, which contains information about the state
of the filesystem at the time of the last backup, to decide which
files to include in the archive being created. That file will then
be updated by @command{tar}. If the file @var{file} does not exist when
this option is specified, @command{tar} will create it, and include all
appropriate files in the archive.
The file, which is archive independent, contains the date it was last
modified and a list of devices, inode numbers and directory names.
@command{tar} will archive files with newer mod dates or inode change
times, and directories with an unchanged inode number and device but
a changed directory name. The file is updated after the files to
be archived are determined, but before the new archive is actually
created.
@node Inc Dumps
@section Using @command{tar} to Perform Incremental Dumps @section Using @command{tar} to Perform Incremental Dumps
@UNREVISED
@cindex incremental dumps @dfn{Incremental backup} is a special form of @GNUTAR{} archive that
@cindex dumps, incremental stores additional metadata so that exact state of the filesystem
can be restored when extracting the archive.
Performing incremental dumps is similar to performing full dumps, @GNUTAR{} currently offers two options for handling incremental
although a few more options will usually be needed. backups: @value{op-listed-incremental} and @value{op-incremental}.
A standard scheme is to do a @emph{monthly} (full) dump once a month, The option @option{--listed-incremental} instructs tar to operate on
a @emph{weekly} dump once a week of everything since the last monthly an incremental archive with additional metadata stored in a standalone
and a @emph{daily} every day of everything since the last (weekly or file, called @dfn{snapshot file}. The purpose of this file is to help
monthly) dump. determine what files have been changed, added or deleted since the
last backup, so that the next incremental backup will contain only
modified files. The name of the snapshot file is given as an argument
to the option:
Here is a sample script to dump the directory hierarchies @samp{/usr} @table @option
and @samp{/var}. @item --listed-incremental=@var{file}
@itemx -g @var{file}
Handle incremental backups with snapshot data in @var{file}.
@end table
To create an incremental backup, you would use
@option{--listed-incremental} together with @option{--create}
(@pxref{create}). For example:
@smallexample @smallexample
#! /bin/sh $ @kbd{tar --create \
tar --create \ --file=archive.1.tar \
--blocking-factor=126 \ --listed-incremental=/var/log/usr.snar \
--file=/dev/rmt/0 \ /usr}
--label="`hostname` /usr /var `date +%Y-%m-%d`" \
--listed-incremental=/var/log/usr-var.snar \
--verbose \
/usr /var
@end smallexample @end smallexample
This script uses the file @file{/var/log/usr-var.snar} as a snapshot to This will create in @file{archive.1.tar} an incremental backup of
store information about the previous tar dump. @file{/usr} filesystem, storing additional metadata in the file
@file{/var/log/usr.snar}. If this file does not exist, it will be
created. The created archive will then be called @dfn{level 0 backup}
(see the next section for more info on backup levels).
The blocking factor 126 is an attempt to make the tape drive stream. Otherwise, if the file @file{/var/log/usr.snar} exists, it is used to
Some tape devices cannot handle 64 kB blocks or larger, and require the determine the modified files. In this case only these files will be
block size to be a multiple of 1 kB; for these devices, 126 is the stored in the archive. Suppose, for example, that after running the
largest blocking factor that can be used. above command, you delete file @file{/usr/doc/old} and create
directory @file{/usr/local/db} with the following contents:
@node incremental and listed-incremental @smallexample
@section The Incremental Options $ @kbd{ls /usr/local/db}
@UNREVISED /usr/local/db/data
/usr/local/db/index
@end smallexample
@value{op-incremental} is used in conjunction with @value{op-create}, Some time later you create another incremental backup. You will
@value{op-extract} or @value{op-list} when backing up and restoring file then see:
systems. An archive cannot be extracted or listed with the
@value{op-incremental} option specified unless it was created with the
option specified. This option should only be used by a script, not by
the user, and is usually disregarded in favor of
@value{op-listed-incremental}, which is described below.
@value{op-incremental} in conjunction with @value{op-create} causes @smallexample
@command{tar} to write, at the beginning of the archive, an entry for $ @kbd{tar --create \
each of the directories that will be archived. The entry for a --file=archive.2.tar \
directory includes a list of all the files in the directory at the --listed-incremental=/var/log/usr.snar \
time the archive was created and a flag for each file indicating /usr}
whether or not the file is going to be put in the archive. tar: usr/local/db: Directory is new
usr/local/db/
usr/local/db/data
usr/local/db/index
@end smallexample
Note that this option causes @command{tar} to create a non-standard @noindent
archive that may not be readable by non-@acronym{GNU} versions of the The created archive @file{archive.2.tar} will contain only these
@command{tar} program. three members. This archive is called @dfn{level 1 backup}. Notice,
that @file{/var/log/usr.snar} will be updated with the new data, so if
you plan to create more @samp{level 1} backups, it is necessary to
create a working copy of the snapshot file before running
@command{tar}. The above example will then be modified as follows:
@value{op-incremental} in conjunction with @value{op-extract} causes @smallexample
@command{tar} to read the lists of directory contents previously stored $ @kbd{cp /var/log/usr.snar /var/log/usr.snar-1}
in the archive, @emph{delete} files in the file system that did not $ @kbd{tar --create \
exist in their directories when the archive was created, and then --file=archive.2.tar \
extract the files in the archive. --listed-incremental=/var/log/usr.snar-1 \
/usr}
This behavior is convenient when restoring a damaged file system from @end smallexample
a succession of incremental backups: it restores the entire state of
the file system to that which obtained when the backup was made. If
@value{op-incremental} isn't specified, the file system will probably
fill up with files that shouldn't exist any more.
@value{op-incremental} in conjunction with @value{op-list} and two
@value{op-verbose} options causes @command{tar} to print, for each
directory in the archive, the list of files in that directory at the
time the archive was created. This information is put out in a format
that is not easy for humans to read, but which is unambiguous for a
program: each file name is preceded by either a @samp{Y} if the file
is present in the archive, an @samp{N} if the file is not included in
the archive, or a @samp{D} if the file is a directory (and is included
in the archive). Each file name is terminated by a newline character.
The last file is followed by an additional newline to
indicate the end of the data.
@value{op-listed-incremental} acts like @value{op-incremental}, but when
used in conjunction with @value{op-create} will also cause @command{tar}
to use the file @var{snapshot-file}, which contains information about
the state of the file system at the time of the last backup, to decide
which files to include in the archive being created. That file will
then be updated by @command{tar}. If the file @var{file} does not exist
when this option is specified, @command{tar} will create it, and include
all appropriate files in the archive.
The file @var{file}, which is archive independent, contains the date
it was last modified and a list of devices, inode numbers and
directory names. @command{tar} will archive files with newer mod dates
or inode change times, and directories with an unchanged inode number
and device but a changed directory name. The file is updated after
the files to be archived are determined, but before the new archive is
actually created.@FIXME-xref{to the description of the file format}.
Incremental dumps depend crucially on time stamps, so the results are Incremental dumps depend crucially on time stamps, so the results are
unreliable if you modify a file's time stamps during dumping (e.g., unreliable if you modify a file's time stamps during dumping (e.g.,
with the @option{--atime-preserve} option), or if you set the clock with the @option{--atime-preserve} option), or if you set the clock
backwards. backwards.
Despite it should be obvious that a device has a non-volatile value, NFS Metadata stored in snapshot files include device numbers, which,
devices have non-dependable values when an automounter gets in the picture. obviously is supposed to be a non-volatile value. However, it turns
This led to a great deal of spurious redumping in incremental dumps, out that NFS devices have non-dependable values when an automounter
so it is somewhat useless to compare two NFS devices numbers over time. gets in the picture. This can lead to a great deal of spurious
So @command{tar} now considers all NFS devices as being equal when it comes redumping in incremental dumps, so it is somewhat useless to compare
to comparing directories; this is fairly gross, but there does not seem two NFS devices numbers over time. The solution implemented currently
is to considers all NFS devices as being equal when it comes to
comparing directories; this is fairly gross, but there does not seem
to be a better way to go. to be a better way to go.
@command{tar} doesn't access @var{snapshot-file} when Note that incremental archives use @command{tar} extensions and may
@value{op-extract} or @value{op-list} are specified, but the not be readable by non-@acronym{GNU} versions of the @command{tar} program.
@value{op-listed-incremental} option must still be given. A
placeholder @var{snapshot-file} can be specified, e.g.,
@file{/dev/null}.
@FIXME{this section needs to be written} To extract from the incremental dumps, use
@option{--listed-incremental} together with @option{--extract}
option (@pxref{extracting files}). In this case, @command{tar} does
not need to access snapshot file, since all the data necessary for
extraction are stored in the archive itself. So, when extracting, you
can give whatever argument to @option{--listed-incremental}, the usual
practice is to use @option{--listed-incremental=/dev/null}.
Alternatively, you can use @option{--incremental}, which needs no
arguments. In general, @option{--incremental} (@option{-G}) can be
used as a shortcut for @option{--listed-incremental} when listing or
extracting incremental backups (for more information, regarding this
option, @pxref{incremental-op}).
When extracting from the incremental backup @GNUTAR{} attempts to
restore the exact state the file system had when the archive was
created. In particular, it will @emph{delete} those files in the file
system that did not exist in their directories when the archive was
created. If you have created several levels of incremental files,
then in order to restore the exact contents the file system had when
the last level was created, you will need to restore from all backups
in turn. Continuing our example, to restore the state of @file{/usr}
file system, one would do@footnote{Notice, that since both archives
were created withouth @option{-P} option (@pxref{absolute}), these
commands should be run from the root filesystem.}:
@smallexample
$ @kbd{tar --extract \
--listed-incremental=/dev/null \
--file archive.1.tar}
$ @kbd{tar --extract \
--listed-incremental=/dev/null \
--file archive.2.tar}
@end smallexample
To list the contents of an incremental archive, use @option{--list}
(@pxref{list}), as usual. To obtain more information about the
archive, use @option{--listed-incremental} or @option{--incremental}
combined with two @option{--verbose} options@footnote{Two
@option{--verbose} options were selected to avoid breaking usual
verbose listing output (@option{--list --verbose}) when using in
scripts.
Versions of @GNUTAR{} up to 1.15.1 used to dump verbatim binary
contents of the DUMPDIR header (with terminating nulls) when
@option{--incremental} or @option{--listed-incremental} option was
given, no matter what the verbosity level. This behavior, and,
especially, the binary output it produced were considered incovenient
and were changed in version 1.16}:
@smallexample
@kbd{tar --list --incremental --verbose --verbose archive.tar}
@end smallexample
This command will print, for each directory in the archive, the list
of files in that directory at the time the archive was created. This
information is put out in a format which is both human-readable and
unambiguous for a program: each file name is printed as
@smallexample
@var{x} @var{file}
@end smallexample
@noindent
where @var{x} is a letter describing the status of the file: @samp{Y}
if the file is present in the archive, @samp{N} if the file is not
included in the archive, or a @samp{D} if the file is a directory (and
is included in the archive).@FIXME-xref{dumpdir format}. Each such
line is terminated by a newline character. The last line is followed
by an additional newline to indicate the end of the data.
@anchor{incremental-op}The option @option{--incremental} (@option{-G})
gives the same behavior as @option{--listed-incremental} when used
with @option{--list} and @option{--extract} options. When used with
@option{--create} option, it creates an incremental archive without
creating snapshot file. Thus, it is impossible to create several
levels of incremental backups with @option{--incremental} option.
@node Backup Levels @node Backup Levels
@section Levels of Backups @section Levels of Backups
@@ -4930,8 +4917,8 @@ their use in detail.
designed to be used together. While it is possible to restore files by designed to be used together. While it is possible to restore files by
hand from an archive which was created using a backup script, and to create hand from an archive which was created using a backup script, and to create
an archive by hand which could then be extracted using the restore script, an archive by hand which could then be extracted using the restore script,
it is easier to use the scripts. @value{xref-incremental}, and it is easier to use the scripts. @value{xref-incremental}, before
@value{xref-listed-incremental}, before making such an attempt. making such an attempt.
@node Backup Parameters @node Backup Parameters
@section Setting Parameters for Backups and Restoration @section Setting Parameters for Backups and Restoration
@@ -5442,8 +5429,8 @@ positioning.}
system if they were not in the file system when the archive was made. system if they were not in the file system when the archive was made.
@end quotation @end quotation
@value{xref-incremental}, and @value{ref-listed-incremental}, @value{xref-incremental}, for an explanation of how the script makes
for an explanation of how the script makes that determination. that determination.
@node Choosing @node Choosing
@chapter Choosing Files and Names for @command{tar} @chapter Choosing Files and Names for @command{tar}
@@ -6087,7 +6074,7 @@ arguments.
@strong{Please Note:} @value{op-after-date} and @value{op-newer-mtime} @strong{Please Note:} @value{op-after-date} and @value{op-newer-mtime}
should not be used for incremental backups. Some files (such as those should not be used for incremental backups. Some files (such as those
in renamed directories) are not selected properly by these options. in renamed directories) are not selected properly by these options.
@xref{incremental and listed-incremental}. @xref{Incremental Dumps}.
@end quotation @end quotation
@noindent @noindent

10
doc/value.texi
View File

@@ -155,9 +155,9 @@
@set pxref-ignore-zeros @pxref{Reading} @set pxref-ignore-zeros @pxref{Reading}
@set op-incremental @option{--incremental} (@option{-G}) @set op-incremental @option{--incremental} (@option{-G})
@set ref-incremental @ref{Inc Dumps} @set ref-incremental @ref{Incremental Dumps}
@set xref-incremental @xref{Inc Dumps} @set xref-incremental @xref{Incremental Dumps}
@set pxref-incremental @pxref{Inc Dumps} @set pxref-incremental @pxref{Incremental Dumps}
@set op-info-script @option{--info-script=@var{script-name}} (@option{--new-volume-script=@var{script-name}}, @option{-F @var{script-name}}) @set op-info-script @option{--info-script=@var{script-name}} (@option{--new-volume-script=@var{script-name}}, @option{-F @var{script-name}})
@set ref-info-script @ref{Multi-Volume Archives} @set ref-info-script @ref{Multi-Volume Archives}
@@ -190,9 +190,7 @@
@set pxref-list @pxref{list} @set pxref-list @pxref{list}
@set op-listed-incremental @option{--listed-incremental=@var{snapshot-file}} (@option{-g @var{snapshot-file}}) @set op-listed-incremental @option{--listed-incremental=@var{snapshot-file}} (@option{-g @var{snapshot-file}})
@set ref-listed-incremental @ref{Inc Dumps} @c No special refs for this option, use {*ref-incremental} instead.
@set xref-listed-incremental @xref{Inc Dumps}
@set pxref-listed-incremental @pxref{Inc Dumps}
@set op-mode @option{--mode=@var{permissions}} @set op-mode @option{--mode=@var{permissions}}
@set ref-mode @ref{Option Summary} @set ref-mode @ref{Option Summary}