This commit is contained in:
DatuX committed 2026-08-26 21:00:26 +02:00
commit c653b0f8f1
55 files changed
+4543 -2327

No files matched your search

+2 -1
View File
@@ -1,7 +1,7 @@
name: Regression tests
on: ["push", "pull_request"]
on: [ "push", "pull_request" ]
@@ -25,3 +25,4 @@ jobs:
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
run: coveralls --service=github || true
+1
View File
@@ -12,3 +12,4 @@ python2.env
venv
.idea
password.sh
.DS_Store
+20 -19
View File
@@ -1,4 +1,3 @@
# ZFS autobackup
[![Tests](https://github.com/psy0rz/zfs_autobackup/workflows/Regression%20tests/badge.svg)](https://github.com/psy0rz/zfs_autobackup/actions?query=workflow%3A%22Regression+tests%22) [![Coverage Status](https://coveralls.io/repos/github/psy0rz/zfs_autobackup/badge.svg)](https://coveralls.io/github/psy0rz/zfs_autobackup) [![Python Package](https://github.com/psy0rz/zfs_autobackup/workflows/Upload%20Python%20Package/badge.svg)](https://pypi.org/project/zfs-autobackup/)
@@ -10,26 +9,31 @@ ZFS-autobackup tries to be the most reliable and easiest to use tool, while havi
You can either use it as a **backup** tool, **replication** tool or **snapshot** tool.
You can select what to backup by setting a custom `ZFS property`. This makes it easy to add/remove specific datasets, or just backup your whole pool.
You can select what to backup by setting a custom `ZFS property`. This makes it easy to add/remove specific datasets, or
just backup your whole pool.
Other settings are just specified on the commandline: Simply setup and test your zfs-autobackup command and fix all the issues you might encounter. When you're done you can just copy/paste your command to a cron or script.
Other settings are just specified on the commandline: Simply setup and test your zfs-autobackup command and fix all the
issues you might encounter. When you're done you can just copy/paste your command to a cron or script.
Since it's using ZFS commands, you can see what it's actually doing by specifying `--debug`. This also helps a lot if you run into some strange problem or errors. You can just copy-paste the command that fails and play around with it on the commandline. (something I missed in other tools)
Since it's using ZFS commands, you can see what it's actually doing by specifying `--debug`. This also helps a lot if
you run into some strange problem or errors. You can just copy-paste the command that fails and play around with it on
the commandline. (something I missed in other tools)
An important feature that's missing from other tools is a reliable `--test` option: This allows you to see what zfs-autobackup will do and tune your parameters. It will do everything, except make changes to your system.
An important feature that's missing from other tools is a reliable `--test` option: This allows you to see what
zfs-autobackup will do and tune your parameters. It will do everything, except make changes to your system.
## Features
* Works across operating systems: Tested with **Linux**, **FreeBSD/FreeNAS** and **SmartOS**.
* Low learning curve: no complex daemons or services, no additional software or networking needed.
* Low learning curve: no complex daemons or services, no additional software or networking needed.
* Plays nicely with existing replication systems. (Like Proxmox HA)
* Automatically selects filesystems to backup by looking at a simple ZFS property.
* Automatically selects filesystems to backup by looking at a simple ZFS property.
* Creates consistent snapshots. (takes all snapshots at once, atomicly.)
* Multiple backups modes:
* Backup local data on the same server.
* "push" local data to a backup-server via SSH.
* "pull" remote data from a server via SSH and backup it locally.
* "pull+push": Zero trust between source and target.
* Backup local data on the same server.
* "push" local data to a backup-server via SSH.
* "pull" remote data from a server via SSH and backup it locally.
* "pull+push": Zero trust between source and target.
* Can be scheduled via simple cronjob or run directly from commandline.
* Also supports complex backup geometries.
* ZFS encryption support: Can decrypt / encrypt or even re-encrypt datasets during transfer.
@@ -48,14 +52,11 @@ An important feature that's missing from other tools is a reliable `--test` opti
* Complete and clean logging.
* All code is regression tested against actual ZFS environments.
* Easy installation:
* Just install zfs-autobackup via pip.
* Only needs to be installed on one side.
* Written in python and uses zfs-commands, no special 3rd party dependency's or compiled libraries needed.
* No annoying config files or properties.
* Easy to use in scripts, for example https://github.com/psy0rz/zfs_autobackup/wiki/Example-Proxmox
* Just install zfs-autobackup via pip.
* Only needs to be installed on one side.
* Written in python and uses zfs-commands, no special 3rd party dependency's or compiled libraries needed.
* No annoying config files or properties.
## Getting started
Please look at our wiki to [Get started](https://github.com/psy0rz/zfs_autobackup/wiki).
@@ -66,6 +67,6 @@ Or read the [Full manual](https://github.com/psy0rz/zfs_autobackup/wiki/Manual)
This project was sponsored by:
* JetBrains
* JetBrains
* https://rsync.net
* [DatuX](https://www.datux.nl)
-1
View File
@@ -1,5 +1,4 @@
colorama
argparse
coverage
python-coveralls
unittest2
+1 -2
View File
@@ -26,12 +26,11 @@ setuptools.setup(
packages=setuptools.find_packages(),
classifiers=[
"Programming Language :: Python :: 2",
"Programming Language :: Python :: 3",
"License :: OSI Approved :: GNU General Public License v3 (GPLv3)",
"Operating System :: OS Independent",
],
python_requires='>=2.7',
python_requires='>=3.2',
install_requires=[
"colorama",
"argparse"
+6
View File
@@ -0,0 +1,6 @@
import os
import sys
_here = os.path.dirname(os.path.abspath(__file__))
if _here not in sys.path:
sys.path.insert(0, _here)
+4
View File
@@ -0,0 +1,4 @@
#!/bin/bash
# to run one: -k test_incremental_resume
find tests/*.py zfs_autobackup/*.py | entr -r ./tests/remoteruntest $@
+10
View File
@@ -0,0 +1,10 @@
#!/bin/bash
set -e
HOST=root@ws2
DST=/home/psy/zfs_autobackup
rsync --delete -avx . "$HOST:$DST"
ssh $HOST "cd $DST && ./tests/run_test $@"
+4 -3
View File
@@ -1,5 +1,6 @@
#!/bin/bash
#run one test. start from main directory
python -m unittest discover tests $@ -vvvf
#run all or one test. start from main directory
# to run one: -k test_incremental_resume
cd tests
python -m unittest discover .. $@ -vvvf
+2 -1
View File
@@ -31,7 +31,8 @@ modprobe brd rd_size=512000
umount /tmp/ZfsCheck*
coverage run --branch --source zfs_autobackup -m unittest discover -vvvvf $SCRIPTDIR $@ 2>&1
cd $SCRIPTDIR
coverage run --branch --source zfs_autobackup -m unittest discover -vvvvf .. $@ 2>&1
EXIT=$?
echo
+20 -20
View File
@@ -20,28 +20,28 @@ class TestBlockHasher(unittest2.TestCase):
def test_empty(self):
block_hasher = BlockHasher(count=1)
self.assertEqual(
list(block_hasher.generate("tests/data/empty")),
list(block_hasher.generate("data/empty")),
[]
)
def test_partial(self):
block_hasher = BlockHasher(count=1)
self.assertEqual(
list(block_hasher.generate("tests/data/partial")),
list(block_hasher.generate("data/partial")),
[(0, "642027d63bb0afd7e0ba197f2c66ad03e3d70de1")]
)
def test_whole(self):
block_hasher = BlockHasher(count=1)
self.assertEqual(
list(block_hasher.generate("tests/data/whole")),
list(block_hasher.generate("data/whole")),
[(0, "3c0bf91170d873b8e327d3bafb6bc074580d11b7")]
)
def test_whole2(self):
block_hasher = BlockHasher(count=1)
self.assertEqual(
list(block_hasher.generate("tests/data/whole_whole2")),
list(block_hasher.generate("data/whole_whole2")),
[
(0, "3c0bf91170d873b8e327d3bafb6bc074580d11b7"),
(1, "2e863f1fcccd6642e4e28453eba10d2d3f74d798")
@@ -51,7 +51,7 @@ class TestBlockHasher(unittest2.TestCase):
def test_wwp(self):
block_hasher = BlockHasher(count=1)
self.assertEqual(
list(block_hasher.generate("tests/data/whole_whole2_partial")),
list(block_hasher.generate("data/whole_whole2_partial")),
[
(0, "3c0bf91170d873b8e327d3bafb6bc074580d11b7"), # whole
(1, "2e863f1fcccd6642e4e28453eba10d2d3f74d798"), # whole2
@@ -62,7 +62,7 @@ class TestBlockHasher(unittest2.TestCase):
def test_wwp_count2(self):
block_hasher = BlockHasher(count=2)
self.assertEqual(
list(block_hasher.generate("tests/data/whole_whole2_partial")),
list(block_hasher.generate("data/whole_whole2_partial")),
[
(0, "959e6b58078f0cfd2fb3d37e978fda51820473ff"), # whole_whole2
(1, "642027d63bb0afd7e0ba197f2c66ad03e3d70de1") # partial
@@ -72,7 +72,7 @@ class TestBlockHasher(unittest2.TestCase):
def test_big(self):
block_hasher = BlockHasher(count=10)
self.assertEqual(
list(block_hasher.generate("tests/data/whole_whole2_partial")),
list(block_hasher.generate("data/whole_whole2_partial")),
[
(0, "309ffffba2e1977d12f3b7469971f30d28b94bd8"), # whole_whole2_partial
])
@@ -80,32 +80,32 @@ class TestBlockHasher(unittest2.TestCase):
def test_blockhash_compare(self):
#no errors
block_hasher = BlockHasher(count=1)
generator = block_hasher.generate("tests/data/whole_whole2_partial")
self.assertEqual([], list(block_hasher.compare("tests/data/whole_whole2_partial", generator)))
generator = block_hasher.generate("data/whole_whole2_partial")
self.assertEqual([], list(block_hasher.compare("data/whole_whole2_partial", generator)))
#compare file is smaller (EOF errors)
block_hasher = BlockHasher(count=1)
generator = block_hasher.generate("tests/data/whole_whole2_partial")
generator = block_hasher.generate("data/whole_whole2_partial")
self.assertEqual(
[(1, '2e863f1fcccd6642e4e28453eba10d2d3f74d798', 'EOF'),
(2, '642027d63bb0afd7e0ba197f2c66ad03e3d70de1', 'EOF')],
list(block_hasher.compare("tests/data/whole", generator)))
list(block_hasher.compare("data/whole", generator)))
#no errors, huge chunks
block_hasher = BlockHasher(count=10)
generator = block_hasher.generate("tests/data/whole_whole2_partial")
self.assertEqual([], list(block_hasher.compare("tests/data/whole_whole2_partial", generator)))
generator = block_hasher.generate("data/whole_whole2_partial")
self.assertEqual([], list(block_hasher.compare("data/whole_whole2_partial", generator)))
# different order to make sure seek functions are ok
block_hasher = BlockHasher(count=1)
checksums = list(block_hasher.generate("tests/data/whole_whole2_partial"))
checksums = list(block_hasher.generate("data/whole_whole2_partial"))
checksums.reverse()
self.assertEqual([], list(block_hasher.compare("tests/data/whole_whole2_partial", checksums)))
self.assertEqual([], list(block_hasher.compare("data/whole_whole2_partial", checksums)))
def test_skip1(self):
block_hasher = BlockHasher(count=1, skip=1)
self.assertEqual(
list(block_hasher.generate("tests/data/whole_whole2_partial")),
list(block_hasher.generate("data/whole_whole2_partial")),
[
(0, "3c0bf91170d873b8e327d3bafb6bc074580d11b7"), # whole
# (1, "2e863f1fcccd6642e4e28453eba10d2d3f74d798"), # whole2
@@ -115,7 +115,7 @@ class TestBlockHasher(unittest2.TestCase):
#should continue the pattern on the next file:
self.assertEqual(
list(block_hasher.generate("tests/data/whole_whole2_partial")),
list(block_hasher.generate("data/whole_whole2_partial")),
[
# (0, "3c0bf91170d873b8e327d3bafb6bc074580d11b7"), # whole
(1, "2e863f1fcccd6642e4e28453eba10d2d3f74d798"), # whole2
@@ -126,7 +126,7 @@ class TestBlockHasher(unittest2.TestCase):
def test_skip6(self):
block_hasher = BlockHasher(count=1, skip=6)
self.assertEqual(
list(block_hasher.generate("tests/data/whole_whole2_partial")),
list(block_hasher.generate("data/whole_whole2_partial")),
[
(0, "3c0bf91170d873b8e327d3bafb6bc074580d11b7"), # whole
# (1, "2e863f1fcccd6642e4e28453eba10d2d3f74d798"), # whole2
@@ -136,7 +136,7 @@ class TestBlockHasher(unittest2.TestCase):
#all blocks of next file are skipped
self.assertEqual(
list(block_hasher.generate("tests/data/whole_whole2_partial")),
list(block_hasher.generate("data/whole_whole2_partial")),
[
# (0, "3c0bf91170d873b8e327d3bafb6bc074580d11b7"), # whole
# (1, "2e863f1fcccd6642e4e28453eba10d2d3f74d798"), # whole2
@@ -146,7 +146,7 @@ class TestBlockHasher(unittest2.TestCase):
#first block of this one is the 6th to be skipped:
self.assertEqual(
list(block_hasher.generate("tests/data/whole_whole2_partial")),
list(block_hasher.generate("data/whole_whole2_partial")),
[
# (0, "3c0bf91170d873b8e327d3bafb6bc074580d11b7"), # whole
(1, "2e863f1fcccd6642e4e28453eba10d2d3f74d798"), # whole2
+300
View File
@@ -0,0 +1,300 @@
from basetest import *
class TestZfsAutobackupClone(unittest2.TestCase):
"""ZFS clone replication: source datasets that are themselves clones get
replicated to the target as clones of the corresponding target-side origin
snapshot, via 'zfs send -i <origin_snap> <clone>@<snap>'."""
def setUp(self):
prepare_zpools()
self.longMessage = True
def test_clone_happy_path(self):
"""Single run, both origin and clone selected. The origin snapshot is
auto-included because a clone in the selection depends on it, so the user
doesn't need --other-snapshots."""
shelltest("zfs snapshot test_source1/fs1@base")
shelltest("zfs clone test_source1/fs1@base test_source1/fs1_clone")
shelltest("zfs set autobackup:test=true test_source1/fs1_clone")
before = shelltest("zfs list -H -o name -r -t snapshot " + TEST_POOLS)
with OutputIO() as buf_out, OutputIO() as buf_err:
with redirect_stdout(buf_out), redirect_stderr(buf_err):
with mocktime("20101111000000"):
self.assertFalse(ZfsAutobackup(
"test test_target1 --no-progress --verbose --test".split(" ")).run())
combined = buf_out.getvalue() + buf_err.getvalue()
print(combined)
self.assertNotIn("guid", combined) # testing can give guid errors
self.assertNotIn("STDERR", combined)
after = shelltest("zfs list -H -o name -r -t snapshot " + TEST_POOLS)
self.assertMultiLineEqual(before, after)
with mocktime("20101111000000"):
self.assertFalse(ZfsAutobackup(
"test test_target1 --no-progress --verbose".split(" ")).run())
r = shelltest("zfs list -H -o name -r -t snapshot,filesystem " + TEST_POOLS)
self.assertMultiLineEqual(r, """
test_source1
test_source1/fs1
test_source1/fs1@base
test_source1/fs1@test-20101111000000
test_source1/fs1/sub
test_source1/fs1/sub@test-20101111000000
test_source1/fs1_clone
test_source1/fs1_clone@test-20101111000000
test_source2
test_source2/fs2
test_source2/fs2/sub
test_source2/fs2/sub@test-20101111000000
test_source2/fs3
test_source2/fs3/sub
test_target1
test_target1/test_source1
test_target1/test_source1/fs1
test_target1/test_source1/fs1@base
test_target1/test_source1/fs1@test-20101111000000
test_target1/test_source1/fs1/sub
test_target1/test_source1/fs1/sub@test-20101111000000
test_target1/test_source1/fs1_clone
test_target1/test_source1/fs1_clone@test-20101111000000
test_target1/test_source2
test_target1/test_source2/fs2
test_target1/test_source2/fs2/sub
test_target1/test_source2/fs2/sub@test-20101111000000
""")
origin = shelltest("zfs get -H -o value origin test_target1/test_source1/fs1_clone").strip()
self.assertEqual(origin, "test_target1/test_source1/fs1@base")
def test_clone_origin_not_on_target_falls_back_to_full_send(self):
"""If the clone's origin isn't replicated, warn and fall back to full send."""
shelltest("zfs inherit autobackup:test test_source1/fs1")
shelltest("zfs snapshot test_source1/fs1@base")
shelltest("zfs clone test_source1/fs1@base test_source1/fs1_clone")
shelltest("zfs set autobackup:test=true test_source1/fs1_clone")
before = shelltest("zfs list -H -o name -r -t snapshot " + TEST_POOLS)
with OutputIO() as buf_out, OutputIO() as buf_err:
with redirect_stdout(buf_out), redirect_stderr(buf_err):
with mocktime("20101111000000"):
self.assertFalse(ZfsAutobackup(
"test test_target1 --no-progress --verbose --test".split(" ")).run())
combined = buf_out.getvalue() + buf_err.getvalue()
print(combined)
self.assertIn("Cannot replicate as clone", combined)
self.assertNotIn("STDERR", combined)
after = shelltest("zfs list -H -o name -r -t snapshot " + TEST_POOLS)
self.assertMultiLineEqual(before, after)
with OutputIO() as buf_out, OutputIO() as buf_err:
with redirect_stdout(buf_out), redirect_stderr(buf_err):
with mocktime("20101111000000"):
self.assertFalse(ZfsAutobackup(
"test test_target1 --no-progress --verbose".split(" ")).run())
combined = buf_out.getvalue() + buf_err.getvalue()
print(combined)
self.assertIn("Cannot replicate as clone", combined)
self.assertNotIn("STDERR", combined)
origin = shelltest("zfs get -H -o value origin test_target1/test_source1/fs1_clone").strip()
self.assertEqual(origin, "-")
def test_clone_after_initial_origin_backup(self):
"""First run backs up the origin; second run replicates a clone of it."""
with mocktime("20101111000000"):
self.assertFalse(ZfsAutobackup(
"test test_target1 --no-progress --verbose".split(" ")).run())
shelltest("zfs clone test_source1/fs1@test-20101111000000 test_source1/fs1_clone")
shelltest("zfs set autobackup:test=true test_source1/fs1_clone")
before = shelltest("zfs list -H -o name -r -t snapshot " + TEST_POOLS)
with OutputIO() as buf_out, OutputIO() as buf_err:
with redirect_stdout(buf_out), redirect_stderr(buf_err):
with mocktime("20101111000001"):
self.assertFalse(ZfsAutobackup(
"test test_target1 --no-progress --verbose --allow-empty --test".split(" ")).run())
combined = buf_out.getvalue() + buf_err.getvalue()
print(combined)
self.assertNotIn("STDERR", combined)
after = shelltest("zfs list -H -o name -r -t snapshot " + TEST_POOLS)
self.assertMultiLineEqual(before, after)
with mocktime("20101111000001"):
self.assertFalse(ZfsAutobackup(
"test test_target1 --no-progress --verbose --allow-empty".split(" ")).run())
origin = shelltest("zfs get -H -o value origin test_target1/test_source1/fs1_clone").strip()
self.assertEqual(origin, "test_target1/test_source1/fs1@test-20101111000000")
def test_no_clone_flag_disables_clone_replication(self):
"""--no-clone reverts to the previous full-send behaviour for clones."""
shelltest("zfs snapshot test_source1/fs1@base")
shelltest("zfs clone test_source1/fs1@base test_source1/fs1_clone")
shelltest("zfs set autobackup:test=true test_source1/fs1_clone")
before = shelltest("zfs list -H -o name -r -t snapshot " + TEST_POOLS)
with OutputIO() as buf_out, OutputIO() as buf_err:
with redirect_stdout(buf_out), redirect_stderr(buf_err):
with mocktime("20101111000000"):
self.assertFalse(ZfsAutobackup(
"test test_target1 --no-progress --verbose --no-clone --test".split(" ")).run())
combined = buf_out.getvalue() + buf_err.getvalue()
print(combined)
self.assertNotIn("STDERR", combined)
after = shelltest("zfs list -H -o name -r -t snapshot " + TEST_POOLS)
self.assertMultiLineEqual(before, after)
with mocktime("20101111000000"):
self.assertFalse(ZfsAutobackup(
"test test_target1 --no-progress --verbose --no-clone".split(" ")).run())
origin = shelltest("zfs get -H -o value origin test_target1/test_source1/fs1_clone").strip()
self.assertEqual(origin, "-")
def test_subsequent_incremental_after_clone_send(self):
"""A second run after a clone has been sent is a plain incremental."""
shelltest("zfs snapshot test_source1/fs1@base")
shelltest("zfs clone test_source1/fs1@base test_source1/fs1_clone")
shelltest("zfs set autobackup:test=true test_source1/fs1_clone")
with mocktime("20101111000000"):
self.assertFalse(ZfsAutobackup(
"test test_target1 --no-progress --verbose".split(" ")).run())
before = shelltest("zfs list -H -o name -r -t snapshot " + TEST_POOLS)
with OutputIO() as buf_out, OutputIO() as buf_err:
with redirect_stdout(buf_out), redirect_stderr(buf_err):
with mocktime("20101111000001"):
self.assertFalse(ZfsAutobackup(
"test test_target1 --no-progress --verbose --allow-empty --test".split(" ")).run())
combined = buf_out.getvalue() + buf_err.getvalue()
print(combined)
self.assertNotIn("STDERR", combined)
after = shelltest("zfs list -H -o name -r -t snapshot " + TEST_POOLS)
self.assertMultiLineEqual(before, after)
with mocktime("20101111000001"):
self.assertFalse(ZfsAutobackup(
"test test_target1 --no-progress --verbose --allow-empty".split(" ")).run())
r = shelltest("zfs list -H -o name -t snapshot test_target1/test_source1/fs1_clone")
self.assertMultiLineEqual(r, """
test_target1/test_source1/fs1_clone@test-20101111000000
test_target1/test_source1/fs1_clone@test-20101111000001
""")
origin = shelltest("zfs get -H -o value origin test_target1/test_source1/fs1_clone").strip()
self.assertEqual(origin, "test_target1/test_source1/fs1@base")
def test_topological_sort_processes_origin_before_clone(self):
"""When the clone alphabetically precedes its origin, the topo sort still
processes the origin first so the clone can pin to its target origin."""
shelltest("zfs snapshot test_source1/fs1@base")
shelltest("zfs clone test_source1/fs1@base test_source1/aaa_clone")
shelltest("zfs set autobackup:test=true test_source1/aaa_clone")
before = shelltest("zfs list -H -o name -r -t snapshot " + TEST_POOLS)
with OutputIO() as buf_out, OutputIO() as buf_err:
with redirect_stdout(buf_out), redirect_stderr(buf_err):
with mocktime("20101111000000"):
self.assertFalse(ZfsAutobackup(
"test test_target1 --no-progress --verbose --test".split(" ")).run())
combined = buf_out.getvalue() + buf_err.getvalue()
print(combined)
self.assertNotIn("STDERR", combined)
after = shelltest("zfs list -H -o name -r -t snapshot " + TEST_POOLS)
self.assertMultiLineEqual(before, after)
with mocktime("20101111000000"):
self.assertFalse(ZfsAutobackup(
"test test_target1 --no-progress --verbose".split(" ")).run())
origin = shelltest("zfs get -H -o value origin test_target1/test_source1/aaa_clone").strip()
self.assertEqual(origin, "test_target1/test_source1/fs1@base")
def test_test_mode_makes_no_changes_for_clones(self):
"""--test must not modify anything, even when clone handling is exercised."""
shelltest("zfs snapshot test_source1/fs1@base")
shelltest("zfs clone test_source1/fs1@base test_source1/fs1_clone")
shelltest("zfs set autobackup:test=true test_source1/fs1_clone")
with OutputIO() as buf_out, OutputIO() as buf_err:
with redirect_stdout(buf_out), redirect_stderr(buf_err):
with mocktime("20101111000000"):
self.assertFalse(ZfsAutobackup(
"test test_target1 --no-progress --verbose --test".split(" ")).run())
combined = buf_out.getvalue() + buf_err.getvalue()
print(combined)
self.assertNotIn("STDERR", combined)
r = shelltest("zfs list -H -o name -r -t snapshot,filesystem test_target1")
self.assertMultiLineEqual(r, """
test_target1
""")
def test_promoted_clone_origin_on_namespace_descendant_warns_and_falls_back(self):
"""After 'zfs promote', the original parent becomes a clone whose origin lives
on one of its own namespace descendants. zfs recv can't land a clone-creating
stream on top of the placeholder we have to create for that descendant, so we
detect this upfront, warn, and fall back to a full send (which will itself
fail at the placeholder — user can destroy target and rerun, or promote the
source dataset to flip the lineage back)."""
shelltest("zfs snapshot test_source1/fs1@s1")
shelltest("zfs clone test_source1/fs1@s1 test_source1/fs1/inner_clone")
shelltest("zfs promote test_source1/fs1/inner_clone")
origin = shelltest("zfs get -H -o value origin test_source1/fs1").strip()
self.assertEqual(origin, "test_source1/fs1/inner_clone@s1")
before = shelltest("zfs list -H -o name -r -t snapshot " + TEST_POOLS)
with OutputIO() as buf_out, OutputIO() as buf_err:
with redirect_stdout(buf_out), redirect_stderr(buf_err):
with mocktime("20101111000000"):
result = ZfsAutobackup(
"test test_target1 --no-progress --verbose --test -F".split(" ")).run()
combined = buf_out.getvalue() + buf_err.getvalue()
print(combined)
self.assertFalse(result)
self.assertIn("lives on a namespace descendant", combined)
self.assertNotIn("STDERR", combined)
after = shelltest("zfs list -H -o name -r -t snapshot " + TEST_POOLS)
self.assertMultiLineEqual(before, after)
with OutputIO() as buf_out, OutputIO() as buf_err:
with redirect_stdout(buf_out), redirect_stderr(buf_err):
with mocktime("20101111000000"):
result = ZfsAutobackup(
"test test_target1 --no-progress --verbose -F".split(" ")).run()
combined = buf_out.getvalue() + buf_err.getvalue()
print(combined)
self.assertFalse(result)
self.assertIn("lives on a namespace descendant", combined)
+44 -43
View File
@@ -1,4 +1,3 @@
from basetest import *
@@ -6,125 +5,127 @@ class TestZfsNode(unittest2.TestCase):
def setUp(self):
prepare_zpools()
self.longMessage=True
self.longMessage = True
def test_destroymissing(self):
#initial backup
with mocktime("19101111000000"): #1000 years in past
def test_destroymissing(self):
# initial backup
with mocktime("19101111000000"): # 1000 years in past
self.assertFalse(ZfsAutobackup("test test_target1 --no-progress --verbose --no-holds".split(" ")).run())
with mocktime("20101111000000"): #far in past
self.assertFalse(ZfsAutobackup("test test_target1 --no-progress --verbose --no-holds --allow-empty".split(" ")).run())
with mocktime("20101111000000"): # far in past
self.assertFalse(
ZfsAutobackup("test test_target1 --no-progress --verbose --no-holds --allow-empty".split(" ")).run())
with self.subTest("Should do nothing yet"):
with OutputIO() as buf:
with redirect_stdout(buf):
self.assertFalse(ZfsAutobackup("test test_target1 --no-progress --verbose --no-snapshot --destroy-missing 0s".split(" ")).run())
self.assertFalse(ZfsAutobackup(
"test test_target1 --no-progress --verbose --no-snapshot --destroy-missing 0s".split(
" ")).run())
print(buf.getvalue())
self.assertNotIn(": Destroy missing", buf.getvalue())
with self.subTest("missing dataset of us that still has children"):
#just deselect it so it counts as 'missing'
# just deselect it so it counts as 'missing'
shelltest("zfs set autobackup:test=child test_source1/fs1")
with OutputIO() as buf:
with redirect_stdout(buf), redirect_stderr(buf):
self.assertFalse(ZfsAutobackup("test test_target1 --no-progress --verbose --no-snapshot --destroy-missing 0s".split(" ")).run())
self.assertFalse(ZfsAutobackup(
"test test_target1 --no-progress --verbose --no-snapshot --destroy-missing 0s".split(
" ")).run())
print(buf.getvalue())
#should have done the snapshot cleanup for destoy missing:
# should have done the snapshot cleanup for destoy missing:
self.assertIn("fs1@test-19101111000000: Destroying", buf.getvalue())
self.assertIn("fs1: Destroy missing: Still has children here.", buf.getvalue())
shelltest("zfs inherit autobackup:test test_source1/fs1")
with self.subTest("Normal destroyed leaf"):
shelltest("zfs destroy -r test_source1/fs1/sub")
#wait for deadline of last snapshot
# wait for deadline of last snapshot
with OutputIO() as buf:
with redirect_stdout(buf):
#100y: lastest should not be old enough, while second to latest snapshot IS old enough:
self.assertFalse(ZfsAutobackup("test test_target1 --no-progress --verbose --no-snapshot --destroy-missing 100y".split(" ")).run())
# 100y: lastest should not be old enough, while second to latest snapshot IS old enough:
self.assertFalse(ZfsAutobackup(
"test test_target1 --no-progress --verbose --no-snapshot --destroy-missing 100y".split(
" ")).run())
print(buf.getvalue())
self.assertIn(": Waiting for deadline", buf.getvalue())
#past deadline, destroy
# past deadline, destroy
with OutputIO() as buf:
with redirect_stdout(buf):
self.assertFalse(ZfsAutobackup("test test_target1 --no-progress --verbose --no-snapshot --destroy-missing 1y".split(" ")).run())
self.assertFalse(ZfsAutobackup(
"test test_target1 --no-progress --verbose --no-snapshot --destroy-missing 1y".split(
" ")).run())
print(buf.getvalue())
self.assertIn("sub: Destroying", buf.getvalue())
with self.subTest("Leaf with other snapshot still using it"):
shelltest("zfs destroy -r test_source1/fs1")
shelltest("zfs snapshot -r test_target1/test_source1/fs1@other1")
with OutputIO() as buf:
with redirect_stdout(buf):
self.assertFalse(ZfsAutobackup("test test_target1 --no-progress --verbose --no-snapshot --destroy-missing 0s".split(" ")).run())
self.assertFalse(ZfsAutobackup(
"test test_target1 --no-progress --verbose --no-snapshot --destroy-missing 0s".split(
" ")).run())
print(buf.getvalue())
#cant finish because still in use:
# cant finish because still in use:
self.assertIn("fs1: Destroy missing: Still in use", buf.getvalue())
shelltest("zfs destroy test_target1/test_source1/fs1@other1")
with self.subTest("In use by clone"):
shelltest("zfs clone test_target1/test_source1/fs1@test-20101111000000 test_target1/clone1")
with OutputIO() as buf:
with redirect_stdout(buf), redirect_stderr(buf):
self.assertFalse(ZfsAutobackup("test test_target1 --no-progress --verbose --no-snapshot --destroy-missing 0s".split(" ")).run())
self.assertFalse(ZfsAutobackup(
"test test_target1 --no-progress --verbose --no-snapshot --destroy-missing 0s".split(
" ")).run())
print(buf.getvalue())
#now tries to destroy our own last snapshot (before the final destroy of the dataset)
# now tries to destroy our own last snapshot (before the final destroy of the dataset)
self.assertIn("fs1@test-20101111000000: Destroying", buf.getvalue())
#but cant finish because still in use:
# but cant finish because still in use:
self.assertIn("fs1: Error during --destroy-missing", buf.getvalue())
shelltest("zfs destroy test_target1/clone1")
with self.subTest("Should leave test_source1 parent"):
with OutputIO() as buf:
with redirect_stdout(buf), redirect_stderr(buf):
self.assertFalse(ZfsAutobackup("test test_target1 --no-progress --verbose --no-snapshot --destroy-missing 0s".split(" ")).run())
self.assertFalse(ZfsAutobackup(
"test test_target1 --no-progress --verbose --no-snapshot --destroy-missing 0s".split(
" ")).run())
print(buf.getvalue())
#should have done the snapshot cleanup for destoy missing:
# should have done the snapshot cleanup for destoy missing:
self.assertIn("fs1: Destroying", buf.getvalue())
with OutputIO() as buf:
with redirect_stdout(buf), redirect_stderr(buf):
self.assertFalse(ZfsAutobackup("test test_target1 --no-progress --verbose --no-snapshot --destroy-missing 0s".split(" ")).run())
self.assertFalse(ZfsAutobackup(
"test test_target1 --no-progress --verbose --no-snapshot --destroy-missing 0s".split(
" ")).run())
print(buf.getvalue())
#on second run it sees the dangling ex-parent but doesnt know what to do with it (since it has no own snapshot)
# on second run it sees the dangling ex-parent but doesnt know what to do with it (since it has no own snapshot)
self.assertIn("test_source1: Destroy missing: has no snapshots made by us", buf.getvalue())
#end result
r=shelltest("zfs list -H -o name -r -t all test_target1")
self.assertMultiLineEqual(r,"""
# end result
r = shelltest("zfs list -H -o name -r -t snapshot,filesystem test_target1")
self.assertMultiLineEqual(r, """
test_target1
test_target1/test_source1
test_target1/test_source2
+94 -73
View File
@@ -2,6 +2,7 @@ from zfs_autobackup.CmdPipe import CmdPipe
from basetest import *
import time
# We have to do a LOT to properly test encryption/decryption/raw transfers
#
# For every scenario we need at least:
@@ -20,7 +21,6 @@ import time
class TestZfsEncryption(unittest2.TestCase):
def setUp(self):
prepare_zpools()
@@ -40,7 +40,9 @@ class TestZfsEncryption(unittest2.TestCase):
# create encrypted source dataset
shelltest("rm /tmp/zfstest.key 2>/dev/null;true")
shelltest("echo {} > /tmp/zfstest.key".format(key))
shelltest("zfs create -o keylocation=file:///tmp/zfstest.key -o keyformat=passphrase -o encryption=on {}".format(path))
shelltest(
"zfs create -o keylocation=file:///tmp/zfstest.key -o keyformat=passphrase -o encryption=on {}".format(
path))
if unload_key:
shelltest("zfs unmount {}".format(path))
@@ -48,23 +50,30 @@ class TestZfsEncryption(unittest2.TestCase):
# r=shelltest("dd if=/dev/zero of=/test_source1/fs1/enc1/data.txt bs=200000 count=1")
def test_raw(self):
def test_raw(self):
"""send encrypted data unaltered (standard operation)"""
self.prepare_encrypted_dataset("11111111", "test_source1/fs1/encryptedsource")
self.prepare_encrypted_dataset("11111111", "test_source1/fs1/encryptedsourcekeyless", unload_key=True) # raw mode shouldn't need a key
self.prepare_encrypted_dataset("11111111", "test_source1/fs1/encryptedsourcekeyless",
unload_key=True) # raw mode shouldn't need a key
self.prepare_encrypted_dataset("22222222", "test_target1/encryptedtarget")
with mocktime("20101111000000"):
self.assertFalse(ZfsAutobackup("test test_target1 --verbose --no-progress --allow-empty --exclude-received".split(" ")).run())
self.assertFalse(ZfsAutobackup("test test_target1/encryptedtarget --verbose --no-progress --no-snapshot --exclude-received".split(" ")).run())
self.assertFalse(ZfsAutobackup(
"test test_target1 --verbose --no-progress --allow-empty".split(" ")).run())
self.assertFalse(ZfsAutobackup(
"test test_target1/encryptedtarget --verbose --no-progress --no-snapshot".split(
" ")).run())
with mocktime("20101111000001"):
self.assertFalse(ZfsAutobackup("test test_target1 --verbose --no-progress --allow-empty --exclude-received".split(" ")).run())
self.assertFalse(ZfsAutobackup("test test_target1/encryptedtarget --verbose --no-progress --no-snapshot --exclude-received".split(" ")).run())
self.assertFalse(ZfsAutobackup(
"test test_target1 --verbose --no-progress --allow-empty".split(" ")).run())
self.assertFalse(ZfsAutobackup(
"test test_target1/encryptedtarget --verbose --no-progress --no-snapshot".split(
" ")).run())
r = shelltest("zfs get -r -t filesystem encryptionroot test_target1")
self.assertMultiLineEqual(r,"""
self.assertMultiLineEqual(r, """
NAME PROPERTY VALUE SOURCE
test_target1 encryptionroot - -
test_target1/encryptedtarget encryptionroot test_target1/encryptedtarget -
@@ -86,19 +95,28 @@ test_target1/test_source2/fs2 encryption
test_target1/test_source2/fs2/sub encryptionroot - -
""")
def test_decrypt(self):
def test_decrypt(self):
"""decrypt data and store unencrypted (--decrypt)"""
self.prepare_encrypted_dataset("11111111", "test_source1/fs1/encryptedsource")
self.prepare_encrypted_dataset("22222222", "test_target1/encryptedtarget")
with mocktime("20101111000000"):
self.assertFalse(ZfsAutobackup("test test_target1 --verbose --no-progress --decrypt --allow-empty --exclude-received".split(" ")).run())
self.assertFalse(ZfsAutobackup("test test_target1/encryptedtarget --verbose --no-progress --decrypt --no-snapshot --exclude-received".split(" ")).run())
self.assertFalse(ZfsAutobackup(
"test test_target1 --verbose --no-progress --decrypt --allow-empty".split(
" ")).run())
# NOTE: this also tests if sending a source to a second target works correctly (this failed during bookmark implementation)
self.assertFalse(ZfsAutobackup(
"test test_target1/encryptedtarget --verbose --no-progress --decrypt --no-snapshot --debug".split(
" ")).run())
with mocktime("20101111000001"):
self.assertFalse(ZfsAutobackup("test test_target1 --verbose --no-progress --decrypt --allow-empty --exclude-received".split(" ")).run())
self.assertFalse(ZfsAutobackup("test test_target1/encryptedtarget --verbose --no-progress --decrypt --no-snapshot --exclude-received".split(" ")).run())
self.assertFalse(ZfsAutobackup(
"test test_target1 --verbose --no-progress --decrypt --allow-empty".split(
" ")).run())
self.assertFalse(ZfsAutobackup(
"test test_target1/encryptedtarget --verbose --no-progress --decrypt --no-snapshot".split(
" ")).run())
r = shelltest("zfs get -r -t filesystem encryptionroot test_target1")
self.assertEqual(r, """
@@ -121,19 +139,27 @@ test_target1/test_source2/fs2 encryptionroot -
test_target1/test_source2/fs2/sub encryptionroot - -
""")
def test_encrypt(self):
def test_encrypt(self):
"""send normal data set and store encrypted on the other side (--encrypt) issue #60 """
self.prepare_encrypted_dataset("11111111", "test_source1/fs1/encryptedsource")
self.prepare_encrypted_dataset("22222222", "test_target1/encryptedtarget")
with mocktime("20101111000000"):
self.assertFalse(ZfsAutobackup("test test_target1 --verbose --no-progress --encrypt --debug --allow-empty --exclude-received --clear-mountpoint".split(" ")).run())
self.assertFalse(ZfsAutobackup("test test_target1/encryptedtarget --verbose --no-progress --encrypt --debug --no-snapshot --exclude-received --clear-mountpoint".split(" ")).run())
self.assertFalse(ZfsAutobackup(
"test test_target1 --verbose --no-progress --encrypt --debug --allow-empty --clear-mountpoint".split(
" ")).run())
self.assertFalse(ZfsAutobackup(
"test test_target1/encryptedtarget --verbose --no-progress --encrypt --debug --no-snapshot --clear-mountpoint".split(
" ")).run())
with mocktime("20101111000001"):
self.assertFalse(ZfsAutobackup("test test_target1 --verbose --no-progress --encrypt --debug --allow-empty --exclude-received --clear-mountpoint".split(" ")).run())
self.assertFalse(ZfsAutobackup("test test_target1/encryptedtarget --verbose --no-progress --encrypt --debug --no-snapshot --exclude-received --clear-mountpoint".split(" ")).run())
self.assertFalse(ZfsAutobackup(
"test test_target1 --verbose --no-progress --encrypt --debug --allow-empty --clear-mountpoint".split(
" ")).run())
self.assertFalse(ZfsAutobackup(
"test test_target1/encryptedtarget --verbose --no-progress --encrypt --debug --no-snapshot --clear-mountpoint".split(
" ")).run())
r = shelltest("zfs get -r -t filesystem encryptionroot test_target1")
self.assertEqual(r, """
@@ -164,16 +190,18 @@ test_target1/test_source2/fs2/sub encryptionroot -
with mocktime("20101111000000"):
self.assertFalse(ZfsAutobackup(
"test test_target1 --verbose --no-progress --decrypt --encrypt --debug --allow-empty --exclude-received --clear-mountpoint".split(" ")).run())
"test test_target1 --verbose --no-progress --decrypt --encrypt --debug --allow-empty --clear-mountpoint".split(
" ")).run())
self.assertFalse(ZfsAutobackup(
"test test_target1/encryptedtarget --verbose --no-progress --decrypt --encrypt --debug --no-snapshot --exclude-received --clear-mountpoint".split(
"test test_target1/encryptedtarget --verbose --no-progress --decrypt --encrypt --debug --no-snapshot --clear-mountpoint".split(
" ")).run())
with mocktime("20101111000001"):
self.assertFalse(ZfsAutobackup(
"test test_target1 --verbose --no-progress --decrypt --encrypt --debug --allow-empty --exclude-received".split(" ")).run())
"test test_target1 --verbose --no-progress --decrypt --encrypt --debug --allow-empty".split(
" ")).run())
self.assertFalse(ZfsAutobackup(
"test test_target1/encryptedtarget --verbose --no-progress --decrypt --encrypt --debug --no-snapshot --exclude-received".split(
"test test_target1/encryptedtarget --verbose --no-progress --decrypt --encrypt --debug --no-snapshot".split(
" ")).run())
r = shelltest("zfs get -r -t filesystem encryptionroot test_target1")
@@ -197,10 +225,7 @@ test_target1/test_source2/fs2 encryptionroot -
test_target1/test_source2/fs2/sub encryptionroot - -
""")
def test_raw_invalid_snapshot(self):
def test_raw_invalid_snapshot(self):
"""in raw mode, its not allowed to have any newer snaphots on target, #219"""
self.prepare_encrypted_dataset("11111111", "test_source1/fs1/encryptedsource")
@@ -208,18 +233,19 @@ test_target1/test_source2/fs2/sub encryptionroot -
with mocktime("20101111000000"):
self.assertFalse(ZfsAutobackup("test test_target1 --verbose --no-progress".split(" ")).run())
#this is invalid in raw mode
# this is invalid in raw mode
shelltest("zfs snapshot test_target1/test_source1/fs1/encryptedsource@incompatible")
with mocktime("20101111000001"):
#should fail because of incompatble snapshot
self.assertEqual(ZfsAutobackup("test test_target1 --verbose --no-progress --allow-empty".split(" ")).run(),1)
#should destroy incompatible and continue
self.assertFalse(ZfsAutobackup("test test_target1 --verbose --no-progress --no-snapshot --destroy-incompatible".split(" ")).run())
# should fail because of incompatble snapshot
self.assertEqual(ZfsAutobackup("test test_target1 --verbose --no-progress --allow-empty".split(" ")).run(),
1)
# should destroy incompatible and continue
self.assertFalse(ZfsAutobackup(
"test test_target1 --verbose --no-progress --no-snapshot --destroy-incompatible".split(" ")).run())
r = shelltest("zfs get -r -t filesystem encryptionroot test_target1")
self.assertMultiLineEqual(r,"""
self.assertMultiLineEqual(r, """
NAME PROPERTY VALUE SOURCE
test_target1 encryptionroot - -
test_target1/test_source1 encryptionroot - -
@@ -231,16 +257,16 @@ test_target1/test_source2/fs2 encryptionroot -
test_target1/test_source2/fs2/sub encryptionroot - -
""")
def test_resume_encrypt_with_no_key(self):
def test_resume_encrypt_with_no_key(self):
"""test what happens if target encryption key not loaded (this led to a kernel crash of freebsd with 2.1.x i think) while trying to resume"""
self.prepare_encrypted_dataset("11111111", "test_source1/fs1/encryptedsource")
self.prepare_encrypted_dataset("22222222", "test_target1/encryptedtarget")
with mocktime("20101111000000"):
self.assertFalse(ZfsAutobackup("test test_target1/encryptedtarget --verbose --no-progress --encrypt --allow-empty --exclude-received --clear-mountpoint".split(" ")).run())
self.assertFalse(ZfsAutobackup(
"test test_target1/encryptedtarget --verbose --no-progress --encrypt --allow-empty --clear-mountpoint".split(
" ")).run())
r = shelltest("zfs set compress=off test_source1 test_target1")
@@ -253,7 +279,7 @@ test_target1/test_source2/fs2/sub encryptionroot -
# should fail and leave resume token
with mocktime("20101111000001"):
self.assertTrue(ZfsAutobackup(
"test test_target1/encryptedtarget --verbose --no-progress --encrypt --exclude-received --allow-empty --clear-mountpoint".split(
"test test_target1/encryptedtarget --verbose --no-progress --encrypt --allow-empty --clear-mountpoint".split(
" ")).run())
#
# free up space
@@ -270,44 +296,39 @@ test_target1/test_source2/fs2/sub encryptionroot -
# resume should fail
with mocktime("20101111000001"):
self.assertEqual(ZfsAutobackup(
"test test_target1/encryptedtarget --verbose --no-progress --encrypt --exclude-received --allow-empty --no-snapshot --clear-mountpoint".split(
" ")).run(),3)
"test test_target1/encryptedtarget --verbose --no-progress --encrypt --allow-empty --no-snapshot --clear-mountpoint".split(
" ")).run(), 3)
# NOTE: On some versions this leaves 2 weird sub-datasets that should'nt be there (its probably a zfs bug?)
# so we ignore this, and just make sure the backup resumes correctly after reloading the key.
# r = shelltest("zfs get -r -t snapshot,filesystem encryptionroot test_target1")
# self.assertEqual(r, """
# NAME PROPERTY VALUE SOURCE
# test_target1 encryptionroot - -
# test_target1/encryptedtarget encryptionroot test_target1/encryptedtarget -
# test_target1/encryptedtarget/test_source1 encryptionroot test_target1/encryptedtarget -
# test_target1/encryptedtarget/test_source1/fs1 encryptionroot test_target1/encryptedtarget -
# test_target1/encryptedtarget/test_source1/fs1@test-20101111000000 encryptionroot test_target1/encryptedtarget -
# test_target1/encryptedtarget/test_source1/fs1/encryptedsource encryptionroot test_target1/encryptedtarget/test_source1/fs1/encryptedsource -
# test_target1/encryptedtarget/test_source1/fs1/encryptedsource@test-20101111000000 encryptionroot test_target1/encryptedtarget/test_source1/fs1/encryptedsource -
# test_target1/encryptedtarget/test_source1/fs1/encryptedsource@test-20101111000001 encryptionroot test_target1/encryptedtarget/test_source1/fs1/encryptedsource -
# test_target1/encryptedtarget/test_source1/fs1/sub encryptionroot test_target1/encryptedtarget -
# test_target1/encryptedtarget/test_source1/fs1/sub@test-20101111000000 encryptionroot test_target1/encryptedtarget -
# test_target1/encryptedtarget/test_source1/fs1/sub/sub encryptionroot - -
# test_target1/encryptedtarget/test_source1/fs1/sub/sub@test-20101111000001 encryptionroot - -
# test_target1/encryptedtarget/test_source2 encryptionroot test_target1/encryptedtarget -
# test_target1/encryptedtarget/test_source2/fs2 encryptionroot test_target1/encryptedtarget -
# test_target1/encryptedtarget/test_source2/fs2/sub encryptionroot test_target1/encryptedtarget -
# test_target1/encryptedtarget/test_source2/fs2/sub@test-20101111000000 encryptionroot test_target1/encryptedtarget -
# test_target1/encryptedtarget/test_source2/fs2/sub/sub encryptionroot - -
# test_target1/encryptedtarget/test_source2/fs2/sub/sub@test-20101111000001 encryptionroot - -
# """)
#NOTE: On some versions this leaves 2 weird sub-datasets that should'nt be there (its probably a zfs bug?)
#so we ignore this, and just make sure the backup resumes correctly after reloading the key.
# r = shelltest("zfs get -r -t all encryptionroot test_target1")
# self.assertEqual(r, """
# NAME PROPERTY VALUE SOURCE
# test_target1 encryptionroot - -
# test_target1/encryptedtarget encryptionroot test_target1/encryptedtarget -
# test_target1/encryptedtarget/test_source1 encryptionroot test_target1/encryptedtarget -
# test_target1/encryptedtarget/test_source1/fs1 encryptionroot test_target1/encryptedtarget -
# test_target1/encryptedtarget/test_source1/fs1@test-20101111000000 encryptionroot test_target1/encryptedtarget -
# test_target1/encryptedtarget/test_source1/fs1/encryptedsource encryptionroot test_target1/encryptedtarget/test_source1/fs1/encryptedsource -
# test_target1/encryptedtarget/test_source1/fs1/encryptedsource@test-20101111000000 encryptionroot test_target1/encryptedtarget/test_source1/fs1/encryptedsource -
# test_target1/encryptedtarget/test_source1/fs1/encryptedsource@test-20101111000001 encryptionroot test_target1/encryptedtarget/test_source1/fs1/encryptedsource -
# test_target1/encryptedtarget/test_source1/fs1/sub encryptionroot test_target1/encryptedtarget -
# test_target1/encryptedtarget/test_source1/fs1/sub@test-20101111000000 encryptionroot test_target1/encryptedtarget -
# test_target1/encryptedtarget/test_source1/fs1/sub/sub encryptionroot - -
# test_target1/encryptedtarget/test_source1/fs1/sub/sub@test-20101111000001 encryptionroot - -
# test_target1/encryptedtarget/test_source2 encryptionroot test_target1/encryptedtarget -
# test_target1/encryptedtarget/test_source2/fs2 encryptionroot test_target1/encryptedtarget -
# test_target1/encryptedtarget/test_source2/fs2/sub encryptionroot test_target1/encryptedtarget -
# test_target1/encryptedtarget/test_source2/fs2/sub@test-20101111000000 encryptionroot test_target1/encryptedtarget -
# test_target1/encryptedtarget/test_source2/fs2/sub/sub encryptionroot - -
# test_target1/encryptedtarget/test_source2/fs2/sub/sub@test-20101111000001 encryptionroot - -
# """)
#reload key and resume correctly.
# reload key and resume correctly.
self.load_key("22222222", "test_target1/encryptedtarget")
# resume should complete
with mocktime("20101111000001"):
self.assertEqual(ZfsAutobackup(
"test test_target1/encryptedtarget --verbose --no-progress --encrypt --exclude-received --allow-empty --no-snapshot --clear-mountpoint".split(
" ")).run(),0)
"test test_target1/encryptedtarget --verbose --no-progress --encrypt --allow-empty --no-snapshot --clear-mountpoint --debug".split(
" ")).run(), 0)
+19
View File
@@ -202,6 +202,25 @@ class TestExecuteNode(unittest2.TestCase):
self.assertEqual(result, ["tEst"])
def test_script_remote_ssh_config_flat_argv(self):
"""script() with ssh_to + ssh_config must build a flat argv (no nested lists).
Regression: previously used cmd.append(["-F", ssh_config]) which embedded a
list inside the argv and broke subprocess.Popen when actually executing.
"""
node = ExecuteNode(ssh_to="user@somehost", ssh_config="/tmp/some_ssh_config")
cmd_pipe = node.script(lines=["echo test"], pipe=True)
argv = cmd_pipe.items[0].cmd
for i, arg in enumerate(argv):
self.assertIsInstance(arg, str, msg="argv[{}] should be str, got {!r}".format(i, arg))
# also verify the -F flag and value are present and adjacent
self.assertIn("-F", argv)
f_index = argv.index("-F")
self.assertEqual(argv[f_index + 1], "/tmp/some_ssh_config")
+22 -31
View File
@@ -10,7 +10,6 @@ class TestExternalFailures(unittest2.TestCase):
# generate a resumable state
# NOTE: this generates two resumable test_target1/test_source1/fs1 and test_target1/test_source1/fs1/sub
def generate_resume(self):
r = shelltest("zfs set compress=off test_source1 test_target1")
# big change on source
@@ -30,12 +29,11 @@ class TestExternalFailures(unittest2.TestCase):
# resume initial backup
def test_initial_resume(self):
# inital backup, leaves resume token
with mocktime("20101111000000"):
self.generate_resume()
# --test should resume and succeed
# test should resume and succeed
with OutputIO() as buf:
with redirect_stdout(buf):
self.assertFalse(ZfsAutobackup("test test_target1 --no-progress --verbose --test".split(" ")).run())
@@ -53,7 +51,7 @@ class TestExternalFailures(unittest2.TestCase):
self.assertIn(": resuming", buf.getvalue())
r = shelltest("zfs list -H -o name -r -t all test_target1")
r = shelltest("zfs list -H -o name -r -t snapshot,filesystem test_target1")
self.assertMultiLineEqual(r, """
test_target1
test_target1/test_source1
@@ -69,10 +67,10 @@ test_target1/test_source2/fs2/sub@test-20101111000000
# resume incremental backup
def test_incremental_resume(self):
# initial backup
with mocktime("20101111000000"):
self.assertFalse(ZfsAutobackup("test test_target1 --no-progress --verbose --allow-empty".split(" ")).run())
self.assertFalse(ZfsAutobackup(
"test test_target1 --no-progress --verbose --allow-empty".split(" ")).run())
# incremental backup leaves resume token
with mocktime("20101111000001"):
@@ -97,7 +95,7 @@ test_target1/test_source2/fs2/sub@test-20101111000000
# did we really resume?
self.assertIn(": resuming", buf.getvalue())
r = shelltest("zfs list -H -o name -r -t all test_target1")
r = shelltest("zfs list -H -o name -r -t snapshot,filesystem test_target1")
self.assertMultiLineEqual(r, """
test_target1
test_target1/test_source1
@@ -113,9 +111,7 @@ test_target1/test_source2/fs2/sub@test-20101111000000
""")
# generate an invalid resume token, and verify if its aborted automaticly
def test_initial_resumeabort(self):
def test_initial_abortedresume(self):
# inital backup, leaves resume token
with mocktime("20101111000000"):
self.generate_resume()
@@ -123,8 +119,6 @@ test_target1/test_source2/fs2/sub@test-20101111000000
# remove corresponding source snapshot, so it becomes invalid
shelltest("zfs destroy test_source1/fs1@test-20101111000000")
# NOTE: it can only abort the initial dataset if it has no subs
shelltest("zfs destroy test_target1/test_source1/fs1/sub; true")
# --test try again, should abort old resume
with mocktime("20101111000001"):
@@ -134,7 +128,7 @@ test_target1/test_source2/fs2/sub@test-20101111000000
with mocktime("20101111000001"):
self.assertFalse(ZfsAutobackup("test test_target1 --no-progress --verbose".split(" ")).run())
r = shelltest("zfs list -H -o name -r -t all test_target1")
r = shelltest("zfs list -H -o name -r -t snapshot,filesystem test_target1")
self.assertMultiLineEqual(r, """
test_target1
test_target1/test_source1
@@ -149,8 +143,7 @@ test_target1/test_source2/fs2/sub@test-20101111000000
""")
# generate an invalid resume token, and verify if its aborted automaticly
def test_incremental_resumeabort(self):
def test_incremental_abortedresume(self):
# initial backup
with mocktime("20101111000000"):
self.assertFalse(ZfsAutobackup("test test_target1 --no-progress --verbose --allow-empty".split(" ")).run())
@@ -166,11 +159,12 @@ test_target1/test_source2/fs2/sub@test-20101111000000
with mocktime("20101111000002"):
self.assertFalse(ZfsAutobackup("test test_target1 --no-progress --verbose --test".split(" ")).run())
# try again, should abort old resume
with mocktime("20101111000002"):
self.assertFalse(ZfsAutobackup("test test_target1 --no-progress --verbose".split(" ")).run())
r = shelltest("zfs list -H -o name -r -t all test_target1")
r = shelltest("zfs list -H -o name -r -t snapshot,filesystem test_target1")
self.assertMultiLineEqual(r, """
test_target1
test_target1/test_source1
@@ -187,7 +181,6 @@ test_target1/test_source2/fs2/sub@test-20101111000000
# create a resume situation, where the other side doesnt want the snapshot anymore ( should abort resume )
def test_abort_unwanted_resume(self):
with mocktime("20101111000000"):
self.assertFalse(ZfsAutobackup("test test_target1 --no-progress --verbose".split(" ")).run())
@@ -200,13 +193,14 @@ test_target1/test_source2/fs2/sub@test-20101111000000
# incremental, doesnt want previous anymore
with mocktime("20101111000002"):
self.assertFalse(ZfsAutobackup(
"test test_target1 --no-progress --verbose --keep-target=0 --allow-empty --debug".split(" ")).run())
"test test_target1 --no-progress --verbose --keep-target=0 --allow-empty --debug".split(
" ")).run())
print(buf.getvalue())
self.assertIn("Aborting resume, we dont want that snapshot anymore.", buf.getvalue())
self.assertIn("Aborting resume", buf.getvalue())
r = shelltest("zfs list -H -o name -r -t all test_target1")
r = shelltest("zfs list -H -o name -r -t snapshot,filesystem test_target1")
self.assertMultiLineEqual(r, """
test_target1
test_target1/test_source1
@@ -222,7 +216,6 @@ test_target1/test_source2/fs2/sub@test-20101111000002
# test with empty snapshot list (this was a bug)
def test_abort_resume_emptysnapshotlist(self):
with mocktime("20101111000000"):
self.assertFalse(ZfsAutobackup("test test_target1 --no-progress --verbose".split(" ")).run())
@@ -240,15 +233,14 @@ test_target1/test_source2/fs2/sub@test-20101111000002
"test test_target1 --no-progress --verbose --no-snapshot".split(
" ")).run())
print(buf.getvalue())
self.assertIn("Aborting resume, its obsolete", buf.getvalue())
# NOTE: v3.4 doesnt abort the resume if there are no snapshots to send.
# print(buf.getvalue())
# self.assertIn("Aborting resume, its obsolete", buf.getvalue())
def test_missing_common(self):
with mocktime("20101111000000"):
self.assertFalse(ZfsAutobackup("test test_target1 --no-progress --verbose --allow-empty".split(" ")).run())
self.assertFalse(ZfsAutobackup(
"test test_target1 --no-progress --verbose --allow-empty --no-bookmarks".split(" ")).run())
# remove common snapshot and leave nothing
shelltest("zfs release zfs_autobackup:test test_source1/fs1@test-20101111000000")
@@ -257,10 +249,9 @@ test_target1/test_source2/fs2/sub@test-20101111000002
with mocktime("20101111000001"):
self.assertTrue(ZfsAutobackup("test test_target1 --no-progress --verbose --allow-empty".split(" ")).run())
#UPDATE: offcourse the one thing that wasn't tested had a bug :( (in ExecuteNode.run()).
# UPDATE: offcourse the one thing that wasn't tested had a bug :( (in ExecuteNode.run()).
def test_ignoretransfererrors(self):
self.skipTest("Not sure how to implement a test for this without some serious hacking and patching.")
self.skipTest("Not sure how to implement a test for this without some serious hacking and patching.")
# #recreate target pool without any features
# # shelltest("zfs set compress=on test_source1; zpool destroy test_target1; zpool create test_target1 -o feature@project_quota=disabled /dev/ram2")
@@ -268,7 +259,7 @@ test_target1/test_source2/fs2/sub@test-20101111000002
# with mocktime("20101111000000"):
# self.assertFalse(ZfsAutobackup("test test_target1 --verbose --allow-empty --no-progress".split(" ")).run())
#
# r = shelltest("zfs list -H -o name -r -t all test_target1")
# r = shelltest("zfs list -H -o name -r -t snapshot,filesystem test_target1")
#
# self.assertMultiLineEqual(r, """
# test_target1
+18
View File
@@ -24,6 +24,24 @@ class TestLog(unittest2.TestCase):
print(list(buf.getvalue()))
self.assertEqual(list(buf.getvalue()), ['\x1b', '[', '2', '2', 'm', ' ', ' ', 'v', 'e', 'r', 'b', 'o', 's', 'e', '\x1b', '[', '0', 'm', '\n', '\x1b', '[', '3', '2', 'm', '#', ' ', 'd', 'e', 'b', 'u', 'g', '\x1b', '[', '0', 'm', '\n', '\x1b', '[', '3', '1', 'm', '\x1b', '[', '1', 'm', '!', ' ', 'e', 'r', 'r', 'o', 'r', '\x1b', '[', '0', 'm', '\n'])
def test_progress_nocolor_no_crash(self):
"""progress() + clear_progress() must not crash when color=False.
Regression: clear_progress() used to do an unconditional `import colorama`
and call colorama.ansi.clear_line(), which fails if colorama isn't installed.
"""
with OutputIO() as buf:
with redirect_stderr(buf):
l = LogConsole(show_verbose=False, show_debug=False, color=False)
l.progress("working")
l.progress("still working") # triggers clear_progress on the previous line
l.clear_progress()
# write something visible after clearing
l.error("done")
self.assertIn("done", buf.getvalue())
def test_nocolor(self):
"""test without color output"""
+7 -11
View File
@@ -1,4 +1,3 @@
from basetest import *
@@ -6,27 +5,28 @@ class TestZfsNode(unittest2.TestCase):
def setUp(self):
prepare_zpools()
self.longMessage=True
self.longMessage = True
def test_keepsource0target10queuedsend(self):
"""Test if thinner doesnt destroy too much early on if there are no common snapshots YET. Issue #84"""
# old behaviour, without bookmarks
with mocktime("20101111000000"):
self.assertFalse(ZfsAutobackup(
"test test_target1 --no-progress --verbose --keep-source=0 --keep-target=10 --allow-empty --no-send".split(
"test test_target1 --no-progress --verbose --keep-source=0 --keep-target=10 --allow-empty --no-send --no-bookmarks".split(
" ")).run())
with mocktime("20101111000001"):
self.assertFalse(ZfsAutobackup(
"test test_target1 --no-progress --verbose --keep-source=0 --keep-target=10 --allow-empty --no-send".split(
"test test_target1 --no-progress --verbose --keep-source=0 --keep-target=10 --allow-empty --no-send --no-bookmarks".split(
" ")).run())
with mocktime("20101111000002"):
self.assertFalse(ZfsAutobackup(
"test test_target1 --no-progress --verbose --keep-source=0 --keep-target=10 --allow-empty".split(
"test test_target1 --no-progress --verbose --keep-source=0 --keep-target=10 --allow-empty --no-bookmarks".split(
" ")).run())
r = shelltest("zfs list -H -o name -r -t all " + TEST_POOLS)
r = shelltest("zfs list -H -o name -r -t snapshot,filesystem " + TEST_POOLS)
self.assertMultiLineEqual(r, """
test_source1
test_source1/fs1
@@ -57,7 +57,6 @@ test_target1/test_source2/fs2/sub@test-20101111000001
test_target1/test_source2/fs2/sub@test-20101111000002
""")
def test_excludepaths(self):
"""Test issue #103"""
@@ -70,8 +69,7 @@ test_target1/test_source2/fs2/sub@test-20101111000002
"test test_target1/target --no-progress --verbose --allow-empty".split(
" ")).run())
r = shelltest("zfs list -H -o name -r -t all " + TEST_POOLS)
r = shelltest("zfs list -H -o name -r -t snapshot,filesystem " + TEST_POOLS)
self.assertMultiLineEqual(r, """
test_source1
test_source1/fs1
@@ -101,5 +99,3 @@ test_target1/target/test_target1/target_shouldnotbeexcluded@test-20101111000000
test_target1/target_shouldnotbeexcluded
test_target1/target_shouldnotbeexcluded@test-20101111000000
""")
+45 -47
View File
@@ -2,14 +2,16 @@ from basetest import *
from zfs_autobackup.ExecuteNode import ExecuteNode
run_orig=ExecuteNode.run
run_counter=0
run_orig = ExecuteNode.run
run_counter = 0
def run_count(*args, **kwargs):
global run_counter
run_counter=run_counter+1
run_counter = run_counter + 1
return (run_orig(*args, **kwargs))
class TestZfsScaling(unittest2.TestCase):
def setUp(self):
@@ -19,83 +21,79 @@ class TestZfsScaling(unittest2.TestCase):
def test_manysnapshots(self):
"""count the number of commands when there are many snapshots."""
snapshot_count=100
snapshot_count = 100
print("Creating many snapshots...")
s=""
for i in range(1970,1970+snapshot_count):
s=s+"zfs snapshot test_source1/fs1@test-{:04}1111000000;".format(i)
s = ""
for i in range(1970, 1970 + snapshot_count):
s = s + "zfs snapshot test_source1/fs1@test-{:04}1111000000;".format(i)
shelltest(s)
global run_counter
run_counter=0
with patch.object(ExecuteNode,'run', run_count) as p:
run_counter = 0
with patch.object(ExecuteNode, 'run', run_count) as p:
with mocktime("20101112000000"):
self.assertFalse(ZfsAutobackup("test test_target1 --no-progress --verbose --keep-source=10000 --keep-target=10000 --no-holds --allow-empty".split(" ")).run())
self.assertFalse(ZfsAutobackup(
"test test_target1 --no-progress --verbose --keep-source=10000 --keep-target=10000 --no-holds --allow-empty".split(
" ")).run())
#this triggers if you make a change with an impact of more than O(snapshot_count/2)
expected_runs=342
# this triggers if you make a change with an impact of more than O(snapshot_count/2)
expected_runs = 549
print("EXPECTED RUNS: {}".format(expected_runs))
print("ACTUAL RUNS : {}".format(run_counter))
self.assertLess(abs(run_counter-expected_runs), snapshot_count/2)
run_counter=0
with patch.object(ExecuteNode,'run', run_count) as p:
self.assertLess(abs(run_counter - expected_runs), snapshot_count / 2)
run_counter = 0
with patch.object(ExecuteNode, 'run', run_count) as p:
with mocktime("20101112000001"):
self.assertFalse(ZfsAutobackup("test test_target1 --no-progress --verbose --keep-source=10000 --keep-target=10000 --no-holds --allow-empty".split(" ")).run())
self.assertFalse(ZfsAutobackup(
"test test_target1 --no-progress --verbose --keep-source=10000 --keep-target=10000 --no-holds --allow-empty".split(
" ")).run())
#this triggers if you make a change with a performance impact of more than O(snapshot_count/2)
expected_runs=47
# this triggers if you make a change with a performance impact of more than O(snapshot_count/2)
expected_runs = 54
print("EXPECTED RUNS: {}".format(expected_runs))
print("ACTUAL RUNS : {}".format(run_counter))
self.assertLess(abs(run_counter-expected_runs), snapshot_count/2)
self.assertLess(abs(run_counter - expected_runs), snapshot_count / 2)
def test_manydatasets(self):
"""count the number of commands when when there are many datasets"""
dataset_count=100
dataset_count = 100
print("Creating many datasets...")
s=""
for i in range(0,dataset_count):
s=s+"zfs create test_source1/fs1/{};".format(i)
s = ""
for i in range(0, dataset_count):
s = s + "zfs create test_source1/fs1/{};".format(i)
shelltest(s)
global run_counter
#first run
run_counter=0
with patch.object(ExecuteNode,'run', run_count) as p:
# first run
run_counter = 0
with patch.object(ExecuteNode, 'run', run_count) as p:
with mocktime("20101112000000"):
self.assertFalse(ZfsAutobackup("test test_target1 --no-progress --verbose --no-holds --allow-empty".split(" ")).run())
self.assertFalse(ZfsAutobackup(
"test test_target1 --no-progress --verbose --no-holds --allow-empty".split(" ")).run())
#this triggers if you make a change with an impact of more than O(snapshot_count/2)`
expected_runs=842
# this triggers if you make a change with an impact of more than O(snapshot_count/2)`
expected_runs = 949
print("EXPECTED RUNS: {}".format(expected_runs))
print("ACTUAL RUNS: {}".format(run_counter))
self.assertLess(abs(run_counter-expected_runs), dataset_count/2)
#second run, should have higher number of expected_runs
run_counter=0
with patch.object(ExecuteNode,'run', run_count) as p:
self.assertLess(abs(run_counter - expected_runs), dataset_count / 2)
# second run, should have higher number of expected_runs
run_counter = 0
with patch.object(ExecuteNode, 'run', run_count) as p:
with mocktime("20101112000001"):
self.assertFalse(ZfsAutobackup("test test_target1 --no-progress --verbose --no-holds --allow-empty".split(" ")).run())
self.assertFalse(ZfsAutobackup(
"test test_target1 --no-progress --verbose --no-holds --allow-empty".split(" ")).run())
#this triggers if you make a change with a performance impact of more than O(snapshot_count/2)
expected_runs=1047
# this triggers if you make a change with a performance impact of more than O(snapshot_count/2)
expected_runs = 1254
print("EXPECTED RUNS: {}".format(expected_runs))
print("ACTUAL RUNS: {}".format(run_counter))
self.assertLess(abs(run_counter-expected_runs), dataset_count/2)
self.assertLess(abs(run_counter - expected_runs), dataset_count / 2)
+14 -12
View File
@@ -16,7 +16,8 @@ class TestSendRecvPipes(unittest2.TestCase):
with self.subTest("local local pipe"):
with mocktime("20101111000000"):
self.assertFalse(ZfsAutobackup(
["test", "test_target1", "--allow-empty", "--exclude-received", "--no-holds", "--no-progress", "--clear-mountpoint",
["test", "test_target1", "--allow-empty", "--no-holds", "--no-progress",
"--clear-mountpoint",
"--send-pipe=dd bs=1M", "--recv-pipe=dd bs=2M"]).run())
shelltest("zfs destroy -r test_target1/test_source1/fs1/sub")
@@ -24,7 +25,7 @@ class TestSendRecvPipes(unittest2.TestCase):
with self.subTest("remote local pipe"):
with mocktime("20101111000001"):
self.assertFalse(ZfsAutobackup(
["test", "test_target1", "--allow-empty", "--exclude-received", "--no-holds", "--no-progress",
["test", "test_target1", "--allow-empty", "--no-holds", "--no-progress",
"--ssh-source=localhost", "--send-pipe=dd bs=1M", "--recv-pipe=dd bs=2M"]).run())
shelltest("zfs destroy -r test_target1/test_source1/fs1/sub")
@@ -32,7 +33,7 @@ class TestSendRecvPipes(unittest2.TestCase):
with self.subTest("local remote pipe"):
with mocktime("20101111000002"):
self.assertFalse(ZfsAutobackup(
["test", "test_target1", "--allow-empty", "--exclude-received", "--no-holds", "--no-progress",
["test", "test_target1", "--allow-empty", "--no-holds", "--no-progress",
"--ssh-target=localhost", "--send-pipe=dd bs=1M", "--recv-pipe=dd bs=2M"]).run())
shelltest("zfs destroy -r test_target1/test_source1/fs1/sub")
@@ -40,11 +41,11 @@ class TestSendRecvPipes(unittest2.TestCase):
with self.subTest("remote remote pipe"):
with mocktime("20101111000003"):
self.assertFalse(ZfsAutobackup(
["test", "test_target1", "--allow-empty", "--exclude-received", "--no-holds", "--no-progress",
["test", "test_target1", "--allow-empty", "--no-holds", "--no-progress",
"--ssh-source=localhost", "--ssh-target=localhost", "--send-pipe=dd bs=1M",
"--recv-pipe=dd bs=2M"]).run())
r = shelltest("zfs list -H -o name -r -t all test_target1")
r = shelltest("zfs list -H -o name -r -t snapshot,filesystem test_target1")
self.assertMultiLineEqual(r, """
test_target1
test_target1/test_source1
@@ -74,7 +75,7 @@ test_target1/test_source2/fs2/sub@test-20101111000003
with self.subTest("compress " + compress):
with mocktime("20101111000000"):
self.assertFalse(ZfsAutobackup(
["test", "test_target1", "--exclude-received", "--no-holds", "--no-progress", "--verbose",
["test", "test_target1", "--no-holds", "--no-progress", "--verbose",
"--compress=" + compress]).run())
shelltest("zfs destroy -r test_target1/test_source1/fs1/sub")
@@ -85,14 +86,15 @@ test_target1/test_source2/fs2/sub@test-20101111000003
with self.subTest("local local pipe"):
with mocktime("20101111000000"):
self.assertFalse(ZfsAutobackup(
["test", "test_target1", "--allow-empty", "--exclude-received", "--no-holds", "--no-progress", "--clear-mountpoint", "--buffer=1M"]).run())
["test", "test_target1", "--allow-empty", "--no-holds", "--no-progress",
"--clear-mountpoint", "--buffer=1M"]).run())
shelltest("zfs destroy -r test_target1/test_source1/fs1/sub")
with self.subTest("remote local pipe"):
with mocktime("20101111000001"):
self.assertFalse(ZfsAutobackup(
["test", "test_target1", "--allow-empty", "--verbose", "--exclude-received", "--no-holds",
["test", "test_target1", "--allow-empty", "--verbose", "--no-holds",
"--no-progress", "--ssh-source=localhost", "--buffer=1M"]).run())
shelltest("zfs destroy -r test_target1/test_source1/fs1/sub")
@@ -100,7 +102,7 @@ test_target1/test_source2/fs2/sub@test-20101111000003
with self.subTest("local remote pipe"):
with mocktime("20101111000002"):
self.assertFalse(ZfsAutobackup(
["test", "test_target1", "--allow-empty", "--exclude-received", "--no-holds", "--no-progress",
["test", "test_target1", "--allow-empty", "--no-holds", "--no-progress",
"--ssh-target=localhost", "--buffer=1M"]).run())
shelltest("zfs destroy -r test_target1/test_source1/fs1/sub")
@@ -108,10 +110,10 @@ test_target1/test_source2/fs2/sub@test-20101111000003
with self.subTest("remote remote pipe"):
with mocktime("20101111000003"):
self.assertFalse(ZfsAutobackup(
["test", "test_target1", "--allow-empty", "--exclude-received", "--no-holds", "--no-progress",
["test", "test_target1", "--allow-empty", "--no-holds", "--no-progress",
"--ssh-source=localhost", "--ssh-target=localhost", "--buffer=1M"]).run())
r = shelltest("zfs list -H -o name -r -t all test_target1")
r = shelltest("zfs list -H -o name -r -t snapshot,filesystem test_target1")
self.assertMultiLineEqual(r, """
test_target1
test_target1/test_source1
@@ -140,7 +142,7 @@ test_target1/test_source2/fs2/sub@test-20101111000003
start = time.time()
with mocktime("20101111000000"):
self.assertFalse(ZfsAutobackup(
["test", "test_target1", "--exclude-received", "--no-holds", "--no-progress", "--rate=50k"]).run())
["test", "test_target1", "--no-holds", "--no-progress", "--rate=50k"]).run())
# not a great way of verifying but it works.
self.assertGreater(time.time() - start, 5)
+46
View File
@@ -2,6 +2,7 @@ from basetest import *
import pprint
from zfs_autobackup.Thinner import Thinner
from zfs_autobackup.ThinnerRule import ThinnerRule
# randint is different in python 2 vs 3
randint_compat = lambda lo, hi: lo + int(random.random() * (hi + 1 - lo))
@@ -99,6 +100,51 @@ class TestThinner(unittest2.TestCase):
self.assertEqual(result, ok)
def test_always_keep_only(self):
"""Thinner('N') with no rules: keep at most N most-recent objects, regardless of age."""
thinner = Thinner("3")
# fewer than N -> keep them all, remove nothing
few = [Thing(100), Thing(200)]
(keeps, removes) = thinner.thin(few, keep_objects=[], now=1000)
self.assertEqual(keeps, few)
self.assertEqual(removes, [])
# exactly N -> keep them all, remove nothing
exact = [Thing(100), Thing(200), Thing(300)]
(keeps, removes) = thinner.thin(exact, keep_objects=[], now=1000)
self.assertEqual(keeps, exact)
self.assertEqual(removes, [])
# more than N -> keep the last N, drop the rest
many = [Thing(100), Thing(200), Thing(300), Thing(400), Thing(500)]
(keeps, removes) = thinner.thin(many, keep_objects=[], now=1000)
self.assertEqual(keeps, many[-3:])
self.assertEqual(removes, many[:-3])
def test_keep_objects_without_timestamp(self):
"""objects with timestamp=None are always kept (treated as 'not ours')."""
thinner = Thinner("1d1w")
no_ts = Thing(None)
old_ts = Thing(100) # well past the ttl
(keeps, removes) = thinner.thin([no_ts, old_ts], keep_objects=[], now=10 ** 9)
self.assertIn(no_ts, keeps)
self.assertIn(old_ts, removes)
def test_empty_thinner_keeps_everything(self):
"""Thinner('') has no rules and always_keep=0 -> nothing is kept by rules, but timestamp=None objects still pass through."""
thinner = Thinner("")
with_ts = Thing(100)
without_ts = Thing(None)
(keeps, removes) = thinner.thin([with_ts, without_ts], keep_objects=[], now=1000)
self.assertEqual(keeps, [without_ts])
self.assertEqual(removes, [with_ts])
def test_full(self):
ok=['2022-03-09 01:56:23',
'2023-01-03 10:53:16',
+3 -3
View File
@@ -15,10 +15,10 @@ class TestTreeHasher(unittest2.TestCase):
def test_treehasher(self):
shelltest("rm -rf /tmp/treehashertest; mkdir /tmp/treehashertest")
shelltest("cp tests/data/whole /tmp/treehashertest")
shelltest("cp data/whole /tmp/treehashertest")
shelltest("mkdir /tmp/treehashertest/emptydir")
shelltest("mkdir /tmp/treehashertest/dir")
shelltest("cp tests/data/whole_whole2_partial /tmp/treehashertest/dir")
shelltest("cp data/whole_whole2_partial /tmp/treehashertest/dir")
# it should ignore these:
shelltest("ln -s / /tmp/treehashertest/symlink")
@@ -66,7 +66,7 @@ class TestTreeHasher(unittest2.TestCase):
with self.subTest("Test mismatch"):
generator = list(tree_hasher.generate("/tmp/treehashertest"))
shelltest("cp tests/data/whole2 /tmp/treehashertest/whole")
shelltest("cp data/whole2 /tmp/treehashertest/whole")
self.assertEqual(list(tree_hasher.compare("/tmp/treehashertest", generator)),
[('whole',
+9 -9
View File
@@ -69,7 +69,7 @@ class TestZfsVerify(unittest2.TestCase):
self.assertFalse(ZfsAutoverify("test test_target1 --verbose --test".split(" ")).run())
with self.subTest("rsync, remote source and target. (not supported, all 6 fail)"):
self.assertEqual(6, ZfsAutoverify("test test_target1 --ssh-source=localhost --ssh-target=localhost --verbose --exclude-received --fs-compare=rsync".split(" ")).run())
self.assertEqual(6, ZfsAutoverify("test test_target1 --ssh-source=localhost --ssh-target=localhost --verbose --fs-compare=rsync".split(" ")).run())
def runchecked(testname, command):
with self.subTest(testname):
@@ -83,20 +83,20 @@ class TestZfsVerify(unittest2.TestCase):
self.assertRegex(buf.getvalue(), "bad_filesystem: FAILED:")
self.assertRegex(buf.getvalue(), "bad_zvol: FAILED:")
runchecked("rsync, remote source", "test test_target1 --ssh-source=localhost --verbose --exclude-received --fs-compare=rsync")
runchecked("rsync, remote target", "test test_target1 --ssh-target=localhost --verbose --exclude-received --fs-compare=rsync")
runchecked("rsync, local", "test test_target1 --verbose --exclude-received --fs-compare=rsync")
runchecked("rsync, remote source", "test test_target1 --ssh-source=localhost --verbose --fs-compare=rsync")
runchecked("rsync, remote target", "test test_target1 --ssh-target=localhost --verbose --fs-compare=rsync")
runchecked("rsync, local", "test test_target1 --verbose --fs-compare=rsync")
runchecked("tar, remote source and remote target",
"test test_target1 --ssh-source=localhost --ssh-target=localhost --verbose --exclude-received --fs-compare=find")
"test test_target1 --ssh-source=localhost --ssh-target=localhost --verbose --fs-compare=find")
runchecked("tar, remote source",
"test test_target1 --ssh-source=localhost --verbose --exclude-received --fs-compare=find")
"test test_target1 --ssh-source=localhost --verbose --fs-compare=find")
runchecked("tar, remote target",
"test test_target1 --ssh-target=localhost --verbose --exclude-received --fs-compare=find")
runchecked("tar, local", "test test_target1 --verbose --exclude-received --fs-compare=find")
"test test_target1 --ssh-target=localhost --verbose --fs-compare=find")
runchecked("tar, local", "test test_target1 --verbose --fs-compare=find")
with self.subTest("no common snapshot"):
#destroy common snapshot, now 3 should fail
shelltest("zfs destroy test_source1/fs1/ok_zvol@test-20101111000000")
self.assertEqual(3, ZfsAutoverify("test test_target1 --verbose --exclude-received".split(" ")).run())
self.assertEqual(3, ZfsAutoverify("test test_target1 --verbose ".split(" ")).run())
+350 -184
View File
@@ -3,14 +3,15 @@ from zfs_autobackup.CmdPipe import CmdPipe
from basetest import *
import time
from zfs_autobackup.LogConsole import LogConsole
from zfs_autobackup.LogConsole import LogConsole
from zfs_autobackup.ZfsSnapshot import ZfsSnapshot
class TestZfsAutobackup(unittest2.TestCase):
def setUp(self):
prepare_zpools()
self.longMessage=True
self.longMessage = True
def test_invalidpars(self):
@@ -18,7 +19,9 @@ class TestZfsAutobackup(unittest2.TestCase):
with OutputIO() as buf:
with redirect_stdout(buf):
self.assertEqual(ZfsAutobackup("test test_target1 --no-progress --resume --verbose --no-snapshot".split(" ")).run(), 0)
self.assertEqual(
ZfsAutobackup("test test_target1 --no-progress --resume --verbose --no-snapshot".split(" ")).run(),
0)
print(buf.getvalue())
self.assertIn("The --resume", buf.getvalue())
@@ -31,15 +34,14 @@ class TestZfsAutobackup(unittest2.TestCase):
# correct message?
self.assertIn("Please create this dataset", buf.getvalue())
def test_snapshotmode(self):
def test_snapshotmode(self):
"""test snapshot tool mode"""
with mocktime("20101111000000"):
self.assertFalse(ZfsAutobackup("test --no-progress --verbose".split(" ")).run())
r=shelltest("zfs list -H -o name -r -t all "+TEST_POOLS)
self.assertMultiLineEqual(r,"""
r = shelltest("zfs list -H -o name -r -t snapshot,filesystem " + TEST_POOLS)
self.assertMultiLineEqual(r, """
test_source1
test_source1/fs1
test_source1/fs1@test-20101111000000
@@ -54,27 +56,26 @@ test_source2/fs3/sub
test_target1
""")
def test_defaults(self):
self.maxDiff=2000
def test_defaults(self):
self.maxDiff = 2000
with self.subTest("no datasets selected"):
with OutputIO() as buf:
with redirect_stderr(buf):
with mocktime("20101111000000"):
self.assertTrue(ZfsAutobackup("nonexisting test_target1 --verbose --debug --no-progress".split(" ")).run())
self.assertTrue(
ZfsAutobackup("nonexisting test_target1 --verbose --debug --no-progress".split(" ")).run())
print(buf.getvalue())
#correct message?
# correct message?
self.assertIn("No source filesystems selected", buf.getvalue())
with self.subTest("defaults with full verbose and debug"):
with mocktime("20101111000000"):
self.assertFalse(ZfsAutobackup("test test_target1 --verbose --debug --no-progress".split(" ")).run())
r=shelltest("zfs list -H -o name -r -t all "+TEST_POOLS)
self.assertMultiLineEqual(r,"""
r = shelltest("zfs list -H -o name -r -t snapshot,filesystem " + TEST_POOLS)
self.assertMultiLineEqual(r, """
test_source1
test_source1/fs1
test_source1/fs1@test-20101111000000
@@ -100,11 +101,11 @@ test_target1/test_source2/fs2/sub@test-20101111000000
with self.subTest("bare defaults, allow empty"):
with mocktime("20101111000001"):
self.assertFalse(ZfsAutobackup("test test_target1 --allow-empty --no-progress".split(" ")).run())
self.assertFalse(
ZfsAutobackup("test test_target1 --allow-empty --no-progress --no-bookmarks".split(" ")).run())
r=shelltest("zfs list -H -o name -r -t all "+TEST_POOLS)
self.assertMultiLineEqual(r,"""
r = shelltest("zfs list -H -o name -r -t snapshot,filesystem " + TEST_POOLS)
self.assertMultiLineEqual(r, """
test_source1
test_source1/fs1
test_source1/fs1@test-20101111000000
@@ -135,9 +136,8 @@ test_target1/test_source2/fs2/sub@test-20101111000001
""")
with self.subTest("verify holds"):
r=shelltest("zfs get -r userrefs test_source1 test_source2 test_target1")
self.assertMultiLineEqual(r,"""
r = shelltest("zfs get -t filesystem,snapshot -r userrefs test_source1 test_source2 test_target1")
self.assertMultiLineEqual(r, """
NAME PROPERTY VALUE SOURCE
test_source1 userrefs - -
test_source1/fs1 userrefs - -
@@ -168,56 +168,59 @@ test_target1/test_source2/fs2/sub@test-20101111000000 userrefs 0 -
test_target1/test_source2/fs2/sub@test-20101111000001 userrefs 1 -
""")
#make sure time handling is correctly. try to make snapshots a year appart and verify that only snapshots mostly 1y old are kept
#So in this case we only want to see 2 snapshots of 2011, and none of the 2010's anymore.
# make sure time handling is correctly. try to make snapshots a year appart and verify that only snapshots mostly 1y old are kept
# So in this case we only want to see 2 snapshots of 2011, and none of the 2010's anymore.
with self.subTest("test time checking"):
with mocktime("20111211000000"):
self.assertFalse(ZfsAutobackup("test test_target1 --allow-empty --verbose --no-progress".split(" ")).run())
self.assertFalse(
ZfsAutobackup("test test_target1 --allow-empty --verbose --no-progress".split(" ")).run())
with mocktime("20111211000001"):
self.assertFalse(ZfsAutobackup("test test_target1 --allow-empty --verbose --keep-source 1y1y --keep-target 1d1y --no-progress".split(" ")).run())
with mocktime("20111212000000"):
self.assertFalse(ZfsAutobackup(
"test test_target1 --allow-empty --verbose --keep-source 1d1y --keep-target 1d1y --no-progress".split(
" ")).run())
r=shelltest("zfs list -H -o name -r -t all "+TEST_POOLS)
self.assertMultiLineEqual(r,"""
r = shelltest("zfs list -H -o name -r -t snapshot,filesystem " + TEST_POOLS)
self.assertMultiLineEqual("""
test_source1
test_source1/fs1
test_source1/fs1@test-20111211000000
test_source1/fs1@test-20111211000001
test_source1/fs1@test-20111212000000
test_source1/fs1/sub
test_source1/fs1/sub@test-20111211000000
test_source1/fs1/sub@test-20111211000001
test_source1/fs1/sub@test-20111212000000
test_source2
test_source2/fs2
test_source2/fs2/sub
test_source2/fs2/sub@test-20111211000000
test_source2/fs2/sub@test-20111211000001
test_source2/fs2/sub@test-20111212000000
test_source2/fs3
test_source2/fs3/sub
test_target1
test_target1/test_source1
test_target1/test_source1/fs1
test_target1/test_source1/fs1@test-20111211000000
test_target1/test_source1/fs1@test-20111211000001
test_target1/test_source1/fs1@test-20111212000000
test_target1/test_source1/fs1/sub
test_target1/test_source1/fs1/sub@test-20111211000000
test_target1/test_source1/fs1/sub@test-20111211000001
test_target1/test_source1/fs1/sub@test-20111212000000
test_target1/test_source2
test_target1/test_source2/fs2
test_target1/test_source2/fs2/sub
test_target1/test_source2/fs2/sub@test-20111211000000
test_target1/test_source2/fs2/sub@test-20111211000001
""")
test_target1/test_source2/fs2/sub@test-20111212000000
""", r)
def test_ignore_othersnaphots(self):
def test_ignore_othersnaphots(self):
r=shelltest("zfs snapshot test_source1/fs1@othersimple")
r=shelltest("zfs snapshot test_source1/fs1@otherdate-20001111000000")
r = shelltest("zfs snapshot test_source1/fs1@othersimple")
r = shelltest("zfs snapshot test_source1/fs1@otherdate-20001111000000")
with mocktime("20101111000000"):
self.assertFalse(ZfsAutobackup("test test_target1 --no-progress --verbose".split(" ")).run())
r=shelltest("zfs list -H -o name -r -t all "+TEST_POOLS)
self.assertMultiLineEqual(r,"""
r = shelltest("zfs list -H -o name -r -t snapshot,filesystem " + TEST_POOLS)
self.assertMultiLineEqual(r, """
test_source1
test_source1/fs1
test_source1/fs1@othersimple
@@ -243,16 +246,17 @@ test_target1/test_source2/fs2/sub
test_target1/test_source2/fs2/sub@test-20101111000000
""")
def test_othersnaphots(self):
def test_othersnaphots(self):
r=shelltest("zfs snapshot test_source1/fs1@othersimple")
r=shelltest("zfs snapshot test_source1/fs1@otherdate-20001111000000")
r = shelltest("zfs snapshot test_source1/fs1@othersimple")
r = shelltest("zfs snapshot test_source1/fs1@otherdate-20001111000000")
with mocktime("20101111000000"):
self.assertFalse(ZfsAutobackup("test test_target1 --no-progress --verbose --other-snapshots".split(" ")).run())
self.assertFalse(
ZfsAutobackup("test test_target1 --no-progress --verbose --other-snapshots".split(" ")).run())
r=shelltest("zfs list -H -o name -r -t all "+TEST_POOLS)
self.assertMultiLineEqual(r,"""
r = shelltest("zfs list -H -o name -r -t snapshot,filesystem " + TEST_POOLS)
self.assertMultiLineEqual(r, """
test_source1
test_source1/fs1
test_source1/fs1@othersimple
@@ -280,16 +284,15 @@ test_target1/test_source2/fs2/sub
test_target1/test_source2/fs2/sub@test-20101111000000
""")
def test_nosnapshot(self):
def test_nosnapshot(self):
with mocktime("20101111000000"):
self.assertFalse(ZfsAutobackup("test test_target1 --verbose --no-snapshot --no-progress".split(" ")).run())
r=shelltest("zfs list -H -o name -r -t all "+TEST_POOLS)
#(only parents are created )
#TODO: it probably shouldn't create these
self.assertMultiLineEqual(r,"""
r = shelltest("zfs list -H -o name -r -t snapshot,filesystem " + TEST_POOLS)
# (only parents are created )
# TODO: it probably shouldn't create these
self.assertMultiLineEqual(r, """
test_source1
test_source1/fs1
test_source1/fs1/sub
@@ -304,14 +307,13 @@ test_target1/test_source2
test_target1/test_source2/fs2
""")
def test_nosend(self):
def test_nosend(self):
with mocktime("20101111000000"):
self.assertFalse(ZfsAutobackup("test test_target1 --verbose --no-send --no-progress".split(" ")).run())
r=shelltest("zfs list -H -o name -r -t all "+TEST_POOLS)
self.assertMultiLineEqual(r,"""
r = shelltest("zfs list -H -o name -r -t snapshot,filesystem " + TEST_POOLS)
self.assertMultiLineEqual(r, """
test_source1
test_source1/fs1
test_source1/fs1@test-20101111000000
@@ -326,15 +328,15 @@ test_source2/fs3/sub
test_target1
""")
def test_ignorereplicated(self):
r=shelltest("zfs snapshot test_source1/fs1@otherreplication")
def test_ignorereplicated(self):
r = shelltest("zfs snapshot test_source1/fs1@otherreplication")
with mocktime("20101111000000"):
self.assertFalse(ZfsAutobackup("test test_target1 --no-progress --verbose --ignore-replicated".split(" ")).run())
self.assertFalse(
ZfsAutobackup("test test_target1 --no-progress --verbose --ignore-replicated".split(" ")).run())
r=shelltest("zfs list -H -o name -r -t all "+TEST_POOLS)
self.assertMultiLineEqual(r,"""
r = shelltest("zfs list -H -o name -r -t snapshot,filesystem " + TEST_POOLS)
self.assertMultiLineEqual(r, """
test_source1
test_source1/fs1
test_source1/fs1@otherreplication
@@ -357,13 +359,13 @@ test_target1/test_source2/fs2/sub
test_target1/test_source2/fs2/sub@test-20101111000000
""")
def test_noholds(self):
def test_noholds(self):
with mocktime("20101111000000"):
self.assertFalse(ZfsAutobackup("test test_target1 --verbose --no-holds --no-progress".split(" ")).run())
r=shelltest("zfs get -r userrefs test_source1 test_source2 test_target1")
self.assertMultiLineEqual(r,"""
r = shelltest("zfs get -t filesystem,snapshot -r userrefs test_source1 test_source2 test_target1")
self.assertMultiLineEqual(r, """
NAME PROPERTY VALUE SOURCE
test_source1 userrefs - -
test_source1/fs1 userrefs - -
@@ -388,14 +390,13 @@ test_target1/test_source2/fs2/sub userrefs - -
test_target1/test_source2/fs2/sub@test-20101111000000 userrefs 0 -
""")
def test_strippath(self):
def test_strippath(self):
with mocktime("20101111000000"):
self.assertFalse(ZfsAutobackup("test test_target1 --verbose --strip-path=1 --no-progress".split(" ")).run())
r=shelltest("zfs list -H -o name -r -t all "+TEST_POOLS)
self.assertMultiLineEqual(r,"""
r = shelltest("zfs list -H -o name -r -t snapshot,filesystem " + TEST_POOLS)
self.assertMultiLineEqual(r, """
test_source1
test_source1/fs1
test_source1/fs1@test-20101111000000
@@ -418,27 +419,185 @@ test_target1/fs2/sub@test-20101111000000
""")
def test_strippath_collision(self):
with self.assertRaisesRegexp(Exception,"collision"):
with self.assertRaisesRegexp(Exception, "collision"):
ZfsAutobackup("test test_target1 --verbose --strip-path=2 --no-progress --debug".split(" ")).run()
def test_strippath_toomuch(self):
with self.assertRaisesRegexp(Exception,"too much"):
with self.assertRaisesRegexp(Exception, "too much"):
ZfsAutobackup("test test_target1 --verbose --strip-path=3 --no-progress --debug".split(" ")).run()
def test_clearrefres(self):
def test_property_format(self):
"""--property-format selects datasets via a custom ZFS user-property name.
#on zfs utils 0.6.x -x isnt supported
r=shelltest("zfs recv -x bla test >/dev/null </dev/zero; echo $?")
if r=="\n2\n":
self.skipTest("This zfs-userspace version doesnt support -x")
The literal "{}" in the format is substituted with BACKUP-NAME, producing
the actual property name that zfs-autobackup looks for on each dataset.
"""
r=shelltest("zfs set refreservation=1M test_source1/fs1")
# the default "autobackup:test" property on test_source1/fs1 and test_source2/fs2
# should be ignored once we change --property-format. Set a property using the
# custom format on test_source2/fs3 instead, which is normally not selected.
shelltest("zfs set mybackup:test=true test_source2/fs3")
with mocktime("20101111000000"):
self.assertFalse(ZfsAutobackup("test test_target1 --no-progress --verbose --clear-refreservation".split(" ")).run())
self.assertFalse(ZfsAutobackup(
"test test_target1 --no-progress --verbose --property-format mybackup:{}".split(" ")).run())
r=shelltest("zfs get -r refreservation test_source1 test_source2 test_target1")
self.assertMultiLineEqual(r,"""
r = shelltest("zfs list -H -o name -r -t snapshot,filesystem " + TEST_POOLS)
self.assertMultiLineEqual(r, """
test_source1
test_source1/fs1
test_source1/fs1/sub
test_source2
test_source2/fs2
test_source2/fs2/sub
test_source2/fs3
test_source2/fs3@test-20101111000000
test_source2/fs3/sub
test_source2/fs3/sub@test-20101111000000
test_target1
test_target1/test_source2
test_target1/test_source2/fs3
test_target1/test_source2/fs3@test-20101111000000
test_target1/test_source2/fs3/sub
test_target1/test_source2/fs3/sub@test-20101111000000
""")
def test_snapshot_format(self):
"""--snapshot-format controls the snapshot name after the "@".
"{}" is substituted with BACKUP-NAME, then strftime codes are expanded
against the current time. The result must not contain the tag-seperator.
"""
with mocktime("20101111000000"):
self.assertFalse(ZfsAutobackup(
"test test_target1 --no-progress --verbose --snapshot-format snap-{}-%Y-%m-%d".split(" ")).run())
r = shelltest("zfs list -H -o name -r -t snapshot,filesystem " + TEST_POOLS)
self.assertMultiLineEqual(r, """
test_source1
test_source1/fs1
test_source1/fs1@snap-test-2010-11-11
test_source1/fs1/sub
test_source1/fs1/sub@snap-test-2010-11-11
test_source2
test_source2/fs2
test_source2/fs2/sub
test_source2/fs2/sub@snap-test-2010-11-11
test_source2/fs3
test_source2/fs3/sub
test_target1
test_target1/test_source1
test_target1/test_source1/fs1
test_target1/test_source1/fs1@snap-test-2010-11-11
test_target1/test_source1/fs1/sub
test_target1/test_source1/fs1/sub@snap-test-2010-11-11
test_target1/test_source2
test_target1/test_source2/fs2
test_target1/test_source2/fs2/sub
test_target1/test_source2/fs2/sub@snap-test-2010-11-11
""")
def test_snapshot_format_tagseperator_collision(self):
"""If --snapshot-format produces a name that contains the tag-seperator, refuse to run."""
with OutputIO() as buf:
with redirect_stderr(buf):
with mocktime("20101111000000"):
# rendered snapshot-format "test__2010_11_11" contains default tag-seperator "__" -> sys.exit(255)
with self.assertRaises(SystemExit) as cm:
ZfsAutobackup("test test_target1 --no-progress --snapshot-format {}__%Y_%m_%d".split(" "))
self.assertEqual(cm.exception.code, 255)
self.assertIn("Tag seperator", buf.getvalue())
def test_tagseperator_invalid(self):
"""--tag-seperator must be non-empty and contain only characters valid in ZFS snapshot names ('_.: -')."""
# empty string
with OutputIO() as buf:
with redirect_stderr(buf):
with self.assertRaises(SystemExit) as cm:
ZfsAutobackup("test test_target1 --no-progress --tag-seperator=".split(" "))
self.assertEqual(cm.exception.code, 255)
self.assertIn("Invalid tag seperator", buf.getvalue())
# single invalid character
with OutputIO() as buf:
with redirect_stderr(buf):
with self.assertRaises(SystemExit) as cm:
ZfsAutobackup("test test_target1 --no-progress --tag-seperator=!".split(" "))
self.assertEqual(cm.exception.code, 255)
self.assertIn("Invalid tag seperator", buf.getvalue())
# multi-char string where one character is invalid
with OutputIO() as buf:
with redirect_stderr(buf):
with self.assertRaises(SystemExit) as cm:
ZfsAutobackup("test test_target1 --no-progress --tag-seperator=_!".split(" "))
self.assertEqual(cm.exception.code, 255)
self.assertIn("Invalid tag seperator", buf.getvalue())
def test_target_path_absolute_rejected(self):
"""TARGET-PATH must be a ZFS path; a leading '/' is rejected (it's not a filesystem mountpoint)."""
with OutputIO() as buf:
with redirect_stderr(buf):
with self.assertRaises(SystemExit) as cm:
ZfsAutobackup("test /test_target1 --no-progress".split(" "))
self.assertEqual(cm.exception.code, 255)
self.assertIn("Target should not start with a /", buf.getvalue())
def test_tag_contains_seperator_rejected(self):
"""--tag must not contain the tag-seperator (otherwise tagless_suffix parsing breaks)."""
with OutputIO() as buf:
with redirect_stderr(buf):
with self.assertRaises(SystemExit) as cm:
# default tag-seperator is "__"; a tag containing "__" should be rejected.
ZfsAutobackup("test test_target1 --no-progress --tag=foo__bar".split(" "))
self.assertEqual(cm.exception.code, 255)
self.assertIn("may not contain tag seperator", buf.getvalue())
def test_ignore_replicated_renamed_warning(self):
"""--ignore-replicated is the old name; it should warn and be mapped to --exclude-unchanged."""
with mocktime("20101111000000"):
with OutputIO() as buf:
with redirect_stdout(buf):
# --ignore-replicated is hidden in --help but still accepted; run a no-op
# invocation so we can capture the rename warning without doing real work.
self.assertFalse(ZfsAutobackup(
"test --no-progress --verbose --no-snapshot --ignore-replicated".split(" ")).run())
self.assertIn("--ignore-replicated has been renamed", buf.getvalue())
def test_hold_format(self):
"""--hold-format controls the zfs-hold name placed on snapshots."""
with mocktime("20101111000000"):
self.assertFalse(ZfsAutobackup(
"test test_target1 --no-progress --verbose --no-bookmark --hold-format myhold_{}".split(" ")).run())
r = shelltest("zfs holds -H test_source1/fs1@test-20101111000000")
self.assertIn("myhold_test", r)
self.assertNotIn("zfs_autobackup:test", r)
def test_clearrefres(self):
# on zfs utils 0.6.x -x isnt supported
r = shelltest("zfs recv -x bla test >/dev/null </dev/zero; echo $?")
if r == "\n2\n":
self.skipTest("This zfs-userspace version doesnt support -x")
r = shelltest("zfs set refreservation=1M test_source1/fs1")
with mocktime("20101111000000"):
self.assertFalse(
ZfsAutobackup("test test_target1 --no-progress --verbose --clear-refreservation".split(" ")).run())
r = shelltest("zfs get -t filesystem,snapshot -r refreservation test_source1 test_source2 test_target1")
self.assertMultiLineEqual(r, """
NAME PROPERTY VALUE SOURCE
test_source1 refreservation none default
test_source1/fs1 refreservation 1M local
@@ -463,20 +622,19 @@ test_target1/test_source2/fs2/sub refreservation none
test_target1/test_source2/fs2/sub@test-20101111000000 refreservation - -
""")
def test_clearmount(self):
def test_clearmount(self):
#on zfs utils 0.6.x -o isnt supported
r=shelltest("zfs recv -o bla=1 test >/dev/null </dev/zero; echo $?")
if r=="\n2\n":
# on zfs utils 0.6.x -o isnt supported
r = shelltest("zfs recv -o bla=1 test >/dev/null </dev/zero; echo $?")
if r == "\n2\n":
self.skipTest("This zfs-userspace version doesnt support -o")
with mocktime("20101111000000"):
self.assertFalse(ZfsAutobackup("test test_target1 --no-progress --verbose --clear-mountpoint --debug".split(" ")).run())
self.assertFalse(
ZfsAutobackup("test test_target1 --no-progress --verbose --clear-mountpoint --debug".split(" ")).run())
r=shelltest("zfs get -r canmount test_source1 test_source2 test_target1")
self.assertMultiLineEqual(r,"""
r = shelltest("zfs get -r canmount -t filesystem,snapshot test_source1 test_source2 test_target1")
self.assertMultiLineEqual(r, """
NAME PROPERTY VALUE SOURCE
test_source1 canmount on default
test_source1/fs1 canmount on default
@@ -501,61 +659,63 @@ test_target1/test_source2/fs2/sub canmount noauto local
test_target1/test_source2/fs2/sub@test-20101111000000 canmount - -
""")
def test_rollback(self):
def test_rollback(self):
#initial backup
# initial backup
with mocktime("20101111000000"):
self.assertFalse(ZfsAutobackup("test test_target1 --no-progress --verbose".split(" ")).run())
#make change
r=shelltest("touch /test_target1/test_source1/fs1/change.txt")
# make change
r = shelltest("touch /test_target1/test_source1/fs1/change.txt")
with mocktime("20101111000001"):
#should fail (busy)
# should fail (busy)
self.assertTrue(ZfsAutobackup("test test_target1 --no-progress --verbose --allow-empty".split(" ")).run())
with mocktime("20101111000002"):
#rollback, should succeed
self.assertFalse(ZfsAutobackup("test test_target1 --no-progress --verbose --allow-empty --rollback".split(" ")).run())
# rollback, should succeed
self.assertFalse(
ZfsAutobackup("test test_target1 --no-progress --verbose --allow-empty --rollback".split(" ")).run())
def test_destroyincompat(self):
def test_destroyincompat(self):
#initial backup
# initial backup
with mocktime("20101111000000"):
self.assertFalse(ZfsAutobackup("test test_target1 --no-progress --verbose".split(" ")).run())
#add multiple compatible snapshot (written is still 0)
r=shelltest("zfs snapshot test_target1/test_source1/fs1@compatible1")
r=shelltest("zfs snapshot test_target1/test_source1/fs1@compatible2")
# add multiple compatible snapshot (written is still 0)
r = shelltest("zfs snapshot test_target1/test_source1/fs1@compatible1")
r = shelltest("zfs snapshot test_target1/test_source1/fs1@compatible2")
with mocktime("20101111000001"):
#should be ok, is compatible
# should be ok, is compatible
self.assertFalse(ZfsAutobackup("test test_target1 --no-progress --verbose --allow-empty".split(" ")).run())
#add incompatible snapshot by changing and snapshotting
r=shelltest("touch /test_target1/test_source1/fs1/change.txt")
r=shelltest("zfs snapshot test_target1/test_source1/fs1@incompatible1")
# add incompatible snapshot by changing and snapshotting
r = shelltest("touch /test_target1/test_source1/fs1/change.txt")
r = shelltest("zfs snapshot test_target1/test_source1/fs1@incompatible1")
with mocktime("20101111000002"):
#--test should fail, now incompatible
self.assertTrue(ZfsAutobackup("test test_target1 --no-progress --verbose --allow-empty --test".split(" ")).run())
# --test should fail, now incompatible
self.assertTrue(
ZfsAutobackup("test test_target1 --no-progress --verbose --allow-empty --test".split(" ")).run())
with mocktime("20101111000002"):
#should fail, now incompatible
# should fail, now incompatible
self.assertTrue(ZfsAutobackup("test test_target1 --no-progress --verbose --allow-empty".split(" ")).run())
with mocktime("20101111000003"):
#--test should succeed by destroying incompatibles
self.assertFalse(ZfsAutobackup("test test_target1 --no-progress --verbose --allow-empty --destroy-incompatible --test".split(" ")).run())
# --test should succeed by destroying incompatibles
self.assertFalse(ZfsAutobackup(
"test test_target1 --no-progress --verbose --allow-empty --destroy-incompatible --test".split(
" ")).run())
with mocktime("20101111000003"):
#should succeed by destroying incompatibles
self.assertFalse(ZfsAutobackup("test test_target1 --no-progress --verbose --allow-empty --destroy-incompatible".split(" ")).run())
# should succeed by destroying incompatibles
self.assertFalse(ZfsAutobackup(
"test test_target1 --no-progress --verbose --allow-empty --destroy-incompatible".split(" ")).run())
r = shelltest("zfs list -H -o name -r -t all test_target1")
r = shelltest("zfs list -H -o name -r -t snapshot,filesystem test_target1")
self.assertMultiLineEqual(r, """
test_target1
test_target1/test_source1
@@ -580,27 +740,27 @@ test_target1/test_source2/fs2/sub@test-20101111000002
test_target1/test_source2/fs2/sub@test-20101111000003
""")
def test_ssh(self):
#test all ssh directions
# test all ssh directions
with mocktime("20101111000000"):
self.assertFalse(ZfsAutobackup("test test_target1 --no-progress --verbose --allow-empty --ssh-source localhost --exclude-received".split(" ")).run())
self.assertFalse(ZfsAutobackup(
"test test_target1 --no-progress --verbose --allow-empty --ssh-source localhost".split(
" ")).run())
with mocktime("20101111000001"):
self.assertFalse(ZfsAutobackup("test test_target1 --no-progress --verbose --allow-empty --ssh-target localhost --exclude-received".split(" ")).run())
self.assertFalse(ZfsAutobackup(
"test test_target1 --no-progress --verbose --allow-empty --ssh-target localhost".split(
" ")).run())
with mocktime("20101111000002"):
self.assertFalse(ZfsAutobackup("test test_target1 --no-progress --verbose --allow-empty --ssh-source localhost --ssh-target localhost".split(" ")).run())
self.assertFalse(ZfsAutobackup(
"test test_target1 --no-progress --verbose --allow-empty --ssh-source localhost --ssh-target localhost".split(
" ")).run())
r=shelltest("zfs list -H -o name -r -t all "+TEST_POOLS)
self.assertMultiLineEqual(r,"""
r = shelltest("zfs list -H -o name -r -t snapshot,filesystem " + TEST_POOLS)
self.assertMultiLineEqual(r, """
test_source1
test_source1/fs1
test_source1/fs1@test-20101111000000
@@ -636,33 +796,34 @@ test_target1/test_source2/fs2/sub@test-20101111000001
test_target1/test_source2/fs2/sub@test-20101111000002
""")
def test_minchange(self):
def test_minchange(self):
#initial
# initial
with mocktime("20101111000000"):
self.assertFalse(ZfsAutobackup("test test_target1 --no-progress --verbose --min-change 100000".split(" ")).run())
self.assertFalse(
ZfsAutobackup("test test_target1 --no-progress --verbose --min-change 100000".split(" ")).run())
#make small change, use umount to reflect the changes immediately
r=shelltest("zfs set compress=off test_source1")
r=shelltest("touch /test_source1/fs1/change.txt")
r=shelltest("zfs umount test_source1/fs1; zfs mount test_source1/fs1")
# make small change, use umount to reflect the changes immediately
r = shelltest("zfs set compress=off test_source1")
r = shelltest("touch /test_source1/fs1/change.txt")
r = shelltest("zfs umount test_source1/fs1; zfs mount test_source1/fs1")
#too small change, takes no snapshots
# too small change, takes no snapshots
with mocktime("20101111000001"):
self.assertFalse(ZfsAutobackup("test test_target1 --no-progress --verbose --min-change 100000".split(" ")).run())
self.assertFalse(
ZfsAutobackup("test test_target1 --no-progress --verbose --min-change 100000".split(" ")).run())
#make big change
r=shelltest("dd if=/dev/zero of=/test_source1/fs1/change.txt bs=200000 count=1")
r=shelltest("zfs umount test_source1/fs1; zfs mount test_source1/fs1")
# make big change
r = shelltest("dd if=/dev/zero of=/test_source1/fs1/change.txt bs=200000 count=1")
r = shelltest("zfs umount test_source1/fs1; zfs mount test_source1/fs1")
#bigger change, should take snapshot
# bigger change, should take snapshot
with mocktime("20101111000002"):
self.assertFalse(ZfsAutobackup("test test_target1 --no-progress --verbose --min-change 100000".split(" ")).run())
self.assertFalse(
ZfsAutobackup("test test_target1 --no-progress --verbose --min-change 100000".split(" ")).run())
r=shelltest("zfs list -H -o name -r -t all "+TEST_POOLS)
self.assertMultiLineEqual(r,"""
r = shelltest("zfs list -H -o name -r -t snapshot,filesystem " + TEST_POOLS)
self.assertMultiLineEqual(r, """
test_source1
test_source1/fs1
test_source1/fs1@test-20101111000000
@@ -688,14 +849,14 @@ test_target1/test_source2/fs2/sub
test_target1/test_source2/fs2/sub@test-20101111000000
""")
def test_test(self):
def test_test(self):
#initial
# initial
with mocktime("20101111000000"):
self.assertFalse(ZfsAutobackup("test test_target1 --no-progress --verbose --test".split(" ")).run())
r=shelltest("zfs list -H -o name -r -t all "+TEST_POOLS)
self.assertMultiLineEqual(r,"""
r = shelltest("zfs list -H -o name -r -t snapshot,filesystem " + TEST_POOLS)
self.assertMultiLineEqual(r, """
test_source1
test_source1/fs1
test_source1/fs1/sub
@@ -707,17 +868,17 @@ test_source2/fs3/sub
test_target1
""")
#actual make initial backup
# actual make initial backup
with mocktime("20101111000001"):
self.assertFalse(ZfsAutobackup("test test_target1 --no-progress --verbose".split(" ")).run())
#test incremental
# test incremental
with mocktime("20101111000002"):
self.assertFalse(ZfsAutobackup("test test_target1 --no-progress --allow-empty --verbose --test".split(" ")).run())
self.assertFalse(
ZfsAutobackup("test test_target1 --no-progress --allow-empty --verbose --test".split(" ")).run())
r=shelltest("zfs list -H -o name -r -t all "+TEST_POOLS)
self.assertMultiLineEqual(r,"""
r = shelltest("zfs list -H -o name -r -t snapshot,filesystem " + TEST_POOLS)
self.assertMultiLineEqual(r, """
test_source1
test_source1/fs1
test_source1/fs1@test-20101111000001
@@ -741,7 +902,6 @@ test_target1/test_source2/fs2/sub
test_target1/test_source2/fs2/sub@test-20101111000001
""")
def test_migrate(self):
"""test migration from other snapshotting systems. zfs-autobackup should be able to continue from any common snapshot, not just its own."""
@@ -752,8 +912,8 @@ test_target1/test_source2/fs2/sub@test-20101111000001
with mocktime("20101111000000"):
self.assertFalse(ZfsAutobackup("test test_target1 --no-progress --verbose".split(" ")).run())
r=shelltest("zfs list -H -o name -r -t all "+TEST_POOLS)
self.assertMultiLineEqual(r,"""
r = shelltest("zfs list -H -o name -r -t snapshot,filesystem " + TEST_POOLS)
self.assertMultiLineEqual(r, """
test_source1
test_source1/fs1
test_source1/fs1@migrate1
@@ -780,20 +940,25 @@ test_target1/test_source2/fs2/sub@test-20101111000000
""")
def test_keep0(self):
"""test if keep-source=0 and keep-target=0 dont delete common snapshot and break backup"""
"""test if keep-source=0 and keep-target=0 dont delete common snapshot and break backup, LEGACY NO BOOKMARK MODE"""
# backup mode
with mocktime("20101111000000"):
self.assertFalse(ZfsAutobackup("test test_target1 --no-progress --verbose --keep-source=0 --keep-target=0".split(" ")).run())
self.assertFalse(ZfsAutobackup(
"test test_target1 --no-progress --verbose --keep-source=0 --keep-target=0 --no-bookmark".split(
" ")).run())
#make snapshot, shouldnt delete 0
# snapshot mode, shouldnt delete 0
with mocktime("20101111000001"):
self.assertFalse(ZfsAutobackup("test --no-progress --verbose --keep-source=0 --keep-target=0 --allow-empty".split(" ")).run())
self.assertFalse(ZfsAutobackup(
"test --no-progress --verbose --keep-source=0 --keep-target=0 --allow-empty".split(" ")).run())
#make snapshot 2, shouldnt delete 0 since it has holds, but will delete 1 since it has no holds
# snapshot mode, shouldnt delete 0 since it has holds, but will delete 1 since it has no holds
with mocktime("20101111000002"):
self.assertFalse(ZfsAutobackup("test --no-progress --verbose --keep-source=0 --keep-target=0 --allow-empty".split(" ")).run())
self.assertFalse(ZfsAutobackup(
"test --no-progress --verbose --keep-source=0 --keep-target=0 --allow-empty".split(" ")).run())
r = shelltest("zfs list -H -o name -r -t all " + TEST_POOLS)
r = shelltest("zfs list -H -o name -r -t snapshot,filesystem,bookmark " + TEST_POOLS)
self.assertMultiLineEqual(r, """
test_source1
test_source1/fs1
@@ -821,11 +986,13 @@ test_target1/test_source2/fs2/sub
test_target1/test_source2/fs2/sub@test-20101111000000
""")
#make another backup but with no-holds. we should naturally endup with only number 3
# make another backup but with no-holds. we should naturally endup with only number 3
with mocktime("20101111000003"):
self.assertFalse(ZfsAutobackup("test test_target1 --no-progress --verbose --keep-source=0 --keep-target=0 --no-holds --allow-empty".split(" ")).run())
self.assertFalse(ZfsAutobackup(
"test test_target1 --no-progress --verbose --keep-source=0 --keep-target=0 --no-holds --allow-empty --debug --no-bookmark".split(
" ")).run())
r = shelltest("zfs list -H -o name -r -t all " + TEST_POOLS)
r = shelltest("zfs list -H -o name -r -t snapshot,filesystem,bookmark " + TEST_POOLS)
self.assertMultiLineEqual(r, """
test_source1
test_source1/fs1
@@ -850,12 +1017,12 @@ test_target1/test_source2/fs2/sub
test_target1/test_source2/fs2/sub@test-20101111000003
""")
# run with snapshot-only for 4, since we used no-holds, it will delete 3 on the source, breaking the backup
with mocktime("20101111000004"):
self.assertFalse(ZfsAutobackup("test --no-progress --verbose --keep-source=0 --keep-target=0 --allow-empty".split(" ")).run())
self.assertFalse(ZfsAutobackup(
"test --no-progress --verbose --keep-source=0 --keep-target=0 --allow-empty".split(" ")).run())
r = shelltest("zfs list -H -o name -r -t all " + TEST_POOLS)
r = shelltest("zfs list -H -o name -r -t snapshot,filesystem " + TEST_POOLS)
self.assertMultiLineEqual(r, """
test_source1
test_source1/fs1
@@ -880,26 +1047,25 @@ test_target1/test_source2/fs2/sub
test_target1/test_source2/fs2/sub@test-20101111000003
""")
def test_progress(self):
r=shelltest("dd if=/dev/urandom of=/test_source1/data.txt bs=5M count=1")
r = shelltest("dd if=/dev/urandom of=/test_source1/data.txt bs=5M count=1")
r = shelltest("zfs snapshot test_source1@test")
l=LogConsole(show_verbose=True, show_debug=True, color=False)
n=ZfsNode(utc=False, snapshot_time_format="bla", hold_name="bla", logger=l)
d=ZfsDataset(n,"test_source1@test")
sp=d.send_pipe([], prev_snapshot=None, resume_token=None, show_progress=True, raw=False, send_pipes=[], send_properties=True, write_embedded=True, zfs_compressed=True)
l = LogConsole(show_verbose=True, show_debug=True, color=False)
n = ZfsNode(utc=False, snapshot_time_format="bla", hold_name="bla", logger=l, readonly=False,
debug_output=False, thinner=None, exclude_snapshot_patterns=None, ssh_to=None, ssh_config=None,
description=None, tag_seperator="_")
d = ZfsSnapshot(n, "test_source1@test")
sp = d.send_pipe([], prev_snapshot=None, resume_token=None, show_progress=True, raw=False, send_pipes=[],
send_properties=True, write_embedded=True, zfs_compressed=True)
with OutputIO() as buf:
with redirect_stderr(buf):
try:
p=n.run(["mbuffer", "-R1M", "-m4096", "-o" ,"/dev/null"], inp=sp)
p = n.run(["mbuffer", "-R1M", "-m4096", "-o", "/dev/null"], inp=sp)
# p=n.run(["dd", "of=/dev/null"], inp=sp)
except:
@@ -907,4 +1073,4 @@ test_target1/test_source2/fs2/sub@test-20101111000003
print(list(buf.getvalue()))
# correct message?
self.assertRegex(buf.getvalue(),".*>>> .*minutes left.*")
self.assertRegex(buf.getvalue(), ".*>>> .*minutes left.*")
+41 -20
View File
@@ -1,23 +1,25 @@
from basetest import *
import time
class TestZfsAutobackup31(unittest2.TestCase):
"""various new 3.1 features"""
def setUp(self):
prepare_zpools()
self.longMessage=True
self.longMessage = True
def test_no_thinning(self):
with mocktime("20101111000000"):
self.assertFalse(ZfsAutobackup("test test_target1 --no-progress --verbose --allow-empty".split(" ")).run())
with mocktime("20101111000001"):
self.assertFalse(ZfsAutobackup("test test_target1 --no-progress --verbose --allow-empty --keep-target=0 --keep-source=0 --no-thinning".split(" ")).run())
self.assertFalse(ZfsAutobackup(
"test test_target1 --no-progress --verbose --allow-empty --keep-target=0 --keep-source=0 --no-thinning".split(
" ")).run())
r=shelltest("zfs list -H -o name -r -t all "+TEST_POOLS)
self.assertMultiLineEqual(r,"""
r = shelltest("zfs list -H -o name -r -t snapshot,filesystem " + TEST_POOLS)
self.assertMultiLineEqual(r, """
test_source1
test_source1/fs1
test_source1/fs1@test-20101111000000
@@ -47,7 +49,6 @@ test_target1/test_source2/fs2/sub@test-20101111000000
test_target1/test_source2/fs2/sub@test-20101111000001
""")
def test_re_replication(self):
"""test re-replication of something thats already a backup (new in v3.1-beta5)"""
@@ -55,26 +56,48 @@ test_target1/test_source2/fs2/sub@test-20101111000001
shelltest("zfs create test_target1/b")
with mocktime("20101111000000"):
self.assertFalse(ZfsAutobackup("test test_target1/a --no-progress --verbose --debug".split(" ")).run())
self.assertFalse(
ZfsAutobackup("test test_target1/a --no-progress --verbose --debug --allow-empty".split(" ")).run())
# NOTE: since v3.4 this changed. autobackup: properties are filtered. So its up to the admin to reset this property on the other side:
shelltest("zfs set autobackup:test=true test_target1/a")
with mocktime("20101111000001"):
self.assertFalse(ZfsAutobackup("test test_target1/b --no-progress --verbose".split(" ")).run())
self.assertFalse(
ZfsAutobackup("test test_target1/b --no-progress --verbose --allow-empty".split(" ")).run())
r=shelltest("zfs list -H -o name -r -t snapshot test_target1")
#NOTE: it wont backup test_target1/a/test_source2/fs2/sub to test_target1/b since it doesnt have the zfs_autobackup property anymore.
self.assertMultiLineEqual(r,"""
r = shelltest("zfs list -H -o name -r -t snapshot test_target1")
# NOTE: it wont backup test_target1/a/test_source2/fs2/sub to test_target1/b since it doesnt have the zfs_autobackup property anymore.
self.assertMultiLineEqual("""
test_target1/a@test-20101111000001
test_target1/a/test_source1@test-20101111000001
test_target1/a/test_source1/fs1@test-20101111000000
test_target1/a/test_source1/fs1@test-20101111000001
test_target1/a/test_source1/fs1/sub@test-20101111000000
test_target1/a/test_source1/fs1/sub@test-20101111000001
test_target1/a/test_source2@test-20101111000001
test_target1/a/test_source2/fs2@test-20101111000001
test_target1/a/test_source2/fs2/sub@test-20101111000000
test_target1/a/test_source2/fs2/sub@test-20101111000001
test_target1/b/test_source1/fs1@test-20101111000000
test_target1/b/test_source1/fs1@test-20101111000001
test_target1/b/test_source1/fs1/sub@test-20101111000000
test_target1/b/test_source1/fs1/sub@test-20101111000001
test_target1/b/test_source2/fs2/sub@test-20101111000000
test_target1/b/test_source2/fs2/sub@test-20101111000001
test_target1/b/test_target1/a@test-20101111000001
test_target1/b/test_target1/a/test_source1@test-20101111000001
test_target1/b/test_target1/a/test_source1/fs1@test-20101111000000
test_target1/b/test_target1/a/test_source1/fs1@test-20101111000001
test_target1/b/test_target1/a/test_source1/fs1/sub@test-20101111000000
""")
test_target1/b/test_target1/a/test_source1/fs1/sub@test-20101111000001
test_target1/b/test_target1/a/test_source2@test-20101111000001
test_target1/b/test_target1/a/test_source2/fs2@test-20101111000001
test_target1/b/test_target1/a/test_source2/fs2/sub@test-20101111000000
test_target1/b/test_target1/a/test_source2/fs2/sub@test-20101111000001
""", r)
def test_zfs_compressed(self):
with mocktime("20101111000000"):
self.assertFalse(
ZfsAutobackup("test test_target1 --no-progress --verbose --debug --zfs-compressed".split(" ")).run())
@@ -86,19 +109,18 @@ test_target1/b/test_target1/a/test_source1/fs1/sub@test-20101111000000
with mocktime("20101111000000"):
self.assertFalse(
ZfsAutobackup("test test_target1 --no-progress --verbose --debug --force --strip-path=1".split(" ")).run())
ZfsAutobackup(
"test test_target1 --no-progress --verbose --debug --force --strip-path=1".split(" ")).run())
r=shelltest("zfs list -H -o name -r -t snapshot test_target1")
self.assertMultiLineEqual(r,"""
r = shelltest("zfs list -H -o name -r -t snapshot test_target1")
self.assertMultiLineEqual(r, """
test_target1@test-20101111000000
test_target1/fs1@test-20101111000000
test_target1/fs1/sub@test-20101111000000
test_target1/fs2/sub@test-20101111000000
""")
def test_exclude_unchanged(self):
shelltest("zfs snapshot -r test_source1@somesnapshot")
with mocktime("20101111000000"):
@@ -106,7 +128,7 @@ test_target1/fs2/sub@test-20101111000000
ZfsAutobackup(
"test test_target1 --verbose --allow-empty --exclude-unchanged=1".split(" ")).run())
#everything should be excluded, but should not return an error (see #190)
# everything should be excluded, but should not return an error (see #190)
with mocktime("20101111000001"):
self.assertFalse(
ZfsAutobackup(
@@ -116,4 +138,3 @@ test_target1/fs2/sub@test-20101111000000
self.assertMultiLineEqual(r, """
test_target1/test_source2/fs2/sub@test-20101111000000
""")
+35 -38
View File
@@ -1,29 +1,30 @@
from basetest import *
class TestZfsAutobackup32(unittest2.TestCase):
"""various new 3.2 features"""
def setUp(self):
prepare_zpools()
self.longMessage=True
self.longMessage = True
def test_invalid_common_snapshot(self):
with mocktime("20101111000000"):
self.assertFalse(ZfsAutobackup("test test_target1 --no-progress --verbose --allow-empty".split(" ")).run())
#create 2 snapshots with the same name, which are invalid as common snapshot
# create 2 snapshots with the same name, which are invalid as common snapshot
shelltest("zfs snapshot test_source1/fs1@invalid")
shelltest("zfs snapshot test_target1/test_source1/fs1@invalid")
with mocktime("20101111000001"):
#try the old way (without guid checking), and fail:
self.assertEqual(ZfsAutobackup("test test_target1 --no-progress --verbose --allow-empty --no-guid-check".split(" ")).run(),1)
#new way should be ok:
# try the old way (without guid checking), and fail:
self.assertEqual(ZfsAutobackup(
"test test_target1 --no-progress --verbose --allow-empty --no-guid-check".split(" ")).run(), 1)
# new way should be ok:
self.assertFalse(ZfsAutobackup("test test_target1 --no-progress --verbose --no-snapshot".split(" ")).run())
r=shelltest("zfs list -H -o name -r -t all "+TEST_POOLS)
self.assertMultiLineEqual(r,"""
r = shelltest("zfs list -H -o name -r -t snapshot,filesystem " + TEST_POOLS)
self.assertMultiLineEqual(r, """
test_source1
test_source1/fs1
test_source1/fs1@test-20101111000000
@@ -56,23 +57,25 @@ test_target1/test_source2/fs2/sub@test-20101111000001
""")
def test_invalid_common_snapshot_with_data(self):
with mocktime("20101111000000"):
self.assertFalse(ZfsAutobackup("test test_target1 --no-progress --verbose --allow-empty".split(" ")).run())
#create 2 snapshots with the same name, which are invalid as common snapshot
# create 2 snapshots with the same name, which are invalid as common snapshot
shelltest("zfs snapshot test_source1/fs1@invalid")
shelltest("touch /test_target1/test_source1/fs1/shouldnotbeHere")
shelltest("zfs snapshot test_target1/test_source1/fs1@invalid")
with mocktime("20101111000001"):
#try the old way and fail:
self.assertEqual(ZfsAutobackup("test test_target1 --no-progress --verbose --allow-empty --destroy-incompatible --no-guid-check".split(" ")).run(),1)
#new way should be ok
self.assertFalse(ZfsAutobackup("test test_target1 --no-progress --verbose --no-snapshot --destroy-incompatible".split(" ")).run())
# try the old way and fail:
self.assertEqual(ZfsAutobackup(
"test test_target1 --no-progress --verbose --allow-empty --destroy-incompatible --no-guid-check".split(
" ")).run(), 1)
# new way should be ok
self.assertFalse(ZfsAutobackup(
"test test_target1 --no-progress --verbose --no-snapshot --destroy-incompatible".split(" ")).run())
r=shelltest("zfs list -H -o name -r -t all "+TEST_POOLS)
self.assertMultiLineEqual(r,"""
r = shelltest("zfs list -H -o name -r -t snapshot,filesystem " + TEST_POOLS)
self.assertMultiLineEqual(r, """
test_source1
test_source1/fs1
test_source1/fs1@test-20101111000000
@@ -103,8 +106,7 @@ test_target1/test_source2/fs2/sub@test-20101111000000
test_target1/test_source2/fs2/sub@test-20101111000001
""")
#check consistent mounting behaviour, see issue #112
# check consistent mounting behaviour, see issue #112
def test_mount_consitency_mounted(self):
"""only filesystems that have canmount=on with a mountpoint should be mounted. """
@@ -113,35 +115,32 @@ test_target1/test_source2/fs2/sub@test-20101111000001
with mocktime("20101111000000"):
self.assertFalse(ZfsAutobackup("test test_target1 --no-progress --verbose --allow-empty".split(" ")).run())
r=shelltest("zfs mount |grep -o /test_target1.*")
self.assertMultiLineEqual(r,"""
r = shelltest("zfs mount |grep -o /test_target1.*")
self.assertMultiLineEqual(r, """
/test_target1
/test_target1/test_source1/fs1
/test_target1/test_source1/fs1/sub
/test_target1/test_source2/fs2/sub
""")
def test_mount_consitency_unmounted(self):
"""only test_target1 should be mounted in this test"""
shelltest("zfs create -V 10M test_source1/fs1/subvol")
with mocktime("20101111000000"):
self.assertFalse(ZfsAutobackup("test test_target1 --no-progress --verbose --allow-empty --clear-mountpoint".split(" ")).run())
self.assertFalse(ZfsAutobackup(
"test test_target1 --no-progress --verbose --allow-empty --clear-mountpoint".split(" ")).run())
r=shelltest("zfs mount |grep -o /test_target1.*")
self.assertMultiLineEqual(r,"""
r = shelltest("zfs mount |grep -o /test_target1.*")
self.assertMultiLineEqual(r, """
/test_target1
""")
def test_transfer_thinning(self):
# test pre/post/during transfer thinning and efficient transfer (no transerring of stuff that gets deleted on target)
# test pre/post/during transfer thinning and efficient transfer (no transerring of stuff that gets deleted on target) NO BOOKMARKS LEGACY MODE
#less output
# less output
shelltest("zfs set autobackup:test2=true test_source1/fs1/sub")
# nobody wants this one, will be destroyed before transferring (over a year ago)
@@ -163,16 +162,17 @@ test_target1/test_source2/fs2/sub@test-20101111000001
with mocktime("20010202000000"):
self.assertFalse(ZfsAutobackup("test2 --allow-empty".split(" ")).run())
#will become common snapshot
with OutputIO() as buf:
with redirect_stdout(buf):
# now do thinning and transfer all at once
with mocktime("20010203000000"):
self.assertFalse(ZfsAutobackup("--keep-source=1d10d --keep-target=1m10m --allow-empty --verbose --clear-mountpoint --other-snapshots test2 test_target1".split(" ")).run())
self.assertFalse(ZfsAutobackup(
"--keep-source=1d10d --keep-target=1m10m --allow-empty --verbose --clear-mountpoint --other-snapshots --no-bookmarks test2 test_target1".split(
" ")).run())
print(buf.getvalue())
self.assertIn(
"""
"""
[Source] test_source1/fs1/sub@test2-20000101000000: Destroying
[Source] test_source1/fs1/sub@test2-20010101000000: -> test_target1/test_source1/fs1/sub (new)
[Source] test_source1/fs1/sub@other1: -> test_target1/test_source1/fs1/sub
@@ -182,9 +182,8 @@ test_target1/test_source2/fs2/sub@test-20101111000001
[Source] test_source1/fs1/sub@test2-20010203000000: -> test_target1/test_source1/fs1/sub
""", buf.getvalue())
r=shelltest("zfs list -H -o name -r -t snapshot test_source1 test_target1")
self.assertMultiLineEqual(r,"""
r = shelltest("zfs list -H -o name -r -t snapshot,bookmark test_source1 test_target1")
self.assertMultiLineEqual(r, """
test_source1/fs1/sub@other1
test_source1/fs1/sub@test2-20010201000000
test_source1/fs1/sub@other2
@@ -196,5 +195,3 @@ test_target1/test_source1/fs1/sub@test2-20010201000000
test_target1/test_source1/fs1/sub@other2
test_target1/test_source1/fs1/sub@test2-20010203000000
""")
+718
View File
@@ -0,0 +1,718 @@
from basetest import *
class TestZfsAutobackup40(unittest2.TestCase):
"""various new 4.0 features"""
def setUp(self):
prepare_zpools()
self.longMessage = True
def test_no_bookmark_source_support(self):
"""test if everything is fine when source has no bookmark support (has no features at all even)"""
subprocess.check_call("zpool destroy test_source1", shell=True)
subprocess.check_call("zpool create -d test_source1 /dev/ram0", shell=True)
shelltest("zpool get all test_source1")
subprocess.check_call("zfs create -p test_source1/fs1/sub", shell=True) # recreate with no features at all
subprocess.check_call("zfs set autobackup:test=true test_source1/fs1", shell=True)
with mocktime("20101111000001"):
self.assertFalse(ZfsAutobackup("test test_target1 --no-progress --verbose --allow-empty".split(" ")).run())
# should fallback on holds on the source snapshot
r = shelltest("zfs holds test_source1/fs1@test-20101111000001")
self.assertIn("zfs_autobackup:test", r)
def test_no_bookmark_target_support(self):
"""test if everything is fine when target has no bookmark support (has no features at all even)"""
# NOTE: not sure if its ok if only the source supports bookmarks, so currently zfs-autobackup requires both sides to support bookmarks to enable it.
subprocess.check_call("zpool destroy test_target1", shell=True)
subprocess.check_call("zpool create -d test_target1 /dev/ram2", shell=True)
with mocktime("20101111000001"):
self.assertFalse(ZfsAutobackup("test test_target1 --no-progress --verbose --allow-empty".split(" ")).run())
# should fallback on holds on the source snapshot
r = shelltest("zfs holds test_source1/fs1@test-20101111000001")
self.assertIn("zfs_autobackup:test", r)
def test_select_bookmark_or_snapshot(self):
"""test if zfs autobackup chooses the most recent common matching dataset when there are both bookmarks and snapshots, some with the wrong GUID"""
with mocktime("20101111000001"):
self.assertFalse(ZfsAutobackup("test test_target1 --no-progress --verbose --allow-empty".split(" ")).run())
target_guid = shelltest("zfs get -H -o value guid test_target1").strip()
# destroy stuff and see if it still selects the correct ones
shelltest("zfs destroy test_source2/fs2/sub@test-20101111000001")
shelltest("zfs destroy test_source1/fs1/sub#test-20101111000001__" + target_guid)
# replace bookmark with incorrect GUID, should fallback to snapshot
# NOTE: the incorrect bookmark should be ignored and not destroyed.
shelltest("zfs destroy test_source1/fs1#test-20101111000001__" + target_guid)
shelltest("zfs snapshot test_source1/fs1@wrong")
shelltest("zfs bookmark test_source1/fs1@wrong \#test-20101111000001__" + target_guid)
shelltest("zfs destroy test_source1/fs1@wrong")
with mocktime("20101111000002"):
self.assertFalse(ZfsAutobackup("test test_target1 --no-progress --verbose --allow-empty".split(" ")).run())
r = shelltest("zfs list -H -o name -r -t all " + TEST_POOLS)
self.assertRegexpMatches(r, """
test_source1
test_source1/fs1
test_source1/fs1@test-20101111000001
test_source1/fs1@test-20101111000002
test_source1/fs1#test-20101111000001__[0-9]*
test_source1/fs1#test-20101111000002__[0-9]*
test_source1/fs1/sub
test_source1/fs1/sub@test-20101111000001
test_source1/fs1/sub@test-20101111000002
test_source1/fs1/sub#test-20101111000002__[0-9]*
test_source2
test_source2/fs2
test_source2/fs2/sub
test_source2/fs2/sub@test-20101111000002
test_source2/fs2/sub#test-20101111000002__[0-9]*
test_source2/fs3
test_source2/fs3/sub
test_target1
test_target1/test_source1
test_target1/test_source1/fs1
test_target1/test_source1/fs1@test-20101111000001
test_target1/test_source1/fs1@test-20101111000002
test_target1/test_source1/fs1/sub
test_target1/test_source1/fs1/sub@test-20101111000001
test_target1/test_source1/fs1/sub@test-20101111000002
test_target1/test_source2
test_target1/test_source2/fs2
test_target1/test_source2/fs2/sub
test_target1/test_source2/fs2/sub@test-20101111000001
test_target1/test_source2/fs2/sub@test-20101111000002
""")
# while we're here, check that there are no holds on source common snapshot (since bookmarks replace holds on source side)
r = shelltest("zfs holds test_source2/fs2/sub@test-20101111000002")
self.assertNotIn("zfs_autobackup:test", r)
def test_disable_bookmarks(self):
"""test if we can disable it on an existing backup with bookmarks, with --no-bookmarks and get the old behaviour (holds on source)"""
# first with bookmarks enabled
with mocktime("20101111000001"):
self.assertFalse(ZfsAutobackup("test test_target1 --no-progress --verbose --allow-empty".split(" ")).run())
r = shelltest("zfs list -H -o name -r -t all test_source1")
self.assertRegexpMatches(r, """
test_source1
test_source1/fs1
test_source1/fs1@test-20101111000001
test_source1/fs1#test-20101111000001__[0-9]*
test_source1/fs1/sub
test_source1/fs1/sub@test-20101111000001
test_source1/fs1/sub#test-20101111000001__[0-9]*
""")
# disable it (this should not touch the old bookmark)
with mocktime("20101111000002"):
self.assertFalse(ZfsAutobackup(
"test test_target1 --no-progress --verbose --allow-empty --no-bookmarks".split(" ")).run())
r = shelltest("zfs list -H -o name -r -t all test_source1")
self.assertRegexpMatches(r, """
test_source1
test_source1/fs1
test_source1/fs1@test-20101111000001
test_source1/fs1@test-20101111000002
test_source1/fs1#test-20101111000001__[0-9]*
test_source1/fs1/sub
test_source1/fs1/sub@test-20101111000001
test_source1/fs1/sub@test-20101111000002
test_source1/fs1/sub#test-20101111000001__[0-9]*
""")
# re-enable (now the old bookmark should be still left alone)
with mocktime("20101111000003"):
self.assertFalse(ZfsAutobackup("test test_target1 --no-progress --verbose --allow-empty".split(" ")).run())
r = shelltest("zfs list -H -o name -r -t all test_source1")
self.assertRegexpMatches(r, """
test_source1
test_source1/fs1
test_source1/fs1@test-20101111000001
test_source1/fs1@test-20101111000002
test_source1/fs1@test-20101111000003
test_source1/fs1#test-20101111000001__[0-9]*
test_source1/fs1#test-20101111000003__[0-9]*
test_source1/fs1/sub
test_source1/fs1/sub@test-20101111000001
test_source1/fs1/sub@test-20101111000002
test_source1/fs1/sub@test-20101111000003
test_source1/fs1/sub#test-20101111000001__[0-9]*
test_source1/fs1/sub#test-20101111000003__[0-9]*
""")
def test_tags(self):
with mocktime("20101111000001"):
self.assertFalse(
ZfsAutobackup("test test_target1 --no-progress --verbose --allow-empty --tag test1".split(" ")).run())
with mocktime("20101111000002"):
self.assertFalse(
ZfsAutobackup("test test_target1 --no-progress --verbose --allow-empty --tag test2".split(" ")).run())
with mocktime("20101111000003"):
# make sure the thinner sees and cleans up the old snaphots that have a tag
self.assertFalse(ZfsAutobackup(
"test test_target1 --no-progress --verbose --allow-empty --keep-source=2 --keep-target=2".split(
" ")).run())
r = shelltest("zfs list -H -r -t snapshot -o name " + TEST_POOLS)
self.assertMultiLineEqual(r, """
test_source1/fs1@test-20101111000002__test2
test_source1/fs1@test-20101111000003
test_source1/fs1/sub@test-20101111000002__test2
test_source1/fs1/sub@test-20101111000003
test_source2/fs2/sub@test-20101111000002__test2
test_source2/fs2/sub@test-20101111000003
test_target1/test_source1/fs1@test-20101111000002__test2
test_target1/test_source1/fs1@test-20101111000003
test_target1/test_source1/fs1/sub@test-20101111000002__test2
test_target1/test_source1/fs1/sub@test-20101111000003
test_target1/test_source2/fs2/sub@test-20101111000002__test2
test_target1/test_source2/fs2/sub@test-20101111000003
""")
def test_double_send_bookmark(self):
"""test sending the same snaphots to 2 targets, and check if they each use their own bookmark and delete them correctly."""
shelltest("zfs create test_target1/a")
shelltest("zfs create test_target1/b")
# full
with mocktime("20101111000001"):
self.assertFalse(
ZfsAutobackup(
"test test_target1/a --no-progress --verbose --allow-empty --debug --tag tagA".split(" ")).run())
# increment, should be from bookmark
with mocktime("20101111000002"):
self.assertFalse(
ZfsAutobackup(
"test test_target1/a --no-progress --verbose --allow-empty --tag tagB".split(" ")).run())
# to target b, now each has one full + two incrementals, which should be from their own bookmarks.
with mocktime("20101111000003"):
self.assertFalse(
ZfsAutobackup(
"test test_target1/b --no-progress --verbose --allow-empty".split(
" ")).run())
# result:
# for target a the bookmarks should be at 20101111000002, for target b the bookmarks should be at 20101111000003
r = shelltest("zfs list -H -r -t all -o name " + TEST_POOLS)
self.assertRegexpMatches(r, """
test_source1
test_source1/fs1
test_source1/fs1@test-20101111000001__tagA
test_source1/fs1@test-20101111000002__tagB
test_source1/fs1@test-20101111000003
test_source1/fs1#test-20101111000002__[0-9]*
test_source1/fs1#test-20101111000003__[0-9]*
test_source1/fs1/sub
test_source1/fs1/sub@test-20101111000001__tagA
test_source1/fs1/sub@test-20101111000002__tagB
test_source1/fs1/sub@test-20101111000003
test_source1/fs1/sub#test-20101111000002__[0-9]*
test_source1/fs1/sub#test-20101111000003__[0-9]*
test_source2
test_source2/fs2
test_source2/fs2/sub
test_source2/fs2/sub@test-20101111000001__tagA
test_source2/fs2/sub@test-20101111000002__tagB
test_source2/fs2/sub@test-20101111000003
test_source2/fs2/sub#test-20101111000002__[0-9]*
test_source2/fs2/sub#test-20101111000003__[0-9]*
test_source2/fs3
test_source2/fs3/sub
test_target1
test_target1/a
test_target1/a/test_source1
test_target1/a/test_source1/fs1
test_target1/a/test_source1/fs1@test-20101111000001__tagA
test_target1/a/test_source1/fs1@test-20101111000002__tagB
test_target1/a/test_source1/fs1/sub
test_target1/a/test_source1/fs1/sub@test-20101111000001__tagA
test_target1/a/test_source1/fs1/sub@test-20101111000002__tagB
test_target1/a/test_source2
test_target1/a/test_source2/fs2
test_target1/a/test_source2/fs2/sub
test_target1/a/test_source2/fs2/sub@test-20101111000001__tagA
test_target1/a/test_source2/fs2/sub@test-20101111000002__tagB
test_target1/b
test_target1/b/test_source1
test_target1/b/test_source1/fs1
test_target1/b/test_source1/fs1@test-20101111000001__tagA
test_target1/b/test_source1/fs1@test-20101111000002__tagB
test_target1/b/test_source1/fs1@test-20101111000003
test_target1/b/test_source1/fs1/sub
test_target1/b/test_source1/fs1/sub@test-20101111000001__tagA
test_target1/b/test_source1/fs1/sub@test-20101111000002__tagB
test_target1/b/test_source1/fs1/sub@test-20101111000003
test_target1/b/test_source2
test_target1/b/test_source2/fs2
test_target1/b/test_source2/fs2/sub
test_target1/b/test_source2/fs2/sub@test-20101111000001__tagA
test_target1/b/test_source2/fs2/sub@test-20101111000002__tagB
test_target1/b/test_source2/fs2/sub@test-20101111000003
""")
def test_missing_common_bookmark(self):
with mocktime("20101111000000"):
self.assertFalse(ZfsAutobackup(
"test test_target1 --no-progress --verbose --allow-empty --no-holds".split(" ")).run())
# remove common bookmark
bookmark = shelltest("zfs list -H -o name -t bookmark test_source1/fs1").strip()
shelltest("zfs destroy " + bookmark)
with mocktime("20101111000001"):
self.assertFalse(ZfsAutobackup("test test_target1 --no-progress --verbose --allow-empty".split(" ")).run())
def test_no_common_falls_back_to_incompatible(self):
"""If no common snapshot/bookmark exists with target, target snapshots
must be reported as incompatible (instead of raising), so the user can
recover with --destroy-incompatible."""
# initial backup populates target and creates source bookmarks
with mocktime("20101111000000"):
self.assertFalse(ZfsAutobackup(
"test test_target1 --no-progress --verbose --allow-empty".split(" ")).run())
# wipe every source snapshot and bookmark so nothing matches the target anymore
shelltest("zfs list -H -o name -t snapshot,bookmark -r test_source1 test_source2 "
"| xargs -rn1 zfs destroy")
# without --destroy-incompatible the run must fail: target snapshots are in the way
with mocktime("20101111000001"):
self.assertTrue(ZfsAutobackup(
"test test_target1 --no-progress --verbose --allow-empty".split(" ")).run())
# target snapshots from the initial backup must still be there (nothing destroyed yet)
r = shelltest("zfs list -H -o name -t snapshot test_target1/test_source1/fs1").strip()
self.assertIn("test-20101111000000", r)
# with --destroy-incompatible -F the run succeeds: target is wiped and re-seeded
with mocktime("20101111000002"):
self.assertFalse(ZfsAutobackup(
"test test_target1 --no-progress --verbose --allow-empty --destroy-incompatible -F".split(" ")).run())
r = shelltest("zfs list -H -o name -t snapshot -r test_target1")
self.assertNotIn("test-20101111000000", r)
self.assertIn("test-20101111000002", r)
def test_incompatible_dataset_test_mode(self):
"""In --test mode, when no common snapshot exists, target dataset is
flagged as incompatible. Verify the output strings and that nothing
is actually changed."""
# initial backup
with mocktime("20101111000000"):
self.assertFalse(ZfsAutobackup(
"test test_target1 --no-progress --verbose --allow-empty".split(" ")).run())
# wipe every source snapshot and bookmark so nothing matches target
shelltest("zfs list -H -o name -t snapshot,bookmark -r test_source1 test_source2 "
"| xargs -rn1 zfs destroy")
# snapshot listing BEFORE running --test (to verify nothing changes)
before = shelltest("zfs list -H -o name -r -t snapshot " + TEST_POOLS)
# --test without --destroy-incompatible -F: must fail and report "Incompatible dataset!"
with OutputIO() as buf_out, OutputIO() as buf_err:
with redirect_stdout(buf_out), redirect_stderr(buf_err):
with mocktime("20101111000001"):
self.assertTrue(ZfsAutobackup(
"test test_target1 --no-progress --verbose --allow-empty --test".split(" ")).run())
combined = buf_out.getvalue() + buf_err.getvalue()
print(combined)
self.assertIn("Incompatible dataset", combined)
self.assertNotIn("STDERR", combined)
# --test WITH --destroy-incompatible -F: must succeed and warn "Overwriting incompatible dataset"
with OutputIO() as buf_out, OutputIO() as buf_err:
with redirect_stdout(buf_out), redirect_stderr(buf_err):
with mocktime("20101111000002"):
self.assertFalse(ZfsAutobackup(
"test test_target1 --no-progress --verbose --allow-empty --test --destroy-incompatible -F".split(" ")).run())
combined = buf_out.getvalue() + buf_err.getvalue()
self.assertIn("Overwriting incompatible dataset", combined)
self.assertNotIn("STDERR", combined)
# --test must not have changed anything on disk
after = shelltest("zfs list -H -o name -r -t snapshot " + TEST_POOLS)
self.assertMultiLineEqual(before, after)
def test_incompatible_snapshot_test_mode(self):
"""In --test mode, when a common snapshot exists but extra incompatible
target snapshots are in the way, those must be reported as incompatible.
Verify the output string and that nothing is actually changed."""
# initial backup
with mocktime("20101111000000"):
self.assertFalse(ZfsAutobackup(
"test test_target1 --no-progress --verbose".split(" ")).run())
# add an incompatible target snapshot (data written after common snapshot)
shelltest("touch /test_target1/test_source1/fs1/change.txt")
shelltest("zfs snapshot test_target1/test_source1/fs1@incompatible1")
# snapshot listing BEFORE running --test (to verify nothing changes)
before = shelltest("zfs list -H -o name -r -t snapshot " + TEST_POOLS)
# --test without --destroy-incompatible: must fail and warn about "Incompatible snapshot"
with OutputIO() as buf_out, OutputIO() as buf_err:
with redirect_stdout(buf_out), redirect_stderr(buf_err):
with mocktime("20101111000001"):
self.assertTrue(ZfsAutobackup(
"test test_target1 --no-progress --verbose --allow-empty --test".split(" ")).run())
combined = buf_out.getvalue() + buf_err.getvalue()
self.assertIn("Incompatible snapshot", combined)
self.assertNotIn("STDERR", combined)
# --test WITH --destroy-incompatible: must succeed and still report "Incompatible snapshot"
with OutputIO() as buf_out, OutputIO() as buf_err:
with redirect_stdout(buf_out), redirect_stderr(buf_err):
with mocktime("20101111000002"):
self.assertFalse(ZfsAutobackup(
"test test_target1 --no-progress --verbose --allow-empty --test --destroy-incompatible".split(" ")).run())
combined = buf_out.getvalue() + buf_err.getvalue()
self.assertIn("Incompatible snapshot", combined)
self.assertNotIn("STDERR", combined)
# --test must not have changed anything on disk
after = shelltest("zfs list -H -o name -r -t snapshot " + TEST_POOLS)
self.assertMultiLineEqual(before, after)
def test_keepsource0target10queuedsend_bookmarks(self):
"""Test if thinner doesnt destroy too much early on if there are no common snapshots YET. Issue #84"""
# new behavior, with bookmarks. (will delete common snapshot, since there is a bookmark)
with mocktime("20101111000000"):
self.assertFalse(ZfsAutobackup(
"test test_target1 --no-progress --verbose --keep-source=0 --keep-target=10 --allow-empty --no-send".split(
" ")).run())
with mocktime("20101111000001"):
self.assertFalse(ZfsAutobackup(
"test test_target1 --no-progress --verbose --keep-source=0 --keep-target=10 --allow-empty --no-send".split(
" ")).run())
with mocktime("20101111000002"):
self.assertFalse(ZfsAutobackup(
"test test_target1 --no-progress --verbose --keep-source=0 --keep-target=10 --allow-empty".split(
" ")).run())
r = shelltest("zfs list -H -o name -r -t snapshot,filesystem " + TEST_POOLS)
self.assertMultiLineEqual(r, """
test_source1
test_source1/fs1
test_source1/fs1/sub
test_source2
test_source2/fs2
test_source2/fs2/sub
test_source2/fs3
test_source2/fs3/sub
test_target1
test_target1/test_source1
test_target1/test_source1/fs1
test_target1/test_source1/fs1@test-20101111000000
test_target1/test_source1/fs1@test-20101111000001
test_target1/test_source1/fs1@test-20101111000002
test_target1/test_source1/fs1/sub
test_target1/test_source1/fs1/sub@test-20101111000000
test_target1/test_source1/fs1/sub@test-20101111000001
test_target1/test_source1/fs1/sub@test-20101111000002
test_target1/test_source2
test_target1/test_source2/fs2
test_target1/test_source2/fs2/sub
test_target1/test_source2/fs2/sub@test-20101111000000
test_target1/test_source2/fs2/sub@test-20101111000001
test_target1/test_source2/fs2/sub@test-20101111000002
""")
def test_migrate_from_mismatching_bookmarkname(self):
"""test migration from a bookmark with a mismatching name."""
shelltest("zfs snapshot test_source1/fs1@migrate1")
shelltest("zfs create test_target1/test_source1")
shelltest("zfs send test_source1/fs1@migrate1| zfs recv test_target1/test_source1/fs1")
# make it so there is only the bookmark to migrate from, which a random non-matching name
shelltest("zfs bookmark test_source1/fs1@migrate1 \#randombookmarkname")
shelltest("zfs destroy test_source1/fs1@migrate1")
with mocktime("20101111000000"):
self.assertFalse(ZfsAutobackup("test test_target1 --no-progress --verbose --test".split(" ")).run())
with mocktime("20101111000000"):
self.assertFalse(ZfsAutobackup("test test_target1 --no-progress --verbose".split(" ")).run())
r = shelltest("zfs list -H -o name -r -t snapshot,filesystem " + TEST_POOLS)
self.assertMultiLineEqual(r, """
test_source1
test_source1/fs1
test_source1/fs1@test-20101111000000
test_source1/fs1/sub
test_source1/fs1/sub@test-20101111000000
test_source2
test_source2/fs2
test_source2/fs2/sub
test_source2/fs2/sub@test-20101111000000
test_source2/fs3
test_source2/fs3/sub
test_target1
test_target1/test_source1
test_target1/test_source1/fs1
test_target1/test_source1/fs1@migrate1
test_target1/test_source1/fs1@test-20101111000000
test_target1/test_source1/fs1/sub
test_target1/test_source1/fs1/sub@test-20101111000000
test_target1/test_source2
test_target1/test_source2/fs2
test_target1/test_source2/fs2/sub
test_target1/test_source2/fs2/sub@test-20101111000000
""")
def test_migrate_from_mismatching_snapshotname(self):
"""test migration from a snapshot with a mismatching name."""
shelltest("zfs snapshot test_source1/fs1@migrate1")
shelltest("zfs create test_target1/test_source1")
shelltest("zfs send test_source1/fs1@migrate1| zfs recv test_target1/test_source1/fs1")
# rename it so the names mismatch and guid matching is needed to resolve it
shelltest("zfs rename test_source1/fs1@migrate1 test_source1/fs1@randomsnapshotname")
# testmode
with mocktime("20101111000000"):
self.assertFalse(ZfsAutobackup("test test_target1 --no-progress --verbose --test".split(" ")).run())
with mocktime("20101111000000"):
self.assertFalse(ZfsAutobackup("test test_target1 --no-progress --verbose".split(" ")).run())
r = shelltest("zfs list -H -o name -r -t snapshot,filesystem " + TEST_POOLS)
self.assertMultiLineEqual(r, """
test_source1
test_source1/fs1
test_source1/fs1@randomsnapshotname
test_source1/fs1@test-20101111000000
test_source1/fs1/sub
test_source1/fs1/sub@test-20101111000000
test_source2
test_source2/fs2
test_source2/fs2/sub
test_source2/fs2/sub@test-20101111000000
test_source2/fs3
test_source2/fs3/sub
test_target1
test_target1/test_source1
test_target1/test_source1/fs1
test_target1/test_source1/fs1@migrate1
test_target1/test_source1/fs1@test-20101111000000
test_target1/test_source1/fs1/sub
test_target1/test_source1/fs1/sub@test-20101111000000
test_target1/test_source2
test_target1/test_source2/fs2
test_target1/test_source2/fs2/sub
test_target1/test_source2/fs2/sub@test-20101111000000
""")
def test_keep0_bookmarks(self):
"""test if keep-source=0 and keep-target=0 only keeps a bookmark at the source. (contrary to legacy holds-mode)"""
# backup mode
with mocktime("20101111000000"):
self.assertFalse(ZfsAutobackup(
"test test_target1 --no-progress --verbose --keep-source=0 --keep-target=0".split(
" ")).run())
# snapshot mode, will delete 0
with mocktime("20101111000001"):
self.assertFalse(ZfsAutobackup(
"test --no-progress --verbose --keep-source=0 --keep-target=0 --allow-empty".split(" ")).run())
# snapshot mode, will detete 1
with mocktime("20101111000002"):
self.assertFalse(ZfsAutobackup(
"test --no-progress --verbose --keep-source=0 --keep-target=0 --allow-empty".split(" ")).run())
# now we have only bookmark 0 and snapshot 2 at source
r = shelltest("zfs list -H -o name -r -t snapshot,filesystem,bookmark " + TEST_POOLS)
self.assertRegexpMatches(r, """
test_source1
test_source1/fs1
test_source1/fs1@test-20101111000002
test_source1/fs1#test-20101111000000__[0-9]*
test_source1/fs1/sub
test_source1/fs1/sub@test-20101111000002
test_source1/fs1/sub#test-20101111000000__[0-9]*
test_source2
test_source2/fs2
test_source2/fs2/sub
test_source2/fs2/sub@test-20101111000002
test_source2/fs2/sub#test-20101111000000__[0-9]*
test_source2/fs3
test_source2/fs3/sub
test_target1
test_target1/test_source1
test_target1/test_source1/fs1
test_target1/test_source1/fs1@test-20101111000000
test_target1/test_source1/fs1/sub
test_target1/test_source1/fs1/sub@test-20101111000000
test_target1/test_source2
test_target1/test_source2/fs2
test_target1/test_source2/fs2/sub
test_target1/test_source2/fs2/sub@test-20101111000000
""")
# make another backup now the source only has bookmarks of 3 left
with mocktime("20101111000003"):
self.assertFalse(ZfsAutobackup(
"test test_target1 --no-progress --verbose --keep-source=0 --keep-target=0 --allow-empty --debug".split(
" ")).run())
r = shelltest("zfs list -H -o name -r -t snapshot,filesystem,bookmark " + TEST_POOLS)
self.assertRegexpMatches(r, """
test_source1
test_source1/fs1
test_source1/fs1#test-20101111000003__[0-9]*
test_source1/fs1/sub
test_source1/fs1/sub#test-20101111000003__[0-9]*
test_source2
test_source2/fs2
test_source2/fs2/sub
test_source2/fs2/sub#test-20101111000003__[0-9]*
test_source2/fs3
test_source2/fs3/sub
test_target1
test_target1/test_source1
test_target1/test_source1/fs1
test_target1/test_source1/fs1@test-20101111000003
test_target1/test_source1/fs1/sub
test_target1/test_source1/fs1/sub@test-20101111000003
test_target1/test_source2
test_target1/test_source2/fs2
test_target1/test_source2/fs2/sub
test_target1/test_source2/fs2/sub@test-20101111000003
""")
def test_transfer_thinning_bookmarks(self):
# test pre/post/during transfer thinning and efficient transfer (no transerring of stuff that gets deleted on target) WITH BOOKMARK SUPPORT
# less output
shelltest("zfs set autobackup:test2=true test_source1/fs1/sub")
# nobody wants this one, will be destroyed before transferring (over a year ago)
with mocktime("20000101000000"):
self.assertFalse(ZfsAutobackup("test2 --allow-empty".split(" ")).run())
# only target wants this one (monthlys)
with mocktime("20010101000000"):
self.assertFalse(ZfsAutobackup("test2 --allow-empty".split(" ")).run())
# both want this one (dayly + monthly)
# other snapshots should influence the middle one that we actually want.
with mocktime("20010201000000"):
shelltest("zfs snapshot test_source1/fs1/sub@other1")
self.assertFalse(ZfsAutobackup("test2 --allow-empty".split(" ")).run())
shelltest("zfs snapshot test_source1/fs1/sub@other2")
# only source wants this one (dayly)
with mocktime("20010202000000"):
self.assertFalse(ZfsAutobackup("test2 --allow-empty".split(" ")).run())
with OutputIO() as buf:
with redirect_stdout(buf):
# now do thinning and transfer all at once
with mocktime("20010203000000"):
self.assertFalse(ZfsAutobackup(
"--keep-source=1d10d --keep-target=1m10m --allow-empty --verbose --clear-mountpoint --other-snapshots test2 test_target1".split(
" ")).run())
print(buf.getvalue())
self.assertIn(
"""
[Source] test_source1/fs1/sub@test2-20000101000000: Destroying
[Source] test_source1/fs1/sub@test2-20010101000000: -> test_target1/test_source1/fs1/sub (new)
[Source] test_source1/fs1/sub@test2-20010101000000: Destroying
[Source] test_source1/fs1/sub@other1: -> test_target1/test_source1/fs1/sub
[Source] test_source1/fs1/sub@test2-20010201000000: -> test_target1/test_source1/fs1/sub
[Source] test_source1/fs1/sub@other2: -> test_target1/test_source1/fs1/sub
[Source] test_source1/fs1/sub@test2-20010203000000: -> test_target1/test_source1/fs1/sub
""", buf.getvalue())
# note: also important that it leaves only the last bookmark on the source.
r = shelltest("zfs list -H -o name -r -t snapshot,bookmark test_source1 test_target1")
self.assertRegexpMatches(r, """
test_source1/fs1/sub@other1
test_source1/fs1/sub@test2-20010201000000
test_source1/fs1/sub@other2
test_source1/fs1/sub@test2-20010202000000
test_source1/fs1/sub@test2-20010203000000
test_source1/fs1/sub#test2-20010203000000__[0-9]*
test_target1/test_source1/fs1/sub@test2-20010101000000
test_target1/test_source1/fs1/sub@other1
test_target1/test_source1/fs1/sub@test2-20010201000000
test_target1/test_source1/fs1/sub@other2
test_target1/test_source1/fs1/sub@test2-20010203000000
""")
def test_filter_autobackup_properties(self):
"""All autobackup:* properties from the source must be filtered during
replication, and properties set directly on the target must not be
touched by subsequent backups."""
# an extra autobackup:* property on the source (besides the default autobackup:test=true on test_source1/fs1)
shelltest("zfs set autobackup:extra=sourceonly test_source1/fs1")
# first backup, creates target hierarchy
with mocktime("20101111000000"):
self.assertFalse(ZfsAutobackup(
"test test_target1 --no-progress --verbose".split(" ")).run())
# set autobackup:* properties directly on the target.
# autobackup:test deliberately collides with the source property to verify the source
# value does not overwrite the target on subsequent receives.
shelltest("zfs set autobackup:targetonly=targetvalue test_target1/test_source1/fs1")
shelltest("zfs set autobackup:test=false test_target1/test_source1/fs1/sub")
properties = "autobackup:test,autobackup:extra,autobackup:targetonly"
before = shelltest(
"zfs get -H -o name,property,value,source -r -s local,received -t filesystem "
+ properties + " test_target1")
# second backup, must not change any autobackup:* property on the target
with mocktime("20101111000001"):
self.assertFalse(ZfsAutobackup(
"test test_target1 --no-progress --verbose --allow-empty".split(" ")).run())
after = shelltest(
"zfs get -H -o name,property,value,source -r -s local,received -t filesystem "
+ properties + " test_target1")
self.assertMultiLineEqual(before, after)
# autobackup:extra was only set on the source and must not have leaked to the target
self.assertNotIn("sourceonly", after)
+48 -43
View File
@@ -9,7 +9,6 @@ class TestZfsCheck(unittest2.TestCase):
def setUp(self):
pass
def test_volume(self):
if exists("/.dockerenv"):
@@ -23,33 +22,36 @@ class TestZfsCheck(unittest2.TestCase):
with self.subTest("Generate"):
with OutputIO() as buf:
with redirect_stdout(buf):
self.assertFalse(ZfsCheck("test_source1/vol@test".split(" "),print_arguments=False).run())
self.assertFalse(ZfsCheck("test_source1/vol@test".split(" "), print_arguments=False).run())
print(buf.getvalue())
self.assertEqual("""0 2c2ceccb5ec5574f791d45b63c940cff20550f9a
1 2c2ceccb5ec5574f791d45b63c940cff20550f9a
""", buf.getvalue())
#store on disk for next step, add one error.
# store on disk for next step, add one error.
with open("/tmp/testhashes", "w") as fh:
fh.write(buf.getvalue()+"1\t2c2ceccb5ec5574f791d45b63c940cff20550f9X")
fh.write(buf.getvalue() + "1\t2c2ceccb5ec5574f791d45b63c940cff20550f9X")
with self.subTest("Compare"):
with OutputIO() as buf:
with redirect_stdout(buf):
self.assertEqual(1, ZfsCheck("test_source1/vol@test --check=/tmp/testhashes".split(" "),print_arguments=False).run())
self.assertEqual(1, ZfsCheck("test_source1/vol@test --check=/tmp/testhashes".split(" "),
print_arguments=False).run())
print(buf.getvalue())
self.assertEqual("Chunk 1 failed: 2c2ceccb5ec5574f791d45b63c940cff20550f9X 2c2ceccb5ec5574f791d45b63c940cff20550f9a\n", buf.getvalue())
self.assertEqual(
"Chunk 1 failed: 2c2ceccb5ec5574f791d45b63c940cff20550f9X 2c2ceccb5ec5574f791d45b63c940cff20550f9a\n",
buf.getvalue())
def test_filesystem(self):
prepare_zpools()
shelltest("cp tests/data/whole /test_source1/testfile")
shelltest("cp data/whole /test_source1/testfile")
shelltest("mkdir /test_source1/emptydir")
shelltest("mkdir /test_source1/dir")
shelltest("cp tests/data/whole2 /test_source1/dir/testfile")
shelltest("cp data/whole2 /test_source1/dir/testfile")
#it should ignore these:
# it should ignore these:
shelltest("ln -s / /test_source1/symlink")
shelltest("mknod /test_source1/c c 1 1")
shelltest("mknod /test_source1/b b 1 1")
@@ -67,24 +69,27 @@ class TestZfsCheck(unittest2.TestCase):
dir/testfile 0 2e863f1fcccd6642e4e28453eba10d2d3f74d798
""", buf.getvalue())
#store on disk for next step, add error
# store on disk for next step, add error
with open("/tmp/testhashes", "w") as fh:
fh.write(buf.getvalue()+"dir/testfile 0 2e863f1fcccd6642e4e28453eba10d2d3f74d79X")
fh.write(buf.getvalue() + "dir/testfile 0 2e863f1fcccd6642e4e28453eba10d2d3f74d79X")
with self.subTest("Compare"):
with OutputIO() as buf:
with redirect_stdout(buf):
self.assertEqual(1, ZfsCheck("test_source1@test --check=/tmp/testhashes".split(" "),print_arguments=False).run())
self.assertEqual(1, ZfsCheck("test_source1@test --check=/tmp/testhashes".split(" "),
print_arguments=False).run())
print(buf.getvalue())
self.assertEqual("dir/testfile: Chunk 0 failed: 2e863f1fcccd6642e4e28453eba10d2d3f74d79X 2e863f1fcccd6642e4e28453eba10d2d3f74d798\n", buf.getvalue())
self.assertEqual(
"dir/testfile: Chunk 0 failed: 2e863f1fcccd6642e4e28453eba10d2d3f74d79X 2e863f1fcccd6642e4e28453eba10d2d3f74d798\n",
buf.getvalue())
def test_file(self):
with self.subTest("Generate"):
with OutputIO() as buf:
with redirect_stdout(buf):
self.assertFalse(ZfsCheck("tests/data/whole".split(" "), print_arguments=False).run())
self.assertFalse(ZfsCheck("data/whole".split(" "), print_arguments=False).run())
print(buf.getvalue())
self.assertEqual("""0 3c0bf91170d873b8e327d3bafb6bc074580d11b7
@@ -92,22 +97,25 @@ dir/testfile 0 2e863f1fcccd6642e4e28453eba10d2d3f74d798
# store on disk for next step, add error
with open("/tmp/testhashes", "w") as fh:
fh.write(buf.getvalue()+"0 3c0bf91170d873b8e327d3bafb6bc074580d11bX")
fh.write(buf.getvalue() + "0 3c0bf91170d873b8e327d3bafb6bc074580d11bX")
with self.subTest("Compare"):
with OutputIO() as buf:
with redirect_stdout(buf):
self.assertEqual(1,ZfsCheck("tests/data/whole --check=/tmp/testhashes".split(" "), print_arguments=False).run())
self.assertEqual(1, ZfsCheck("data/whole --check=/tmp/testhashes".split(" "),
print_arguments=False).run())
print(buf.getvalue())
self.assertEqual("Chunk 0 failed: 3c0bf91170d873b8e327d3bafb6bc074580d11bX 3c0bf91170d873b8e327d3bafb6bc074580d11b7\n", buf.getvalue())
self.assertEqual(
"Chunk 0 failed: 3c0bf91170d873b8e327d3bafb6bc074580d11bX 3c0bf91170d873b8e327d3bafb6bc074580d11b7\n",
buf.getvalue())
def test_tree(self):
shelltest("rm -rf /tmp/testtree; mkdir /tmp/testtree")
shelltest("cp tests/data/whole /tmp/testtree")
shelltest("cp tests/data/whole_whole2 /tmp/testtree")
shelltest("cp tests/data/whole2 /tmp/testtree")
shelltest("cp tests/data/partial /tmp/testtree")
shelltest("cp tests/data/whole_whole2_partial /tmp/testtree")
shelltest("cp data/whole /tmp/testtree")
shelltest("cp data/whole_whole2 /tmp/testtree")
shelltest("cp data/whole2 /tmp/testtree")
shelltest("cp data/partial /tmp/testtree")
shelltest("cp data/whole_whole2_partial /tmp/testtree")
####################################
with self.subTest("Generate, skip 1"):
@@ -115,13 +123,12 @@ dir/testfile 0 2e863f1fcccd6642e4e28453eba10d2d3f74d798
with redirect_stdout(buf):
self.assertFalse(ZfsCheck("/tmp/testtree --skip=1".split(" "), print_arguments=False).run())
#since order varies, just check count (there is one empty line for some reason, only when testing like this)
# since order varies, just check count (there is one empty line for some reason, only when testing like this)
print(buf.getvalue().split("\n"))
self.assertEqual(len(buf.getvalue().split("\n")),4)
self.assertEqual(len(buf.getvalue().split("\n")), 4)
######################################
with self.subTest("Compare, all incorrect, skip 1"):
# store on disk for next step, add error
with open("/tmp/testhashes", "w") as fh:
fh.write("""
@@ -134,13 +141,14 @@ whole_whole2_partial 0 309ffffba2e1977d12f3b7469971f30d28b94bdX
with OutputIO() as buf:
with redirect_stdout(buf):
self.assertEqual(ZfsCheck("/tmp/testtree --check=/tmp/testhashes --skip=1".split(" "), print_arguments=False).run(), 3)
self.assertEqual(ZfsCheck("/tmp/testtree --check=/tmp/testhashes --skip=1".split(" "),
print_arguments=False).run(), 3)
print(buf.getvalue())
self.assertMultiLineEqual("""partial: Chunk 0 failed: 642027d63bb0afd7e0ba197f2c66ad03e3d70deX 642027d63bb0afd7e0ba197f2c66ad03e3d70de1
whole2: Chunk 0 failed: 2e863f1fcccd6642e4e28453eba10d2d3f74d79X 2e863f1fcccd6642e4e28453eba10d2d3f74d798
whole_whole2_partial: Chunk 0 failed: 309ffffba2e1977d12f3b7469971f30d28b94bdX 309ffffba2e1977d12f3b7469971f30d28b94bd8
""",buf.getvalue())
""", buf.getvalue())
####################################
with self.subTest("Generate"):
@@ -148,10 +156,10 @@ whole_whole2_partial: Chunk 0 failed: 309ffffba2e1977d12f3b7469971f30d28b94bdX 3
with redirect_stdout(buf):
self.assertFalse(ZfsCheck("/tmp/testtree".split(" "), print_arguments=False).run())
#file order on disk can vary, so sort it..
sorted=buf.getvalue().split("\n")
# file order on disk can vary, so sort it..
sorted = buf.getvalue().split("\n")
sorted.sort()
sorted="\n".join(sorted)+"\n"
sorted = "\n".join(sorted) + "\n"
print(sorted)
self.assertEqual("""
@@ -181,14 +189,14 @@ whole_whole2_partial 0 309ffffba2e1977d12f3b7469971f30d28b94bd8
"""test if stuff is cleaned up correctly, in debugging mode , when a pipe breaks. """
prepare_zpools()
shelltest("cp tests/data/whole /test_source1/testfile")
shelltest("cp data/whole /test_source1/testfile")
shelltest("zfs snapshot test_source1@test")
#breaks pipe when head exists
#important to use --debug, since that generates extra output which would be problematic if we didnt do correct SIGPIPE handling
shelltest("python -m zfs_autobackup.ZfsCheck test_source1@test --debug | head -n1")
# breaks pipe when head exists
# important to use --debug, since that generates extra output which would be problematic if we didnt do correct SIGPIPE handling
shelltest("cd ..;python3 -m zfs_autobackup.ZfsCheck test_source1@test --debug | head -n1")
#should NOT be mounted anymore if cleanup went ok:
# should NOT be mounted anymore if cleanup went ok:
self.assertNotRegex(shelltest("mount"), "test_source1@test")
def test_brokenpipe_cleanup_volume(self):
@@ -199,12 +207,12 @@ whole_whole2_partial 0 309ffffba2e1977d12f3b7469971f30d28b94bd8
shelltest("zfs create -V200M test_source1/vol")
shelltest("zfs snapshot test_source1/vol@test")
#breaks pipe when grep exists:
#important to use --debug, since that generates extra output which would be problematic if we didnt do correct SIGPIPE handling
shelltest("python -m zfs_autobackup.ZfsCheck test_source1/vol@test --debug| grep -m1 'Hashing file'")
# breaks pipe when grep exists:
# important to use --debug, since that generates extra output which would be problematic if we didnt do correct SIGPIPE handling
shelltest("cd ..;python3 -m zfs_autobackup.ZfsCheck test_source1/vol@test --debug| grep -m1 'Hashing file'")
# time.sleep(1)
r = shelltest("zfs list -H -o name -r -t all " + TEST_POOLS)
r = shelltest("zfs list -H -o name -r -t snapshot,filesystem,volume " + TEST_POOLS)
self.assertMultiLineEqual("""
test_source1
test_source1/fs1
@@ -217,7 +225,4 @@ test_source2/fs2/sub
test_source2/fs3
test_source2/fs3/sub
test_target1
""",r )
""", r)
+55 -30
View File
@@ -1,6 +1,7 @@
from basetest import *
from zfs_autobackup.LogStub import LogStub
from zfs_autobackup.ExecuteNode import ExecuteError
from zfs_autobackup.ZfsSnapshot import ZfsSnapshot
class TestZfsNode(unittest2.TestCase):
@@ -12,13 +13,16 @@ class TestZfsNode(unittest2.TestCase):
def test_consistent_snapshot(self):
logger = LogStub()
description = "[Source]"
node = ZfsNode(utc=False, snapshot_time_format="test-%Y%m%d%H%M%S", hold_name="zfs_autobackup:test", logger=logger, description=description)
node = ZfsNode(utc=False, snapshot_time_format="test-%Y%m%d%H%M%S", hold_name="zfs_autobackup:test",
logger=logger, description=description, debug_output=False, ssh_to=None, ssh_config=None,
exclude_snapshot_patterns=None, thinner=None, readonly=False, tag_seperator='_')
with self.subTest("first snapshot"):
(selected_datasets, excluded_datasets)=node.selected_datasets(property_name="autobackup:test", exclude_paths=[], exclude_received=False,
exclude_unchanged=0)
(selected_datasets, excluded_datasets) = node.selected_datasets(property_name="autobackup:test",
exclude_paths=[],
exclude_unchanged=0)
node.consistent_snapshot(selected_datasets, "test-20101111000001", 100000)
r = shelltest("zfs list -H -o name -r -t all " + TEST_POOLS)
r = shelltest("zfs list -H -o name -r -t snapshot,filesystem " + TEST_POOLS)
self.assertEqual(r, """
test_source1
test_source1/fs1
@@ -35,10 +39,11 @@ test_target1
""")
with self.subTest("second snapshot, no changes, no snapshot"):
(selected_datasets, excluded_datasets)=node.selected_datasets(property_name="autobackup:test", exclude_paths=[], exclude_received=False,
exclude_unchanged=0)
(selected_datasets, excluded_datasets) = node.selected_datasets(property_name="autobackup:test",
exclude_paths=[],
exclude_unchanged=0)
node.consistent_snapshot(selected_datasets, "test-20101111000002", 1)
r = shelltest("zfs list -H -o name -r -t all " + TEST_POOLS)
r = shelltest("zfs list -H -o name -r -t snapshot,filesystem " + TEST_POOLS)
self.assertEqual(r, """
test_source1
test_source1/fs1
@@ -55,9 +60,11 @@ test_target1
""")
with self.subTest("second snapshot, no changes, empty snapshot"):
(selected_datasets, excluded_datasets) =node.selected_datasets(property_name="autobackup:test", exclude_paths=[], exclude_received=False, exclude_unchanged=0)
(selected_datasets, excluded_datasets) = node.selected_datasets(property_name="autobackup:test",
exclude_paths=[],
exclude_unchanged=0)
node.consistent_snapshot(selected_datasets, "test-20101111000002", 0)
r = shelltest("zfs list -H -o name -r -t all " + TEST_POOLS)
r = shelltest("zfs list -H -o name -r -t snapshot,filesystem " + TEST_POOLS)
self.assertEqual(r, """
test_source1
test_source1/fs1
@@ -79,12 +86,16 @@ test_target1
def test_consistent_snapshot_prepostcmds(self):
logger = LogStub()
description = "[Source]"
node = ZfsNode(utc=False, snapshot_time_format="test", hold_name="test", logger=logger, description=description, debug_output=True)
node = ZfsNode(utc=False, snapshot_time_format="test", hold_name="test", logger=logger, description=description,
debug_output=True, ssh_to=None, ssh_config=None, exclude_snapshot_patterns=None, thinner=None,
readonly=False, tag_seperator='_')
with self.subTest("Test if all cmds are executed correctly (no failures)"):
with OutputIO() as buf:
with redirect_stdout(buf):
(selected_datasets, excluded_datasets) =node.selected_datasets(property_name="autobackup:test", exclude_paths=[], exclude_received=False, exclude_unchanged=0)
(selected_datasets, excluded_datasets) = node.selected_datasets(property_name="autobackup:test",
exclude_paths=[],
exclude_unchanged=0)
node.consistent_snapshot(selected_datasets, "test-1",
0,
pre_snapshot_cmds=["echo pre1", "echo pre2"],
@@ -96,12 +107,14 @@ test_target1
self.assertIn("STDOUT > post1", buf.getvalue())
self.assertIn("STDOUT > post2", buf.getvalue())
with self.subTest("Failure in the middle, only pre1 and both post1 and post2 should be executed, no snapshot should be attempted"):
with self.subTest(
"Failure in the middle, only pre1 and both post1 and post2 should be executed, no snapshot should be attempted"):
with OutputIO() as buf:
with redirect_stdout(buf):
with self.assertRaises(ExecuteError):
(selected_datasets, excluded_datasets) =node.selected_datasets(property_name="autobackup:test", exclude_paths=[], exclude_received=False, exclude_unchanged=0)
(selected_datasets, excluded_datasets) = node.selected_datasets(property_name="autobackup:test",
exclude_paths=[],
exclude_unchanged=0)
node.consistent_snapshot(selected_datasets, "test-1",
0,
pre_snapshot_cmds=["echo pre1", "false", "echo pre2"],
@@ -118,8 +131,10 @@ test_target1
with OutputIO() as buf:
with redirect_stdout(buf):
with self.assertRaises(ExecuteError):
#same snapshot name as before so it fails
(selected_datasets, excluded_datasets) =node.selected_datasets(property_name="autobackup:test", exclude_paths=[], exclude_received=False, exclude_unchanged=0)
# same snapshot name as before so it fails
(selected_datasets, excluded_datasets) = node.selected_datasets(property_name="autobackup:test",
exclude_paths=[],
exclude_unchanged=0)
node.consistent_snapshot(selected_datasets, "test-1",
0,
pre_snapshot_cmds=["echo pre1", "echo pre2"],
@@ -136,20 +151,22 @@ test_target1
# Assert that timestamps keep relative order both for utc and for localtime
logger = LogStub()
description = "[Source]"
node_local = ZfsNode(utc=False, snapshot_time_format="test-%Y%m%d%H%M%S", hold_name="zfs_autobackup:test", logger=logger, description=description)
node_utc = ZfsNode(utc=True, snapshot_time_format="test-%Y%m%d%H%M%S", hold_name="zfs_autobackup:test", logger=logger, description=description)
node_local = ZfsNode(utc=False, snapshot_time_format="test-%Y%m%d%H%M%S", hold_name="zfs_autobackup:test",
logger=logger, description=description, debug_output=False, ssh_to=None, ssh_config=None,
exclude_snapshot_patterns=None, thinner=None, readonly=False, tag_seperator='_')
node_utc = ZfsNode(utc=True, snapshot_time_format="test-%Y%m%d%H%M%S", hold_name="zfs_autobackup:test",
logger=logger, description=description, debug_output=False, ssh_to=None, ssh_config=None,
exclude_snapshot_patterns=None, thinner=None, readonly=False, tag_seperator='_')
for node in [node_local, node_utc]:
with self.subTest("timestamp ordering " + ("utc" if node == node_utc else "localtime")):
dataset_a = ZfsDataset(node,"test_source1@test-20101111000001")
dataset_b = ZfsDataset(node,"test_source1@test-20101111000002")
dataset_c = ZfsDataset(node,"test_source1@test-20240101020202")
dataset_a = ZfsSnapshot(node, "test_source1@test-20101111000001")
dataset_b = ZfsSnapshot(node, "test_source1@test-20101111000002")
dataset_c = ZfsSnapshot(node, "test_source1@test-20240101020202")
self.assertGreater(dataset_b.timestamp, dataset_a.timestamp)
self.assertGreater(dataset_c.timestamp, dataset_b.timestamp)
def test_getselected(self):
# should be excluded by property
shelltest("zfs create test_source1/fs1/subexcluded")
shelltest("zfs set autobackup:test=false test_source1/fs1/subexcluded")
@@ -165,9 +182,12 @@ test_target1
logger = LogStub()
description = "[Source]"
node = ZfsNode(utc=False, snapshot_time_format="test-%Y%m%d%H%M%S", hold_name="zfs_autobackup:test", logger=logger, description=description)
(selected_datasets, excluded_datasets)=node.selected_datasets(property_name="autobackup:test", exclude_paths=[], exclude_received=False,
exclude_unchanged=1)
node = ZfsNode(utc=False, snapshot_time_format="test-%Y%m%d%H%M%S", hold_name="zfs_autobackup:test",
logger=logger, description=description, debug_output=False, ssh_to=None, ssh_config=None,
exclude_snapshot_patterns=None, thinner=None, readonly=False, tag_seperator='_')
(selected_datasets, excluded_datasets) = node.selected_datasets(property_name="autobackup:test",
exclude_paths=[],
exclude_unchanged=1)
s = pformat(selected_datasets)
print(s)
@@ -177,11 +197,12 @@ test_target1
(local): test_source2/fs2/sub,
(local): test_source1/fs1/onlyparent]""")
def test_validcommand(self):
logger = LogStub()
description = "[Source]"
node = ZfsNode(utc=False, snapshot_time_format="test-%Y%m%d%H%M%S", hold_name="zfs_autobackup:test", logger=logger, description=description)
node = ZfsNode(utc=False, snapshot_time_format="test-%Y%m%d%H%M%S", hold_name="zfs_autobackup:test",
logger=logger, description=description, debug_output=False, ssh_to=None, ssh_config=None,
exclude_snapshot_patterns=None, thinner=None, readonly=False, tag_seperator='_')
with self.subTest("test invalid option"):
self.assertFalse(node.valid_command(["zfs", "send", "--invalid-option", "nonexisting"]))
@@ -191,7 +212,9 @@ test_target1
def test_supportedsendoptions(self):
logger = LogStub()
description = "[Source]"
node = ZfsNode(utc=False, snapshot_time_format="test-%Y%m%d%H%M%S", hold_name="zfs_autobackup:test", logger=logger, description=description)
node = ZfsNode(utc=False, snapshot_time_format="test-%Y%m%d%H%M%S", hold_name="zfs_autobackup:test",
logger=logger, description=description, debug_output=False, ssh_to=None, ssh_config=None,
exclude_snapshot_patterns=None, thinner=None, readonly=False, tag_seperator='_')
# -D propably always supported
self.assertGreater(len(node.supported_send_options), 0)
@@ -199,7 +222,9 @@ test_target1
logger = LogStub()
description = "[Source]"
# NOTE: this could hang via ssh if we dont close filehandles properly. (which was a previous bug)
node = ZfsNode(utc=False, snapshot_time_format="test-%Y%m%d%H%M%S", hold_name="zfs_autobackup:test", logger=logger, description=description, ssh_to='localhost')
node = ZfsNode(utc=False, snapshot_time_format="test-%Y%m%d%H%M%S", hold_name="zfs_autobackup:test",
logger=logger, description=description, ssh_to='localhost', debug_output=False, ssh_config=None,
exclude_snapshot_patterns=None, thinner=None, readonly=False, tag_seperator='_')
self.assertIsInstance(node.supported_recv_options, list)
-1
View File
@@ -1 +0,0 @@
.
+15 -17
View File
@@ -19,15 +19,14 @@ class BlockHasher():
def __init__(self, count=10000, bs=4096, hash_class=hashlib.sha1, skip=0):
self.count = count
self.bs = bs
self.chunk_size=bs*count
self.chunk_size = bs * count
self.hash_class = hash_class
# self.coverage=coverage
self.skip=skip
self._skip_count=0
self.stats_total_bytes=0
self.skip = skip
self._skip_count = 0
self.stats_total_bytes = 0
def _seek_next_chunk(self, fh, fsize):
"""seek fh to next chunk and update skip counter.
@@ -37,8 +36,8 @@ class BlockHasher():
"""
#ignore rempty files
if fsize==0:
# ignore rempty files
if fsize == 0:
return False
# need to skip chunks?
@@ -53,7 +52,7 @@ class BlockHasher():
# seek to next chunk, reset skip count
fh.seek(self.chunk_size * self._skip_count, os.SEEK_CUR)
self._skip_count = self.skip
return fh.tell()//self.chunk_size
return fh.tell() // self.chunk_size
else:
# should read this chunk, reset skip count
self._skip_count = self.skip
@@ -67,24 +66,23 @@ class BlockHasher():
yields nothing for empty files.
"""
with open(fname, "rb") as fh:
fh.seek(0, os.SEEK_END)
fsize=fh.tell()
fsize = fh.tell()
fh.seek(0)
while fh.tell()<fsize:
chunk_nr=self._seek_next_chunk(fh, fsize)
while fh.tell() < fsize:
chunk_nr = self._seek_next_chunk(fh, fsize)
if chunk_nr is False:
return
#read chunk
# read chunk
hash = self.hash_class()
block_nr = 0
while block_nr != self.count:
block=fh.read(self.bs)
if block==b"":
block = fh.read(self.bs)
if block == b"":
break
hash.update(block)
block_nr = block_nr + 1
@@ -121,7 +119,7 @@ class BlockHasher():
yield (chunk_nr, hexdigest, hash.hexdigest())
except Exception as e:
yield ( chunk_nr , hexdigest, 'ERROR: '+str(e))
yield (chunk_nr, hexdigest, 'ERROR: ' + str(e))
except Exception as e:
yield ( '-', '-', 'ERROR: '+ str(e))
yield ('-', '-', 'ERROR: ' + str(e))
-39
View File
@@ -1,39 +0,0 @@
# NOTE: this should inherit from (object) to function correctly with python 2.7
class CachedProperty(object):
""" A property that is only computed once per instance and
then stores the result in _cached_properties of the object.
Source: https://github.com/bottlepy/bottle/commit/fa7733e075da0d790d809aa3d2f53071897e6f76
"""
def __init__(self, func):
self.__doc__ = getattr(func, '__doc__')
self.func = func
def __get__(self, obj, cls):
if obj is None:
return self
propname = self.func.__name__
if not hasattr(obj, '_cached_properties'):
obj._cached_properties = {}
if propname not in obj._cached_properties:
obj._cached_properties[propname] = self.func(obj)
# value = obj.__dict__[propname] = self.func(obj)
return obj._cached_properties[propname]
@staticmethod
def clear(obj):
"""clears cache of obj"""
if hasattr(obj, '_cached_properties'):
obj._cached_properties = {}
@staticmethod
def is_cached(obj, propname):
if hasattr(obj, '_cached_properties') and propname in obj._cached_properties:
return True
else:
return False
+16 -15
View File
@@ -10,12 +10,12 @@ class CliBase(object):
Overridden in subclasses that add stuff for the specific programs."""
# also used by setup.py
VERSION = "3.3"
HEADER = "{} v{} - (c)2022 E.H.Eefting (edwin@datux.nl)".format(os.path.basename(sys.argv[0]), VERSION)
VERSION = "4.0-rc1"
HEADER = "{} v{} - (c)2026 E.H.Eefting (edwin@datux.nl)".format(os.path.basename(sys.argv[0]), VERSION)
def __init__(self, argv, print_arguments=True):
self.parser=self.get_parser()
self.parser = self.get_parser()
self.args = self.parse_args(argv)
# helps with investigating failed regression tests:
@@ -52,6 +52,8 @@ class CliBase(object):
self.log = LogConsole(show_debug=args.debug, show_verbose=args.verbose, color=sys.stdout.isatty())
self.log.show_progress=args.progress
self.verbose(self.HEADER)
self.verbose("")
@@ -66,25 +68,24 @@ class CliBase(object):
epilog='Full manual at: https://github.com/psy0rz/zfs_autobackup')
# Basic options
group=parser.add_argument_group("Common options")
group = parser.add_argument_group("Common options")
group.add_argument('--help', '-h', action='store_true', help='show help')
group.add_argument('--test', '--dry-run', '-n', action='store_true',
help='Dry run, dont change anything, just show what would be done (still does all read-only '
'operations)')
help='Dry run, dont change anything, just show what would be done (still does all read-only '
'operations)')
group.add_argument('--verbose', '-v', action='store_true', help='verbose output')
group.add_argument('--debug', '-d', action='store_true',
help='Show zfs commands that are executed, stops after an exception.')
help='Show zfs commands that are executed, stops after an exception.')
group.add_argument('--debug-output', action='store_true',
help='Show zfs commands and their output/exit codes. (noisy)')
help='Show zfs commands and their output/exit codes. (noisy)')
group.add_argument('--progress', action='store_true',
help='show zfs progress output. Enabled automaticly on ttys. (use --no-progress to disable)')
help='show zfs progress output. Enabled automaticly on ttys. (use --no-progress to disable)')
group.add_argument('--no-progress', action='store_true',
help=argparse.SUPPRESS) # needed to workaround a zfs recv -v bug
help=argparse.SUPPRESS) # needed to workaround a zfs recv -v bug
group.add_argument('--utc', action='store_true',
help='Use UTC instead of local time when dealing with timestamps for both formatting and parsing. To snapshot in an ISO 8601 compliant time format you may for example specify --snapshot-format "{}-%%Y-%%m-%%dT%%H:%%M:%%SZ". Changing this parameter after-the-fact (existing snapshots) will cause their timestamps to be interpreted as a different time than before.')
help='Use UTC instead of local time when dealing with timestamps for both formatting and parsing. To snapshot in an ISO 8601 compliant time format you may for example specify --snapshot-format "{}-%%Y-%%m-%%dT%%H:%%M:%%SZ". Changing this parameter after-the-fact (existing snapshots) will cause their timestamps to be interpreted as a different time than before.')
group.add_argument('--version', action='store_true',
help='Show version.')
help='Show version.')
return parser
@@ -100,8 +101,8 @@ class CliBase(object):
def debug(self, txt):
self.log.debug(txt)
def progress(self, txt):
self.log.progress(txt)
def progress(self, txt, current=None, total=None, failed=None):
self.log.progress(txt, current, total, failed)
def clear_progress(self):
self.log.clear_progress()
+10 -13
View File
@@ -41,7 +41,7 @@ class CmdItem:
self.exit_handler = exit_handler
self.shell = shell
self.process = None
self.next = None #next item in pipe, set by CmdPipe
self.next = None # next item in pipe, set by CmdPipe
def __str__(self):
"""return copy-pastable version of command."""
@@ -126,7 +126,7 @@ class CmdPipe:
success = True
for item in self.items:
if item.exit_handler is not None:
success=item.exit_handler(item.process.returncode) and success
success = item.exit_handler(item.process.returncode) and success
return success
@@ -144,21 +144,20 @@ class CmdPipe:
for item in self.items:
if item.process.stdout in read_ready:
line = item.process.stdout.readline().decode('utf-8').rstrip()
if line != "":
item.stdout_handler(line)
else:
raw = item.process.stdout.readline()
if raw == b"":
eof_count = eof_count + 1
if item.next:
item.next.process.stdin.close()
else:
item.stdout_handler(raw.decode('utf-8').rstrip())
if item.process.stderr in read_ready:
line = item.process.stderr.readline().decode('utf-8').rstrip()
if line != "":
item.stderr_handler(line)
else:
raw = item.process.stderr.readline()
if raw == b"":
eof_count = eof_count + 1
else:
item.stderr_handler(raw.decode('utf-8').rstrip())
if item.process.poll() is not None:
done_count = done_count + 1
@@ -167,8 +166,6 @@ class CmdPipe:
if eof_count == len(selectors) and done_count == len(self.items):
break
def __create(self):
"""create actual processes, do piping and return selectors."""
+34 -26
View File
@@ -17,7 +17,7 @@ class ExecuteError(Exception):
class ExecuteNode(LogStub):
"""an endpoint to execute local or remote commands via ssh"""
PIPE=1
PIPE = 1
def __init__(self, ssh_config=None, ssh_to=None, readonly=False, debug_output=False):
"""ssh_config: custom ssh config
@@ -51,35 +51,35 @@ class ExecuteNode(LogStub):
def _quote(self, cmd):
"""return quoted version of command. if it has value PIPE it will add an actual | """
if cmd==self.PIPE:
return('|')
if cmd == self.PIPE:
return ('|')
else:
return cmd_quote(cmd)
def _shell_cmd(self, cmd, cwd):
"""prefix specified ssh shell to command and escape shell characters"""
ret=[]
ret = []
#add remote shell
# add remote shell
if not self.is_local():
#note: dont escape this part (executed directly without shell)
ret=["ssh"]
# note: dont escape this part (executed directly without shell)
ret = ["ssh"]
if self.ssh_config is not None:
ret.extend(["-F", self.ssh_config])
ret.append(self.ssh_to)
#note: DO escape from here, executed in either local or remote shell.
# note: DO escape from here, executed in either local or remote shell.
shell_str=""
shell_str = ""
#add cwd change?
# add cwd change?
if cwd is not None:
shell_str=shell_str + "cd " + self._quote(cwd) + "; "
shell_str = shell_str + "cd " + self._quote(cwd) + "; "
shell_str=shell_str + " ".join(map(self._quote, cmd))
shell_str = shell_str + " ".join(map(self._quote, cmd))
ret.append(shell_str)
@@ -100,8 +100,10 @@ class ExecuteNode(LogStub):
:param cmd: the actual command, should be a list, where the first item is the command
and the rest are parameters. use ExecuteNode.PIPE to add an unescaped |
(if you want to use system piping instead of python piping)
:type cmd: list[str|ExecuteNode.PIPE]
:param pipe: return CmdPipe instead of executing it. (pipe this into another run() command via inp=...)
:param inp: Can be None, a string or a CmdPipe that was previously returned.
:type inp: CmdPipe | str | None
:param tab_split: split tabbed files in output into a list
:param valid_exitcodes: list of valid exit codes for this command. Use [] to accept all exit codes. Default [0]
:param readonly: make this True if the command doesn't make any changes and is safe to execute in testmode
@@ -109,6 +111,7 @@ class ExecuteNode(LogStub):
:param return_stderr: return both stdout and stderr as a tuple. (normally only returns stdout)
:param return_all: return both stdout and stderr and exit_code as a tuple. (normally only returns stdout)
:param cwd: Change current working directory before executing command.
:rtype: list[str] | tuple[list[str], list[str]] | tuple[list[str], list[str], int] | CmdPipe
"""
@@ -121,7 +124,7 @@ class ExecuteNode(LogStub):
# stderr parser
error_lines = []
returned_exit_code=None
returned_exit_code = None
def stderr_handler(line):
if tab_split:
@@ -139,7 +142,8 @@ class ExecuteNode(LogStub):
self.debug("EXIT > {}".format(exit_code))
if (valid_exitcodes != []) and (exit_code not in valid_exitcodes):
self.error("Command \"{}\" returned exit code {} (valid codes: {})".format(cmd_item, exit_code, valid_exitcodes))
self.error("Command \"{}\" returned exit code {} (valid codes: {})".format(cmd_item, exit_code,
valid_exitcodes))
return False
return True
@@ -149,7 +153,7 @@ class ExecuteNode(LogStub):
if pipe:
# dont specify output handler, so it will get piped to next process
stdout_handler=None
stdout_handler = None
else:
# handle output manually, dont pipe it
def stdout_handler(line):
@@ -160,7 +164,8 @@ class ExecuteNode(LogStub):
self._parse_stdout(line)
# add shell command and handlers to pipe
cmd_item=CmdItem(cmd=self._shell_cmd(cmd, cwd), readonly=readonly, stderr_handler=stderr_handler, exit_handler=exit_handler, shell=self.is_local(), stdout_handler=stdout_handler)
cmd_item = CmdItem(cmd=self._shell_cmd(cmd, cwd), readonly=readonly, stderr_handler=stderr_handler,
exit_handler=exit_handler, shell=self.is_local(), stdout_handler=stdout_handler)
cmd_pipe.add(cmd_item)
# return CmdPipe instead of executing?
@@ -174,7 +179,7 @@ class ExecuteNode(LogStub):
# execute and calls handlers in CmdPipe
if not cmd_pipe.execute():
raise(ExecuteError("Last command returned error"))
raise (ExecuteError("Last command returned error"))
if return_all:
return output_lines, error_lines, cmd_item.process and cmd_item.process.returncode
@@ -183,7 +188,8 @@ class ExecuteNode(LogStub):
else:
return output_lines
def script(self, lines, inp=None, stdout_handler=None, stderr_handler=None, exit_handler=None, valid_exitcodes=None, readonly=False, hide_errors=False, pipe=False):
def script(self, lines, inp=None, stdout_handler=None, stderr_handler=None, exit_handler=None, valid_exitcodes=None,
readonly=False, hide_errors=False, pipe=False):
"""Run a multiline script on the node.
This is much more low level than run() and allows for finer grained control.
@@ -212,14 +218,14 @@ class ExecuteNode(LogStub):
# add stuff to existing pipe
cmd_pipe = inp
internal_stdout_handler=None
internal_stdout_handler = None
if stdout_handler is not None:
if self.debug_output:
def internal_stdout_handler(line):
self.debug("STDOUT > " + line.rstrip())
stdout_handler(line)
else:
internal_stdout_handler=stdout_handler
internal_stdout_handler = stdout_handler
def internal_stderr_handler(line):
self._parse_stderr(line, hide_errors)
@@ -243,16 +249,16 @@ class ExecuteNode(LogStub):
return True
#build command
cmd=[]
# build command
cmd = []
#add remote shell
# add remote shell
if not self.is_local():
#note: dont escape this part (executed directly without shell)
# note: dont escape this part (executed directly without shell)
cmd.append("ssh")
if self.ssh_config is not None:
cmd.append(["-F", self.ssh_config])
cmd.extend(["-F", self.ssh_config])
cmd.append(self.ssh_to)
@@ -260,7 +266,9 @@ class ExecuteNode(LogStub):
cmd.append("\n".join(lines))
# add shell command and handlers to pipe
cmd_item=CmdItem(cmd=cmd, readonly=readonly, stderr_handler=internal_stderr_handler, exit_handler=internal_exit_handler, stdout_handler=internal_stdout_handler, shell=self.is_local())
cmd_item = CmdItem(cmd=cmd, readonly=readonly, stderr_handler=internal_stderr_handler,
exit_handler=internal_exit_handler, stdout_handler=internal_stdout_handler,
shell=self.is_local())
cmd_pipe.add(cmd_item)
self.debug("SCRIPT > {}".format(cmd_pipe))
+52 -19
View File
@@ -2,28 +2,34 @@
from __future__ import print_function
import sys
import time
try:
import colorama
_COLORAMA_AVAILABLE = True
except ImportError:
_COLORAMA_AVAILABLE = False
class LogConsole:
"""Log-class that outputs to console, adding colors if needed"""
# ANSI escape used as a fallback when colorama isn't available
_CLEAR_LINE = "\033[2K\r"
def __init__(self, show_debug, show_verbose, color):
self.last_log = ""
self.show_debug = show_debug
self.show_verbose = show_verbose
self._progress_uncleared=False
self.show_progress=True
if color:
# try to use color, failback if colorama not available
self.colorama=False
try:
import colorama
global colorama
self.colorama = True
except ImportError:
pass
self._progress_uncleared = False
else:
self.colorama=False
self.colorama = color and _COLORAMA_AVAILABLE
self._failed=None
self._current=None
self._total=None
def error(self, txt):
self.clear_progress()
@@ -59,16 +65,43 @@ class LogConsole:
print("# " + txt)
sys.stdout.flush()
def progress(self, txt):
def progress(self, txt, current=None, total=None, failed=None):
if not self.show_progress:
return
if current is not None:
self._current=current
if total is not None:
self._total=total
if failed is not None:
self._failed=failed
"""print progress output to stderr (stays on same line)"""
self.clear_progress()
self._progress_uncleared=True
print(">>> {}\r".format(txt), end='', file=sys.stderr)
self._progress_uncleared = True
if self._current is None:
line=f">>> {txt}"
else:
if self._failed:
line=f">>> [{self._current}/{self._total}] {txt} ({self._failed} failed)"
else:
line=f">>> [{self._current}/{self._total}] {txt}"
print(line+"\r", end='', file=sys.stderr)
sys.stderr.flush()
def clear_progress(self):
if not self.show_progress:
return
if self._progress_uncleared:
import colorama
print(colorama.ansi.clear_line(), end='', file=sys.stderr)
# sys.stderr.flush()
self._progress_uncleared=False
if self.colorama:
print(colorama.ansi.clear_line(), end='', file=sys.stderr)
else:
print(self._CLEAR_LINE, end='', file=sys.stderr)
self._progress_uncleared = False
sys.stderr.flush()
+3 -3
View File
@@ -1,5 +1,5 @@
#Used for baseclasses that dont implement their own logging (Like ExecuteNode)
#Usually logging is implemented in subclasses (Like ZfsNode thats a subclass of ExecuteNode), but for regression testing its nice to have these stubs.
# Used for baseclasses that dont implement their own logging (Like ExecuteNode)
# Usually logging is implemented in subclasses (Like ZfsNode thats a subclass of ExecuteNode), but for regression testing its nice to have these stubs.
class LogStub:
"""Just a stub, usually overriden in subclasses."""
@@ -15,4 +15,4 @@ class LogStub:
print("WARNING: " + txt)
def error(self, txt):
print("ERROR : " + txt)
print("ERROR : " + txt)
+23 -19
View File
@@ -1,4 +1,3 @@
from .ThinnerRule import ThinnerRule
@@ -48,7 +47,6 @@ class Thinner:
now: if specified, use this time as current time
"""
# always keep a number of the last objets?
if self.always_keep:
# all of them
@@ -70,24 +68,30 @@ class Thinner:
# traverse objects
for thisobject in objects:
# important they are ints!
timestamp = int(thisobject.timestamp)
age = int(now) - timestamp
# store in the correct time blocks, per period-size, if not too old yet
# e.g.: look if there is ANY timeblock that wants to keep this object
keep = False
for rule in self.rules:
if age <= rule.ttl:
block_nr = int(timestamp / rule.period)
if block_nr not in time_blocks[rule.period]:
time_blocks[rule.period][block_nr] = True
keep = True
# keep it according to schedule, or keep it because it is in the keep_objects list
if keep or thisobject in keep_objects or thisobject in always_keep_objects:
# ignore stuff without timestamp, always keep those.
if thisobject.timestamp is None:
keeps.append(thisobject)
else:
removes.append(thisobject)
return keeps, removes
# important they are ints!
timestamp = int(thisobject.timestamp)
age = int(now) - timestamp
# store in the correct time blocks, per period-size, if not too old yet
# e.g.: look if there is ANY timeblock that wants to keep this object
keep = False
for rule in self.rules:
if age <= rule.ttl:
block_nr = int(timestamp / rule.period)
if block_nr not in time_blocks[rule.period]:
time_blocks[rule.period][block_nr] = True
keep = True
# keep it according to schedule, or keep it because it is in the keep_objects list
if keep or thisobject in keep_objects or thisobject in always_keep_objects:
keeps.append(thisobject)
else:
removes.append(thisobject)
return keeps, removes
+1 -1
View File
@@ -68,4 +68,4 @@ class ThinnerRule:
def __str__(self):
"""get schedule as a schedule string"""
return self.rule_str
return self.rule_str
+11 -16
View File
@@ -14,7 +14,7 @@ class TreeHasher():
:type block_hasher: BlockHasher
"""
self.block_hasher=block_hasher
self.block_hasher = block_hasher
def generate(self, start_path):
"""Use BlockHasher on every file in a tree, yielding the results
@@ -28,12 +28,11 @@ class TreeHasher():
for (dirpath, dirnames, filenames) in os.walk(start_path, onerror=walkerror):
for f in filenames:
file_path=os.path.join(dirpath, f)
file_path = os.path.join(dirpath, f)
if (not os.path.islink(file_path)) and os.path.isfile(file_path):
for (chunk_nr, hash) in self.block_hasher.generate(file_path):
yield ( os.path.relpath(file_path,start_path), chunk_nr, hash )
yield (os.path.relpath(file_path, start_path), chunk_nr, hash)
def compare(self, start_path, generator):
"""reads from generator and compares blocks
@@ -43,18 +42,14 @@ class TreeHasher():
"""
count=0
def filter_file_name( file_name, chunk_nr, hexdigest):
return ( chunk_nr, hexdigest )
count = 0
def filter_file_name(file_name, chunk_nr, hexdigest):
return (chunk_nr, hexdigest)
for file_name, group_generator in itertools.groupby(generator, lambda x: x[0]):
count=count+1
block_generator=itertools.starmap(filter_file_name, group_generator)
for ( chunk_nr, compare_hexdigest, actual_hexdigest) in self.block_hasher.compare(os.path.join(start_path,file_name), block_generator):
yield ( file_name, chunk_nr, compare_hexdigest, actual_hexdigest )
count = count + 1
block_generator = itertools.starmap(filter_file_name, group_generator)
for (chunk_nr, compare_hexdigest, actual_hexdigest) in self.block_hasher.compare(
os.path.join(start_path, file_name), block_generator):
yield (file_name, chunk_nr, compare_hexdigest, actual_hexdigest)
+58 -30
View File
@@ -36,81 +36,111 @@ class ZfsAuto(CliBase):
self.warning("--ignore-replicated has been renamed, using --exclude-unchanged")
args.exclude_unchanged = True
if args.exclude_received:
self.log.error("--exclude-received is no longer needed. zfs-autobackup now filters all the autobackup:... properties for newly received datasets to solve this problem. To get rid of existing properties on the target do this one time: 'zfs inherit -r autobackup:backup1 pool/backup1'")
sys.exit(255)
# Note: Before version v3.1-beta5, we always used exclude_received. This was a problem if you wanted to
# replicate an existing backup to another host and use the same backupname/snapshots. However, exclude_received
# may still need to be used to explicitly exclude a backup with the 'received' source property to avoid accidental
# recursive replication of a zvol that is currently being received in another session (as it will have changes).
# Follow up note: This isnt a problem anymore since v4.0, we now filter the autobackup-property by default to prevent these difficult issues.
self.exclude_paths = []
if args.ssh_source == args.ssh_target:
if args.target_path:
# target and source are the same, make sure to exclude target_path
self.verbose("NOTE: Source and target are on the same host, excluding target-path from selection.")
self.exclude_paths.append(args.target_path)
else:
if not args.exclude_received and not args.include_received:
self.verbose("NOTE: Source and target are on the same host, adding --exclude-received to commandline. (use --include-received to overrule)")
args.exclude_received = True
if args.test:
self.warning("TEST MODE - SIMULATING WITHOUT MAKING ANY CHANGES")
#format all the names
# format all the names
self.property_name = args.property_format.format(args.backup_name)
self.snapshot_time_format = args.snapshot_format.format(args.backup_name)
self.hold_name = args.hold_format.format(args.backup_name)
self.tag_seperator = args.tag_seperator
dt = datetime_now(args.utc)
self.verbose("")
self.verbose("Current time {} : {}".format(args.utc and "UTC" or " ", dt.strftime("%Y-%m-%d %H:%M:%S")))
self.verbose(
"Current time {} : {}".format(args.utc and "UTC" or " ", dt.strftime("%Y-%m-%d %H:%M:%S")))
self.verbose("Selecting dataset property : {}".format(self.property_name))
self.verbose("Snapshot format : {}".format(self.snapshot_time_format))
self.verbose("Timezone : {}".format("UTC" if args.utc else "Local"))
seperator_test = datetime_now(False).strftime(self.snapshot_time_format)
# according to man 8 zfs:
valid_tags = "_.: -"
if self.tag_seperator == '' or any(c not in valid_tags for c in self.tag_seperator):
self.log.error("Invalid tag seperator. Allowed characters: '{}'".format(valid_tags))
sys.exit(255)
if self.tag_seperator in seperator_test:
self.log.error("Tag seperator '{}' may not be used in snapshot format: {}".format(self.tag_seperator,
self.snapshot_time_format))
sys.exit(255)
if args.tag and self.tag_seperator in args.tag:
self.log.error(
"Tag '{}' may not contain tag seperator '{}'".format(args.tag, self.tag_seperator))
sys.exit(255)
if args.tag:
self.verbose("Tag : {}".format(self.tag_seperator + args.tag))
return args
def get_parser(self):
parser = super(ZfsAuto, self).get_parser()
#positional arguments
# positional arguments
parser.add_argument('backup_name', metavar='BACKUP-NAME', default=None, nargs='?',
help='Name of the backup to select')
parser.add_argument('target_path', metavar='TARGET-PATH', default=None, nargs='?',
help='Target ZFS filesystem (optional)')
# SSH options
group=parser.add_argument_group("SSH options")
group = parser.add_argument_group("SSH options")
group.add_argument('--ssh-config', metavar='CONFIG-FILE', default=None, help='Custom ssh client config')
group.add_argument('--ssh-source', metavar='USER@HOST', default=None,
help='Source host to pull backup from.')
help='Source host to pull backup from.')
group.add_argument('--ssh-target', metavar='USER@HOST', default=None,
help='Target host to push backup to.')
help='Target host to push backup to.')
group=parser.add_argument_group("String formatting options")
group = parser.add_argument_group("String formatting options")
group.add_argument('--property-format', metavar='FORMAT', default="autobackup:{}",
help='Dataset selection string format. Default: %(default)s')
help='Name of the ZFS user-property used to select datasets for this backup. '
'The literal "{}" is substituted with BACKUP-NAME. '
'Default: %(default)s')
group.add_argument('--snapshot-format', metavar='FORMAT', default="{}-%Y%m%d%H%M%S",
help='ZFS Snapshot string format. Default: %(default)s')
help='Format of the snapshot-name part after the "@". '
'The literal "{}" is substituted with BACKUP-NAME, then strftime-codes '
'(e.g. %%Y %%m %%d %%H %%M %%S) are expanded against the current time. '
'Must not contain the tag-seperator. Default: %(default)s')
group.add_argument('--hold-format', metavar='FORMAT', default="zfs_autobackup:{}",
help='ZFS hold string format. Default: %(default)s')
help='Name of the zfs-hold placed on snapshots to prevent accidental deletion. '
'The literal "{}" is substituted with BACKUP-NAME. '
'Default: %(default)s')
group.add_argument('--strip-path', metavar='N', default=0, type=int,
help='Number of directories to strip from target path.')
group.add_argument('--tag-seperator', metavar='STRING', default="__",
help="Tag seperator for snapshots and bookmarks. Default: %(default)s")
group.add_argument('--tag', metavar='TAG', default=None,
help='Backup tag to add to snapshots names. (For administrative purposes)')
group=parser.add_argument_group("Selection options")
group = parser.add_argument_group("Selection options")
group.add_argument('--ignore-replicated', action='store_true', help=argparse.SUPPRESS)
group.add_argument('--exclude-unchanged', metavar='BYTES', default=0, type=int,
help='Exclude datasets that have less than BYTES data changed since any last snapshot. (Use with proxmox HA replication)')
group.add_argument('--exclude-received', action='store_true',
help='Exclude datasets that have the origin of their autobackup: property as "received". '
'This can avoid recursive replication between two backup partners.')
group.add_argument('--include-received', action='store_true',
help=argparse.SUPPRESS)
help='Exclude datasets that have less than BYTES data changed since any last snapshot. (Use with proxmox HA replication)')
group.add_argument('--exclude-received', action='store_true', help=argparse.SUPPRESS)
def regex_argument_type(input_line):
@@ -119,16 +149,14 @@ class ZfsAuto(CliBase):
return re.compile(input_line)
except:
raise ValueError("Could not parse argument '{}' as a regular expression".format(input_line))
group.add_argument('--exclude-snapshot-pattern', action='append', default=[], type=regex_argument_type, help="Regular expression to match snapshots that will be ignored.")
group.add_argument('--exclude-snapshot-pattern', action='append', default=[], type=regex_argument_type,
help="Regular expression to match snapshots that will be ignored.")
return parser
def print_error_sources(self):
self.error(
"No source filesystems selected, please do a 'zfs set autobackup:{0}=true' on the source datasets "
"you want to select.".format(
self.args.backup_name))
"No source filesystems selected, please do a 'zfs set {}=true' on the source datasets "
"you want to select.".format(self.property_name))
def make_target_name(self, source_dataset):
"""make target_name from a source_dataset"""
return self.args.target_path + "/" + source_dataset.lstrip_path(self.args.strip_path)
+83 -211
View File
@@ -1,16 +1,14 @@
import argparse
from signal import signal, SIGPIPE
from .util import output_redir, sigpipe_handler, datetime_now
from .ZfsAuto import ZfsAuto
from . import compressors
from .ExecuteNode import ExecuteNode
from .Thinner import Thinner
from .ZfsDataset import ZfsDataset
from .ZfsAuto import ZfsAuto
from .ZfsNode import ZfsNode
from .ThinnerRule import ThinnerRule
from .sync_datasets import sync_datasets
from .util import sigpipe_handler, datetime_now
class ZfsAutobackup(ZfsAuto):
"""The main zfs-autobackup class. Start here, at run() :)"""
@@ -31,6 +29,15 @@ class ZfsAutobackup(ZfsAuto):
if args.allow_empty:
args.min_change = 0
# With --keep-source=0 nothing survives on the source between runs, so the
# per-dataset is_changed_ours() check has no reference snapshot to compare
# against — every dataset would get a fresh snapshot anyway. Force min-change=0
# to skip the our_snapshots / written@... lookups entirely (significant speedup
# on large dataset counts).
if args.keep_source == "0" and args.min_change > 0:
self.verbose("NOTE: --keep-source=0: forcing --min-change=0 (no source reference to compare against).")
args.min_change = 0
# if args.destroy_incompatible:
# args.rollback = True
@@ -47,6 +54,10 @@ class ZfsAutobackup(ZfsAuto):
if args.compress and args.zfs_compressed:
self.warning("Using --compress with --zfs-compressed, might be inefficient.")
if args.decrypt:
self.warning(
"Properties will not be sent over for datasets that will be decrypted. (zfs bug https://github.com/openzfs/zfs/issues/16275)")
return args
def get_parser(self):
@@ -73,12 +84,17 @@ class ZfsAutobackup(ZfsAuto):
group.add_argument('--no-guid-check', action='store_true',
help='Dont check guid of common snapshots. (faster)')
group = parser.add_argument_group("Transfer options")
group.add_argument('--no-send', action='store_true',
help='Don\'t transfer snapshots (useful for cleanups, or if you want a separate send-cronjob)')
group.add_argument('--no-holds', action='store_true',
help='Don\'t hold snapshots. (Faster. Allows you to destroy common snapshot.)')
group.add_argument('--no-bookmarks', action='store_true',
help='Don\'t use bookmarks.')
group.add_argument('--no-clone', action='store_true',
help="Don't preserve clone relationships during replication. "
"Source datasets that are clones will be replicated as full standalone datasets.")
group.add_argument('--clear-refreservation', action='store_true',
help='Filter "refreservation" property. (recommended, saves space. same as '
'--filter-properties refreservation)')
@@ -91,6 +107,7 @@ class ZfsAutobackup(ZfsAuto):
group.add_argument('--set-properties', metavar='PROPERTY=VALUE,...', type=str,
help='List of propererties to override when receiving filesystems. (you can still restore '
'them with zfs inherit -S)')
group.add_argument('--rollback', action='store_true',
help='Rollback changes to the latest target snapshot before starting. (normally you can '
'prevent changes by setting the readonly property on the target_path to on)')
@@ -110,12 +127,6 @@ class ZfsAutobackup(ZfsAuto):
group.add_argument('--zfs-compressed', action='store_true',
help='Transfer blocks that already have zfs-compression as-is.')
group.add_argument('--clones', metavar='POLICY', default='never',
choices=('never', 'simple'),
help='Clones support. (The default policy "never" expands clones into full independent datasets. '
'"simple" tries to reproduce the clone when the origin snapshot is already copied '
'in the same target root)')
group = parser.add_argument_group("Data transfer options")
group.add_argument('--compress', metavar='TYPE', default=None, nargs='?', const='zstd-fast',
choices=compressors.choices(),
@@ -125,8 +136,8 @@ class ZfsAutobackup(ZfsAuto):
help='Limit data transfer rate in Bytes/sec (e.g. 128K. requires mbuffer.)')
group.add_argument('--buffer', metavar='SIZE', default=None,
help='Add zfs send and recv buffers to smooth out IO bursts. (e.g. 128M. requires mbuffer)')
parser.add_argument('--buffer-chunk-size', metavar="BUFFERCHUNKSIZE", default=None,
help='Tune chunk size when mbuffer is used. (requires mbuffer.)')
group.add_argument('--buffer-chunk-size', metavar="BUFFERCHUNKSIZE", default=None,
help='Tune chunk size when mbuffer is used. (requires mbuffer.)')
group.add_argument('--send-pipe', metavar="COMMAND", default=[], action='append',
help='pipe zfs send output through COMMAND (can be used multiple times)')
group.add_argument('--recv-pipe', metavar="COMMAND", default=[], action='append',
@@ -148,104 +159,6 @@ class ZfsAutobackup(ZfsAuto):
return parser
# NOTE: this method also uses self.args. args that need extra processing are passed as function parameters:
def thin_missing_targets(self, target_dataset, used_target_datasets):
"""thin target datasets that are missing on the source.
:type used_target_datasets: list[ZfsDataset]
:type target_dataset: ZfsDataset
"""
self.debug("Thinning obsolete datasets")
missing_datasets = [dataset for dataset in target_dataset.recursive_datasets if
dataset not in used_target_datasets]
count = 0
for dataset in missing_datasets:
self.debug("analyse missing {}".format(dataset))
count = count + 1
if self.args.progress:
self.progress("Analysing missing {}/{}".format(count, len(missing_datasets)))
try:
dataset.debug("Missing on source, thinning")
dataset.thin()
except Exception as e:
dataset.error("Error during thinning of missing datasets ({})".format(str(e)))
# if self.args.progress:
# self.clear_progress()
# NOTE: this method also uses self.args. args that need extra processing are passed as function parameters:
def destroy_missing_targets(self, target_dataset, used_target_datasets):
"""destroy target datasets that are missing on the source and that meet the requirements
:type used_target_datasets: list[ZfsDataset]
:type target_dataset: ZfsDataset
"""
self.debug("Destroying obsolete datasets")
missing_datasets = [dataset for dataset in target_dataset.recursive_datasets if
dataset not in used_target_datasets]
count = 0
for dataset in missing_datasets:
count = count + 1
if self.args.progress:
self.progress("Analysing destroy missing {}/{}".format(count, len(missing_datasets)))
try:
# cant do anything without our own snapshots
if not dataset.our_snapshots:
if dataset.datasets:
# its not a leaf, just ignore
dataset.debug("Destroy missing: ignoring")
else:
dataset.verbose(
"Destroy missing: has no snapshots made by us (please destroy manually).")
else:
# past the deadline?
deadline_ttl = ThinnerRule("0s" + self.args.destroy_missing).ttl
now = datetime_now(self.args.utc).timestamp()
if dataset.our_snapshots[-1].timestamp + deadline_ttl > now:
dataset.verbose("Destroy missing: Waiting for deadline.")
else:
dataset.debug("Destroy missing: Removing our snapshots.")
# remove all our snaphots, except last, to safe space in case we fail later on
for snapshot in dataset.our_snapshots[:-1]:
snapshot.destroy(fail_exception=True)
# does it have other snapshots?
has_others = False
for snapshot in dataset.snapshots:
if not snapshot.is_ours():
has_others = True
break
if has_others:
dataset.verbose("Destroy missing: Still in use by other snapshots")
else:
if dataset.datasets:
dataset.verbose("Destroy missing: Still has children here.")
else:
dataset.verbose("Destroy missing.")
dataset.our_snapshots[-1].destroy(fail_exception=True)
dataset.destroy(fail_exception=True)
except Exception as e:
# if self.args.progress:
# self.clear_progress()
dataset.error("Error during --destroy-missing: {}".format(str(e)))
# if self.args.progress:
# self.clear_progress()
def get_send_pipes(self, logger):
"""determine the zfs send pipe"""
@@ -328,102 +241,24 @@ class ZfsAutobackup(ZfsAuto):
return ret
def make_target_name(self, source_dataset):
"""make target_name from a source_dataset"""
stripped=source_dataset.lstrip_path(self.args.strip_path)
if stripped!="":
return self.args.target_path + "/" + stripped
else:
return self.args.target_path
def check_target_names(self, source_node, source_datasets, target_node):
"""check all target names for collesions etc due to strip-options"""
self.debug("Checking target names:")
target_datasets={}
target_datasets = {}
for source_dataset in source_datasets:
target_name = self.make_target_name(source_dataset)
target_name = source_dataset.map_to_target_path(self.args.target_path, self.args.strip_path)
source_dataset.debug("-> {}".format(target_name))
if target_name in target_datasets:
raise Exception("Target collision: Target path {} encountered twice, due to: {} and {}".format(target_name, source_dataset, target_datasets[target_name]))
raise Exception(
"Target collision: Target path {} encountered twice, due to: {} and {}".format(target_name,
source_dataset,
target_datasets[
target_name]))
target_datasets[target_name]=source_dataset
# NOTE: this method also uses self.args. args that need extra processing are passed as function parameters:
def sync_datasets(self, source_node, source_datasets, target_node):
"""Sync datasets, or thin-only on both sides
:type target_node: ZfsNode
:type source_datasets: list of ZfsDataset
:type source_node: ZfsNode
"""
send_pipes = self.get_send_pipes(source_node.verbose)
recv_pipes = self.get_recv_pipes(target_node.verbose)
fail_count = 0
count = 0
target_datasets = []
for source_dataset in source_datasets:
# stats
if self.args.progress:
count = count + 1
self.progress("Analysing dataset {}/{} ({} failed)".format(count, len(source_datasets), fail_count))
try:
# determine corresponding target_dataset
target_name = self.make_target_name(source_dataset)
target_dataset = target_node.get_dataset(target_name)
target_datasets.append(target_dataset)
# ensure parents exists
# TODO: this isnt perfect yet, in some cases it can create parents when it shouldn't.
if not self.args.no_send \
and target_dataset.parent \
and target_dataset.parent not in target_datasets \
and not target_dataset.parent.exists:
target_dataset.debug("Creating unmountable parents")
target_dataset.parent.create_filesystem(parents=True)
# determine common zpool features (cached, so no problem we call it often)
source_features = source_node.get_pool(source_dataset).features
target_features = target_node.get_pool(target_dataset).features
common_features = source_features and target_features
# sync the snapshots of this dataset
source_dataset.sync_snapshots(target_dataset, show_progress=self.args.progress,
features=common_features, filter_properties=self.filter_properties_list(),
set_properties=self.set_properties_list(),
ignore_recv_exit_code=self.args.ignore_transfer_errors,
holds=not self.args.no_holds, rollback=self.args.rollback,
also_other_snapshots=self.args.other_snapshots,
no_send=self.args.no_send,
destroy_incompatible=self.args.destroy_incompatible,
send_pipes=send_pipes, recv_pipes=recv_pipes,
decrypt=self.args.decrypt, encrypt=self.args.encrypt,
zfs_compressed=self.args.zfs_compressed, force=self.args.force,
guid_check=not self.args.no_guid_check,
clones=self.args.clones,
make_target_name=lambda source_dataset: self.make_target_name(source_dataset))
except Exception as e:
fail_count = fail_count + 1
source_dataset.error("FAILED: " + str(e))
if self.args.debug:
self.verbose("Debug mode, aborting on first error")
raise
target_path_dataset = target_node.get_dataset(self.args.target_path)
if not self.args.no_thinning:
self.thin_missing_targets(target_dataset=target_path_dataset, used_target_datasets=target_datasets)
if self.args.destroy_missing is not None:
self.destroy_missing_targets(target_dataset=target_path_dataset, used_target_datasets=target_datasets)
return fail_count
target_datasets[target_name] = source_dataset
def thin_source(self, source_datasets):
@@ -482,14 +317,17 @@ class ZfsAutobackup(ZfsAuto):
ssh_config=self.args.ssh_config,
ssh_to=self.args.ssh_source, readonly=self.args.test,
debug_output=self.args.debug_output, description=description, thinner=source_thinner,
exclude_snapshot_patterns=self.args.exclude_snapshot_pattern)
exclude_snapshot_patterns=self.args.exclude_snapshot_pattern,
tag_seperator=self.tag_seperator)
################# select source datasets
self.set_title("Selecting")
( source_datasets, excluded_datasets) = source_node.selected_datasets(property_name=self.property_name,
exclude_received=self.args.exclude_received,
exclude_paths=self.exclude_paths,
exclude_unchanged=self.args.exclude_unchanged)
if self.args.progress:
self.progress(f"Loading...")
(source_datasets, excluded_datasets) = source_node.selected_datasets(property_name=self.property_name,
exclude_paths=self.exclude_paths,
exclude_unchanged=self.args.exclude_unchanged)
if not source_datasets and not excluded_datasets:
self.print_error_sources()
return 255
@@ -498,6 +336,9 @@ class ZfsAutobackup(ZfsAuto):
if not self.args.no_snapshot:
self.set_title("Snapshotting")
snapshot_name = datetime_now(self.args.utc).strftime(self.snapshot_time_format)
if self.args.tag:
snapshot_name = snapshot_name + self.tag_seperator + self.args.tag
source_node.consistent_snapshot(source_datasets, snapshot_name,
min_changed_bytes=self.args.min_change,
pre_snapshot_cmds=self.args.pre_snapshot_cmd,
@@ -520,26 +361,57 @@ class ZfsAutobackup(ZfsAuto):
ssh_to=self.args.ssh_target,
readonly=self.args.test, debug_output=self.args.debug_output,
description="[Target]",
thinner=target_thinner)
exclude_snapshot_patterns=self.args.exclude_snapshot_pattern,
thinner=target_thinner, tag_seperator=self.tag_seperator)
target_node.verbose("Receive datasets under: {}".format(self.args.target_path))
self.set_title("Synchronising")
# check if exists, to prevent vague errors
target_dataset = target_node.get_dataset(self.args.target_path)
target_dataset = target_node.get_container(self.args.target_path)
if not target_dataset.exists:
raise (Exception(
"Target path '{}' does not exist. Please create this dataset first.".format(target_dataset)))
bookmark_tag = target_dataset.properties['guid']
target_node.verbose("Bookmark tag: {}".format(bookmark_tag))
self.set_title("Synchronising")
# check for collisions due to strip-path
self.check_target_names(source_node, source_datasets, target_node)
# do the actual sync
# NOTE: even with no_send, no_thinning and no_snapshot it does a usefull thing because it checks if the common snapshots and shows incompatible snapshots
fail_count = self.sync_datasets(
fail_count = sync_datasets(
self.log,
source_node=source_node,
source_datasets=source_datasets,
target_node=target_node)
target_node=target_node,
bookmark_tag=bookmark_tag,
send_pipes=self.get_send_pipes(source_node.verbose),
recv_pipes=self.get_recv_pipes(target_node.verbose),
target_path=self.args.target_path,
strip_path=self.args.strip_path,
no_clone=self.args.no_clone,
no_send=self.args.no_send,
no_bookmarks=self.args.no_bookmarks,
no_thinning=self.args.no_thinning,
filter_properties=self.filter_properties_list(),
set_properties=self.set_properties_list(),
ignore_transfer_errors=self.args.ignore_transfer_errors,
holds=not self.args.no_holds,
rollback=self.args.rollback,
other_snapshots=self.args.other_snapshots,
destroy_incompatible=self.args.destroy_incompatible,
decrypt=self.args.decrypt,
encrypt=self.args.encrypt,
zfs_compressed=self.args.zfs_compressed,
force=self.args.force,
guid_check=not self.args.no_guid_check,
property_format=self.args.property_format,
debug=self.args.debug,
target_dataset_base=target_dataset,
destroy_missing=self.args.destroy_missing,
utc=self.args.utc)
# no target specified, run in snapshot-only mode
else:
@@ -581,7 +453,7 @@ def cli():
signal(SIGPIPE, sigpipe_handler)
failed_datasets=ZfsAutobackup(sys.argv[1:], False).run()
failed_datasets = ZfsAutobackup(sys.argv[1:], False).run()
sys.exit(min(failed_datasets, 255))
+28 -33
View File
@@ -76,14 +76,14 @@ def verify_filesystem(source_snapshot, source_mnt, target_snapshot, target_mnt,
source_snapshot.mount(source_mnt)
target_snapshot.mount(target_mnt)
if method=='rsync':
if method == 'rsync':
compare_trees_rsync(source_snapshot.zfs_node, source_mnt, target_snapshot.zfs_node, target_mnt)
# elif method == 'tar':
# compare_trees_tar(source_snapshot.zfs_node, source_mnt, target_snapshot.zfs_node, target_mnt)
elif method == 'find':
compare_trees_find(source_snapshot.zfs_node, source_mnt, target_snapshot.zfs_node, target_mnt)
else:
raise(Exception("program errror, unknown method"))
raise (Exception("program errror, unknown method"))
finally:
source_snapshot.unmount(source_mnt)
@@ -109,7 +109,6 @@ def verify_filesystem(source_snapshot, source_mnt, target_snapshot, target_mnt,
# return hashed
# def deacitvate_volume_snapshot(snapshot):
# clone_name=get_tmp_clone_name(snapshot)
# clone=snapshot.zfs_node.get_dataset(clone_name)
@@ -119,13 +118,13 @@ def verify_volume(source_dataset, source_snapshot, target_dataset, target_snapsh
"""compare the contents of two zfs volume snapshots"""
# try:
source_dev= activate_volume_snapshot(source_snapshot)
target_dev= activate_volume_snapshot(target_snapshot)
source_dev = activate_volume_snapshot(source_snapshot)
target_dev = activate_volume_snapshot(target_snapshot)
source_hash= hash_dev(source_snapshot.zfs_node, source_dev)
target_hash= hash_dev(target_snapshot.zfs_node, target_dev)
source_hash = hash_dev(source_snapshot.zfs_node, source_dev)
target_hash = hash_dev(target_snapshot.zfs_node, target_dev)
if source_hash!=target_hash:
if source_hash != target_hash:
raise Exception("md5hash difference: {} != {}".format(source_hash, target_hash))
# finally:
@@ -150,7 +149,7 @@ class ZfsAutoverify(ZfsAuto):
def parse_args(self, argv):
"""do extra checks on common args"""
args=super(ZfsAutoverify, self).parse_args(argv)
args = super(ZfsAutoverify, self).parse_args(argv)
if args.target_path == None:
self.log.error("Please specify TARGET-PATH")
@@ -161,17 +160,17 @@ class ZfsAutoverify(ZfsAuto):
def get_parser(self):
"""extend common parser with extra stuff needed for zfs-autobackup"""
parser=super(ZfsAutoverify, self).get_parser()
parser = super(ZfsAutoverify, self).get_parser()
group=parser.add_argument_group("Verify options")
group = parser.add_argument_group("Verify options")
group.add_argument('--fs-compare', metavar='METHOD', default="find", choices=["find", "rsync"],
help='Compare method to use for filesystems. (find, rsync) Default: %(default)s ')
help='Compare method to use for filesystems. (find, rsync) Default: %(default)s ')
return parser
def verify_datasets(self, source_mnt, source_datasets, target_node, target_mnt):
fail_count=0
fail_count = 0
count = 0
for source_dataset in source_datasets:
@@ -182,24 +181,24 @@ class ZfsAutoverify(ZfsAuto):
try:
# determine corresponding target_dataset
target_name = self.make_target_name(source_dataset)
target_dataset = target_node.get_dataset(target_name)
target_dataset = target_node.get_container(source_dataset.map_to_target_path(self.args.target_path, self.args.strip_path))
# find common snapshots to verify
source_snapshot = source_dataset.find_common_snapshot(target_dataset, True)
target_snapshot = target_dataset.find_snapshot(source_snapshot)
if source_snapshot is None or target_snapshot is None:
raise(Exception("Cant find common snapshot"))
raise (Exception("Cant find common snapshot"))
target_snapshot.verbose("Verifying...")
if source_dataset.properties['type']=="filesystem":
if source_dataset.properties['type'] == "filesystem":
verify_filesystem(source_snapshot, source_mnt, target_snapshot, target_mnt, self.args.fs_compare)
elif source_dataset.properties['type']=="volume":
elif source_dataset.properties['type'] == "volume":
verify_volume(source_dataset, source_snapshot, target_dataset, target_snapshot)
else:
raise(Exception("{} has unknown type {}".format(source_dataset, source_dataset.properties['type'])))
raise (
Exception("{} has unknown type {}".format(source_dataset, source_dataset.properties['type'])))
except Exception as e:
@@ -219,11 +218,10 @@ class ZfsAutoverify(ZfsAuto):
def run(self):
source_node=None
source_mnt=None
target_node=None
target_mnt=None
source_node = None
source_mnt = None
target_node = None
target_mnt = None
try:
@@ -240,10 +238,9 @@ class ZfsAutoverify(ZfsAuto):
################# select source datasets
self.set_title("Selecting")
( source_datasets, excluded_datasets) = source_node.selected_datasets(property_name=self.property_name,
exclude_received=self.args.exclude_received,
exclude_paths=self.exclude_paths,
exclude_unchanged=self.args.exclude_unchanged)
(source_datasets, excluded_datasets) = source_node.selected_datasets(property_name=self.property_name,
exclude_paths=self.exclude_paths,
exclude_unchanged=self.args.exclude_unchanged)
if not source_datasets and not excluded_datasets:
self.print_error_sources()
return 255
@@ -260,7 +257,7 @@ class ZfsAutoverify(ZfsAuto):
self.set_title("Verifying")
source_mnt, target_mnt= create_mountpoints(source_node, target_node)
source_mnt, target_mnt = create_mountpoints(source_node, target_node)
fail_count = self.verify_datasets(
source_mnt=source_mnt,
@@ -302,15 +299,13 @@ class ZfsAutoverify(ZfsAuto):
cleanup_mountpoint(target_node, target_mnt)
def cli():
import sys
raise(Exception("This program is incomplete, dont use it yet."))
raise (Exception("This program is incomplete, dont use it yet."))
signal(SIGPIPE, sigpipe_handler)
failed = ZfsAutoverify(sys.argv[1:], False).run()
sys.exit(min(failed,255))
sys.exit(min(failed, 255))
if __name__ == "__main__":
+26
View File
@@ -0,0 +1,26 @@
from .ZfsPointInTime import ZfsPointInTime
class ZfsBookmark(ZfsPointInTime):
"""A ZFS bookmark"""
@property
def prefix(self):
(filesystem, snapshot) = self.name.split("#")
return filesystem
@property
def suffix(self):
(filesystem, snapshot_name) = self.name.split("#")
return snapshot_name
@property
def typed_suffix(self):
return "#" + self.suffix
def destroy(self, fail_exception=False, **kwargs):
self.debug("Destroying")
return super().destroy(fail_exception=fail_exception)
+61 -42
View File
@@ -9,8 +9,10 @@ from .BlockHasher import BlockHasher
from .ZfsNode import ZfsNode
from .util import *
from .CliBase import CliBase
from .ZfsSnapshot import ZfsSnapshot
# NOTE: zfscheck has no ssh support, because it would be too problematic during cleanup of temporary mounts and clones.
class ZfsCheck(CliBase):
def __init__(self, argv, print_arguments=True):
@@ -18,7 +20,9 @@ class ZfsCheck(CliBase):
# NOTE: common options argument parsing are in CliBase
super(ZfsCheck, self).__init__(argv, print_arguments)
self.node = ZfsNode(self.log, utc=self.args.utc, readonly=self.args.test, debug_output=self.args.debug_output)
self.node = ZfsNode(self.log, utc=self.args.utc, readonly=self.args.test, debug_output=self.args.debug_output,
ssh_to=None, ssh_config=None, exclude_snapshot_patterns=None, thinner=None,
tag_seperator='__', description="", hold_name=None, snapshot_time_format=None)
self.block_hasher = BlockHasher(count=self.args.count, bs=self.args.block_size, skip=self.args.skip)
@@ -27,14 +31,16 @@ class ZfsCheck(CliBase):
parser = super(ZfsCheck, self).get_parser()
# positional arguments
parser.add_argument('target', metavar='TARGET', default=None, nargs='?', help='Target to checkum. (can be blockdevice, directory or ZFS snapshot)')
parser.add_argument('target', metavar='TARGET', default=None, nargs='?',
help='Target to checkum. (can be blockdevice, directory or ZFS snapshot)')
group = parser.add_argument_group('Checker options')
group.add_argument('--block-size', metavar="BYTES", default=4096, help="Read block-size, default %(default)s",
type=int)
group.add_argument('--count', metavar="COUNT", default=int((100 * (1024 ** 2)) / 4096),
help="Hash chunks of COUNT blocks. Default %(default)s . (CHUNK size is BYTES * COUNT) ", type=int) # 100MiB
help="Hash chunks of COUNT blocks. Default %(default)s . (CHUNK size is BYTES * COUNT) ",
type=int) # 100MiB
group.add_argument('--check', '-c', metavar="FILE", default=None, const=True, nargs='?',
help="Read hashes from STDIN (or FILE) and compare them")
@@ -57,15 +63,17 @@ class ZfsCheck(CliBase):
self.verbose("Target : {}".format(args.target))
self.verbose("Block size : {} bytes".format(args.block_size))
self.verbose("Block count : {}".format(args.count))
self.verbose("Effective chunk size : {} bytes".format(args.count*args.block_size))
self.verbose("Skip chunk count : {} (checks {:.2f}% of data)".format(args.skip, 100/(1+args.skip)))
self.verbose("Effective chunk size : {} bytes".format(args.count * args.block_size))
self.verbose("Skip chunk count : {} (checks {:.2f}% of data)".format(args.skip, 100 / (1 + args.skip)))
self.verbose("")
return args
def prepare_zfs_filesystem(self, snapshot):
"""
:type snapshot: ZfsSnapshot
:rtype: str
"""
mnt = "/tmp/" + tmp_name()
self.debug("Create temporary mount point {}".format(mnt))
self.node.run(["mkdir", mnt])
@@ -73,6 +81,9 @@ class ZfsCheck(CliBase):
return mnt
def cleanup_zfs_filesystem(self, snapshot):
"""
:type snapshot: ZfsSnapshot
"""
mnt = "/tmp/" + tmp_name()
snapshot.unmount(mnt)
self.debug("Cleaning up temporary mount point")
@@ -81,7 +92,10 @@ class ZfsCheck(CliBase):
# NOTE: https://www.google.com/search?q=Mount+Path+Limit+freebsd
# Freebsd has limitations regarding path length, so we have to clone it so the part stays sort
def prepare_zfs_volume(self, snapshot):
"""clone volume, waits and tries to findout /dev path to the volume, in a compatible way. (linux/freebsd/smartos)"""
"""clone volume, waits and tries to findout /dev path to the volume, in a compatible way. (linux/freebsd/smartos)
:type snapshot: ZfsSnapshot
:rtype: str
"""
clone_name = get_tmp_clone_name(snapshot)
clone = snapshot.clone(clone_name)
@@ -106,13 +120,18 @@ class ZfsCheck(CliBase):
time.sleep(1)
raise (Exception("Timeout while waiting for /dev entry to appear. (looking in: {}). Hint: did you forget to load the encryption key?".format(locations)))
raise (Exception(
"Timeout while waiting for /dev entry to appear. (looking in: {}). Hint: did you forget to load the encryption key?".format(
locations)))
def cleanup_zfs_volume(self, snapshot):
"""destroys temporary volume snapshot"""
"""destroys temporary volume snapshot
:type snapshot: ZfsSnapshot
"""
clone_name = get_tmp_clone_name(snapshot)
clone = snapshot.zfs_node.get_dataset(clone_name)
clone.destroy(deferred=True, verbose=False)
clone = snapshot.zfs_node.get_container(clone_name)
clone.destroy()
def generate_tree_hashes(self, prepared_target):
@@ -144,34 +163,34 @@ class ZfsCheck(CliBase):
"""parse input lines and yield items to use in compare functions"""
if self.args.check is True:
input_fh=sys.stdin
input_fh = sys.stdin
else:
input_fh=open(self.args.check, 'r')
input_fh = open(self.args.check, 'r')
last_progress_time = time.time()
progress_checked = 0
progress_skipped = 0
line=input_fh.readline()
skip=0
line = input_fh.readline()
skip = 0
while line:
i=line.rstrip().split("\t")
#ignores lines without tabs
if (len(i)>1):
i = line.rstrip().split("\t")
# ignores lines without tabs
if (len(i) > 1):
if skip==0:
progress_checked=progress_checked+1
if skip == 0:
progress_checked = progress_checked + 1
yield i
skip=self.args.skip
skip = self.args.skip
else:
skip=skip-1
progress_skipped=progress_skipped+1
skip = skip - 1
progress_skipped = progress_skipped + 1
if self.args.progress and time.time() - last_progress_time > 1:
last_progress_time = time.time()
self.progress("Checked {} hashes (skipped {})".format(progress_checked, progress_skipped))
line=input_fh.readline()
line = input_fh.readline()
self.verbose("Checked {} hashes (skipped {})".format(progress_checked, progress_skipped))
@@ -224,7 +243,7 @@ class ZfsCheck(CliBase):
if "@" in self.args.target:
# zfs snapshot
snapshot=self.node.get_dataset(self.args.target)
snapshot = self.node.get_snapshot(self.args.target)
if not snapshot.exists:
raise Exception("ZFS snapshot {} does not exist!".format(snapshot))
dataset_type = snapshot.parent.properties['type']
@@ -240,7 +259,7 @@ class ZfsCheck(CliBase):
def cleanup_target(self):
if "@" in self.args.target:
# zfs snapshot
snapshot=self.node.get_dataset(self.args.target)
snapshot = self.node.get_snapshot(self.args.target)
if not snapshot.exists:
return
@@ -253,28 +272,28 @@ class ZfsCheck(CliBase):
def run(self):
compare_generator=None
hash_generator=None
compare_generator = None
hash_generator = None
try:
prepared_target=self.prepare_target()
is_dir=os.path.isdir(prepared_target)
prepared_target = self.prepare_target()
is_dir = os.path.isdir(prepared_target)
#run as compare
# run as compare
if self.args.check is not None:
input_generator=self.generate_input()
input_generator = self.generate_input()
if is_dir:
compare_generator = self.generate_tree_compare(prepared_target, input_generator)
else:
compare_generator=self.generate_file_compare(prepared_target, input_generator)
errors=self.print_errors(compare_generator)
#run as generator
compare_generator = self.generate_file_compare(prepared_target, input_generator)
errors = self.print_errors(compare_generator)
# run as generator
else:
if is_dir:
hash_generator = self.generate_tree_hashes(prepared_target)
else:
hash_generator=self.generate_file_hashes(prepared_target)
hash_generator = self.generate_file_hashes(prepared_target)
errors=self.print_hashes(hash_generator)
errors = self.print_hashes(hash_generator)
except Exception as e:
self.error("Exception: " + str(e))
@@ -286,10 +305,10 @@ class ZfsCheck(CliBase):
return 255
finally:
#important to call check_output so that cleanup still functions in case of a broken pipe:
# important to call check_output so that cleanup still functions in case of a broken pipe:
# util.check_output()
#close generators, to make sure files are not in use anymore when cleaning up
# close generators, to make sure files are not in use anymore when cleaning up
if hash_generator is not None:
hash_generator.close()
if compare_generator is not None:
@@ -302,8 +321,8 @@ class ZfsCheck(CliBase):
def cli():
import sys
signal(SIGPIPE, sigpipe_handler)
failed=ZfsCheck(sys.argv[1:], False).run()
sys.exit(min(failed,255))
failed = ZfsCheck(sys.argv[1:], False).run()
sys.exit(min(failed, 255))
if __name__ == "__main__":
+622
View File
@@ -0,0 +1,622 @@
from typing import cast
from .ZfsBookmark import ZfsBookmark
from .ZfsDataset import ZfsDataset
from .ZfsPointInTime import ZfsPointInTime
from .ZfsSnapshot import ZfsSnapshot
class ZfsContainer(ZfsDataset):
"""Either a ZFS Filesystem or ZFS Dataset"""
def __init__(self, zfs_node, name, force_exists=None):
super().__init__(zfs_node, name, force_exists=force_exists)
self._written_since_ours = None # type: None|int
self._recursive_datasets = None # type: None|list[ZfsContainer]
self._datasets = None # type: None|list[ZfsContainer]
self._snapshots_bookmarks = None # type: None|list[ZfsSnapshot|ZfsBookmark]
# def __str__(self):
#
# return f"{self.zfs_node}: {self.name} {'force_exists' if self.force_exists else ''} {len(self._snapshots_bookmarks) if self._snapshots_bookmarks is not None else ''}"
@property
def parent(self):
"""get parent dataset
:rtype: ZfsContainer | None
"""
stripped = self.rstrip_path(1)
if stripped:
return self.zfs_node.get_container(stripped)
else:
return None
def map_to_target_path(self, target_path, strip_path):
"""Map this dataset's name to a target path by stripping strip_path components and prepending target_path.
:type target_path: str
:type strip_path: int
:rtype: str
"""
stripped = self.lstrip_path(strip_path)
if stripped:
return target_path + "/" + stripped
else:
return target_path
def invalidate_cache(self):
super().invalidate_cache()
self._written_since_ours = None
self._recursive_datasets = None
self._datasets = None
self._snapshots_bookmarks = None
@property
def snapshots(self):
"""get all snapshots of this dataset
:rtype: list[ZfsSnapshot]
"""
ret = []
for snapshot in self.snapshots_bookmarks:
# check is snapshot is a ZfsSnapshot, and append to ret if it is
if isinstance(snapshot, ZfsSnapshot):
ret.append(snapshot)
return ret
@property
def bookmarks(self):
"""get all bookmarks of this dataset
Args:
:rtype: list[ZfsBookmark]
"""
ret = []
for bookmark in self.snapshots_bookmarks:
if type(bookmark) is ZfsBookmark:
ret.append(bookmark)
return ret
@property
def snapshots_bookmarks(self):
"""get all snapshots and bookmarks of this dataset (ordered by createtxg, so its suitable to determine incremental zfs send order)
:rtype: list[ZfsSnapshot|ZfsBookmark]
"""
# cached?
if self._snapshots_bookmarks is None:
self.debug("Getting snapshots and bookmarks")
cmd = [
"zfs", "list", "-d", "1", "-r", "-t", "snapshot,bookmark", "-H", "-o", "name", "-s", "createtxg",
self.name
]
names = cast("list[str]", self.zfs_node.run(cmd=cmd, readonly=True))
self._snapshots_bookmarks = cast("list[ZfsSnapshot|ZfsBookmark]",
self.zfs_node.get_datasets(names, force_exists=True))
return self._snapshots_bookmarks
def find_incompatible_snapshots(self, target_common_snapshot, raw):
"""returns a list[snapshots] that is incompatible for a zfs recv onto
the common_snapshot. all direct followup snapshots with written=0 are
compatible.
in raw-mode nothing is compatible. issue #219
Args:
:type target_common_snapshot: ZfsSnapshot
:type raw: bool
"""
ret = []
if self.snapshots:
if target_common_snapshot is None:
# No common snapshot found: every existing target snapshot is in
# the way and must be handled by handle_incompatible_snapshots.
return list(self.snapshots)
followup = True
common_index = self.find_snapshot_index(target_common_snapshot)
assert common_index is not None
for snapshot in self.snapshots[common_index + 1:]:
if raw or not followup or int(snapshot.properties['written']) != 0:
followup = False
ret.append(snapshot)
return ret
@property
def written_since_ours(self):
"""get number of bytes written since our last snapshot
:rtype: int
"""
if self._written_since_ours is None:
latest_snapshot = self.our_snapshots[-1]
self.debug("Getting bytes written since our last snapshot")
cmd = ["zfs", "get", "-H", "-ovalue", "-p", "written@" + str(latest_snapshot), self.name]
output = cast("list[str]", self.zfs_node.run(readonly=True, tab_split=False, cmd=cmd, valid_exitcodes=[0]))
self._written_since_ours = int(output[0])
return self._written_since_ours
def is_changed(self, min_changed_bytes=1):
"""dataset is changed since ANY latest snapshot ?
Args:
:type min_changed_bytes: int
"""
self.debug("Checking if dataset is changed {} or more bytes".format(min_changed_bytes))
if min_changed_bytes == 0:
return True
if int(self.properties['written']) < min_changed_bytes:
return False
else:
return True
def is_changed_ours(self, min_changed_bytes=1):
"""dataset is changed since OUR latest snapshot?
Args:
:type min_changed_bytes: int
"""
if min_changed_bytes == 0:
return True
if not self.our_snapshots:
return True
# NOTE: filesystems can have a very small amount written without actual changes in some cases
if self.written_since_ours < min_changed_bytes:
return False
return True
def find_snapshot(self, snapshot_name):
"""find snapshot by snapshot name (can be a suffix or a different
ZfsSnapshot) Returns None if it can't find it.
Note that matches with our own snapshots will be done tagless.
Args:
:rtype: ZfsSnapshot|None
:type snapshot_name: str|ZfsPointInTime|None
"""
if snapshot_name is None:
return None
if isinstance(snapshot_name, ZfsPointInTime):
tagless_suffix = snapshot_name.tagless_suffix
else:
tagless_suffix = snapshot_name
for snapshot_name in self.snapshots:
if snapshot_name.tagless_suffix == tagless_suffix:
return snapshot_name
return None
def find_next_snapshot(self, snapshot_bookmark):
"""find next snapshot in this dataset, according to snapshot or bookmark. None if it doesn't exist
Args:
:type snapshot_bookmark: ZfsSnapshot|ZfsBookmark
:rtype: ZfsSnapshot|None
"""
found = False
for snapshot in self.snapshots_bookmarks:
if snapshot == snapshot_bookmark:
found = True
else:
if found and type(snapshot) is ZfsSnapshot:
return snapshot
return None
def automount(self):
"""Mount the dataset as if one did a zfs mount -a, but only for this dataset
Failure to mount doesn't result in an exception, but outputs errors to STDERR.
"""
self.debug("Auto mounting")
if self.properties['type'] != "filesystem":
return
if self.properties['canmount'] != 'on':
return
if self.properties['mountpoint'] == 'legacy':
return
if self.properties['mountpoint'] == 'none':
return
if self.properties['encryption'] != 'off' and self.properties['keystatus'] == 'unavailable':
return
self.zfs_node.run(["zfs", "mount", self.name], valid_exitcodes=[0, 1])
def abort_resume(self):
"""abort current resume state"""
self.debug("Aborting resume")
self.zfs_node.run(["zfs", "recv", "-A", self.name])
if not self.snapshots:
#abort of an initial resume
self.force_exists=False
def rollback(self):
"""rollback to latest existing snapshot on this dataset"""
for snapshot in reversed(self.snapshots):
if snapshot.exists:
snapshot.rollback()
return
def thin_list(self, keeps=None, ignores=None):
"""determines list[snapshots] that should be kept or deleted based on
the thinning schedule. cull the herd!
returns: ( keeps, obsoletes )
Args:
:param keeps: list[snapshots] to always keep (usually the last)
:param ignores: snapshots to completely ignore (usually incompatible target snapshots that are going to be destroyed anyway)
:type keeps: list[ZfsSnapshot]
:type ignores: list[ZfsSnapshot]
:rtype: tuple[list[ZfsSnapshot], list[ZfsSnapshot]]
"""
if ignores is None:
ignores = []
if keeps is None:
keeps = []
snapshots = [snapshot for snapshot in self.our_snapshots if snapshot not in ignores]
return self.zfs_node.thin_list(snapshots, keep_snapshots=keeps)
def thin(self, skip_holds=False):
"""destroys snapshots according to thin_list, except last snapshot
Args:
:type skip_holds: bool
"""
(keeps, obsoletes) = self.thin_list(keeps=self.our_snapshots[-1:])
for obsolete in obsoletes:
if skip_holds and obsolete.is_hold():
obsolete.verbose("Keeping (common snapshot)")
else:
obsolete.destroy()
self.snapshots.remove(obsolete)
@property
def recursive_datasets(self, types="filesystem,volume"):
"""get all (non-snapshot) datasets recursively under us
Args:
:type types: str
:rtype: list[ZfsContainer]
"""
if self._recursive_datasets is None:
self.debug("Getting all recursive datasets under us")
names = cast("list[str]", self.zfs_node.run(tab_split=False, readonly=True, valid_exitcodes=[0], cmd=[
"zfs", "list", "-r", "-t", types, "-o", "name", "-H", self.name
]))
self._recursive_datasets = cast("list[ZfsContainer]",
self.zfs_node.get_datasets(names[1:], force_exists=True))
return self._recursive_datasets
@property
def datasets(self, types="filesystem,volume"):
"""get all (non-snapshot) datasets directly under us
Args:
:type types: str
:rtype: list[ZfsContainer]
"""
if self._datasets is None:
self.debug("Getting all datasets under us")
names = cast("list[str]", self.zfs_node.run(tab_split=False, readonly=True, valid_exitcodes=[0], cmd=[
"zfs", "list", "-r", "-t", types, "-o", "name", "-H", "-d", "1", self.name
]))
self._datasets = cast("list[ZfsContainer]",
self.zfs_node.get_datasets(names[1:], force_exists=True))
return self._datasets
def create_filesystem(self, parents=False, unmountable=True):
"""create this container as a filesystem
Args:
:type parents: bool
:type unmountable: bool
"""
# recurse up
parent = self.parent
if parents and parent is not None and not parent.exists:
parent.create_filesystem(parents, unmountable)
cmd = ["zfs", "create"]
if unmountable:
cmd.extend(["-o", "canmount=off"])
cmd.append(self.name)
self.zfs_node.run(cmd)
self.force_exists = True
if self.zfs_node.readonly:
self._snapshots_bookmarks=[]
self.simulate_properties()
def cache_snapshot_bookmark(self, snapshot, force=False):
"""Update our snapshot and bookmark cache (if we have any). Use force if you want to force the caching, potentially triggering a zfs list
Args:
:type snapshot: ZfsSnapshot|ZfsBookmark
:type force: bool
"""
if force:
self.snapshots_bookmarks.append(snapshot)
elif self._snapshots_bookmarks is not None:
self._snapshots_bookmarks.append(snapshot)
@property
def our_snapshots(self):
"""get list[snapshots] creates by us of this dataset"""
ret = []
for snapshot in self.snapshots:
if snapshot.is_ours:
ret.append(snapshot)
return ret
def find_guid_bookmark_snapshot(self, guid):
"""find the first bookmark or snapshot that matches, prefers bookmarks.
Args:
:type guid:str
:rtype: ZfsBookmark|ZfsSnapshot|None
"""
# Since this is slower, we only use it if the name matching with find_snapshot and find_bookmark doesn work.
for bookmark in self.bookmarks:
if bookmark.properties['guid'] == guid:
return bookmark
for snapshot in self.snapshots:
if snapshot.properties['guid'] == guid:
return snapshot
return None
def find_bookmark(self, snapshot_bookmark, preferred_tag):
"""find bookmark by bookmark name (can be a suffix or a different
ZfsSnapshot or ZfsBookmark) Returns None if it can't find it.
We try to find the bookmark with the preferred tag (which is usually a target path guid, to prevent conflicting bookmarks by multiple sends)
If that fails, we return any bookmark that matches (and ignore the tag)
Args:
:rtype: ZfsBookmark|None
:type snapshot_bookmark: str|ZfsBookmark|ZfsSnapshot|None
:type preferred_tag: str
"""
if snapshot_bookmark is None:
return None
if not isinstance(snapshot_bookmark, ZfsDataset):
tagless_suffix = snapshot_bookmark
else:
tagless_suffix = snapshot_bookmark.tagless_suffix
for snapshot_bookmark in self.bookmarks:
if snapshot_bookmark.tagless_suffix == tagless_suffix and snapshot_bookmark.tag == preferred_tag:
return snapshot_bookmark
for snapshot_bookmark in self.bookmarks:
if snapshot_bookmark.tagless_suffix == tagless_suffix:
return snapshot_bookmark
return None
def find_exact_bookmark(self, bookmark_name):
"""find exact bookmark name, or retruns none
:rtype: ZfsBookmark|None
"""
for snapshot_bookmark in self.bookmarks:
if snapshot_bookmark.name == bookmark_name:
return snapshot_bookmark
return None
def find_snapshot_index(self, snapshot):
"""find exact snapshot index by snapshot (can be a snapshot_name or
ZfsSnapshot)
Args:
:type snapshot: str or ZfsSnapshot
:rtype: int|None
"""
if not isinstance(snapshot, ZfsDataset):
snapshot_name = snapshot
else:
snapshot_name = snapshot.suffix
index = 0
for snapshot in self.snapshots:
if snapshot.suffix == snapshot_name:
return index
index = index + 1
return None
def find_common_snapshot(self, target_dataset, guid_check, bookmark_tag):
"""find latest common snapshot/bookmark between us and target returns None if its
an initial transfer.
On the source it prefers the specified bookmark_name
Args:
:rtype: tuple[ZfsSnapshot|ZfsBookmark, ZfsSnapshot] | tuple[None,None]
:returns: (source_common_snapshot, target_common_snapshot)
:type guid_check: bool
:type target_dataset: ZfsContainer
:type bookmark_tag: str
"""
for target_snapshot in reversed(target_dataset.snapshots):
# Prefer bookmarks to snapshots
source_bookmark = self.find_bookmark(target_snapshot, preferred_tag=bookmark_tag)
if source_bookmark:
if guid_check and source_bookmark.properties['guid'] != target_snapshot.properties['guid']:
source_bookmark.warning("Bookmark has mismatching GUID, ignoring.")
else:
source_bookmark.debug("Common bookmark")
return source_bookmark, target_snapshot
# Source snapshot with same suffix?
source_snapshot = self.find_snapshot(target_snapshot)
if source_snapshot:
if guid_check and not source_snapshot.guid_matches(target_snapshot):
source_snapshot.warning("Snapshot has mismatching GUID, ignoring.")
else:
source_snapshot.debug("Common snapshot")
return source_snapshot, target_snapshot
# Extensive GUID search (slower but works with all names)
source_bookmark_snapshot = self.find_guid_bookmark_snapshot(target_snapshot.properties['guid'])
if source_bookmark_snapshot is not None:
return source_bookmark_snapshot, target_snapshot
return None, None
def is_selected(self, value, source, inherited, exclude_paths, exclude_unchanged):
"""determine if dataset should be selected for backup (called from
ZfsNode)
Args:
:type exclude_paths: list[str]
:type value: str
:type source: str
:type inherited: bool
:type exclude_unchanged: int
:param value: Value of the zfs property ("false"/"true"/"child"/parent/"-")
:param source: Source of the zfs property ("local"/"received", "-")
:param inherited: True of the value/source was inherited from a higher dataset.
Returns: True : Selected
False: Excluded
None: No property found
"""
# sanity checks
if source not in ["local", "received", "-"]:
# probably a program error in zfs-autobackup or new feature in zfs
raise (Exception(
"{} autobackup-property has illegal source: '{}' (possible BUG)".format(self.name, source)))
if value not in ["false", "true", "child", "parent", "-"]:
# user error
raise (Exception(
"{} autobackup-property has illegal value: '{}'".format(self.name, value)))
# non specified, ignore
if value == "-":
return None
# only select childs of this dataset, ignore
if value == "child" and not inherited:
return False
# only select parent, no childs, ignore
if value == "parent" and inherited:
return False
# manually excluded by property
if value == "false":
self.verbose("Excluded")
return False
# from here on the dataset is selected by property, now do additional exclusion checks
# our path starts with one of the excluded paths?
for exclude_path in exclude_paths:
# if self.name.startswith(exclude_path):
if (self.name + "/").startswith(exclude_path + "/"):
# too noisy for verbose
self.debug("Excluded (path in exclude list)")
return False
if not self.is_changed(exclude_unchanged):
self.verbose("Excluded (by --exclude-unchanged)")
return False
self.verbose("Selected")
return True
def destroy(self, fail_exception=False, **kwargs):
self.verbose("Destroying")
return super().destroy(fail_exception=fail_exception)
@property
def origin(self):
"""The origin snapshot of this clone, or None if this dataset is not a clone or the origin cannot be parsed.
:rtype: ZfsSnapshot|None
"""
origin = self.properties.get('origin', '-')
if origin == '-':
return None
if '@' not in origin:
self.warning("Cannot parse clone origin '{}'.".format(origin))
return None
return self.zfs_node.get_snapshot(origin)
+134 -1149
View File
File diff suppressed because it is too large. Load diff
+158 -62
View File
@@ -1,36 +1,45 @@
# python 2 compatibility
from __future__ import print_function
from operator import attrgetter
import re
import shlex
import subprocess
import sys
import time
from .ExecuteNode import ExecuteNode
from .Thinner import Thinner
from .CachedProperty import CachedProperty
from .ZfsContainer import ZfsContainer
from .ZfsPool import ZfsPool
from .ZfsDataset import ZfsDataset
from .ZfsSnapshot import ZfsSnapshot
from .ZfsBookmark import ZfsBookmark
from .ExecuteNode import ExecuteError
from .util import datetime_now
class ZfsNode(ExecuteNode):
"""a node that contains zfs datasets. implements global (systemwide/pool wide) zfs commands"""
def __init__(self, logger, utc=False, snapshot_time_format="", hold_name="", ssh_config=None, ssh_to=None, readonly=False,
description="",
debug_output=False, thinner=None, exclude_snapshot_patterns=[]):
# def __init__(self, logger, utc=False, snapshot_time_format="", hold_name="", ssh_config=None, ssh_to=None,
# readonly=False,
# description="",
# debug_output=False, thinner=None, exclude_snapshot_patterns=None, tag_seperator='~'):
def __init__(self, logger, utc, snapshot_time_format, hold_name, ssh_config, ssh_to,
readonly,
description,
debug_output, thinner, exclude_snapshot_patterns, tag_seperator):
self.utc = utc
self.snapshot_time_format = snapshot_time_format
self.tag_seperator = tag_seperator
self.hold_name = hold_name
self.description = description
self.logger = logger
self._supported_send_options = None
self._supported_recv_options = None
self.exclude_snapshot_patterns = exclude_snapshot_patterns
if ssh_config:
@@ -49,46 +58,56 @@ class ZfsNode(ExecuteNode):
else:
self.verbose("Keep no old snaphots")
self.__thinner = thinner
self._thinner = thinner
# list of ZfsPools
self.__pools = {}
self.__datasets = {}
self._pools = {} # type: dict[str, ZfsPool]
self._datasets = {} # type: dict[str, ZfsContainer | ZfsBookmark | ZfsSnapshot]
self._progress_total_bytes = 0
self._progress_start_time = time.time()
ExecuteNode.__init__(self, ssh_config=ssh_config, ssh_to=ssh_to, readonly=readonly, debug_output=debug_output)
def thin(self, objects, keep_objects):
def thin_list(self, snapshots, keep_snapshots):
"""Returns a list of snapshots to keep and remove, according to current time and thinner settings.
:return: ( keeps, removes )
:type snapshots: list[ZfsSnapshot]
:type keep_snapshots: list[ZfsSnapshot]
:rtype: ( list[ZfsSnapshot], list[ZfsSnapshot] )
"""
# NOTE: if thinning is disabled with --no-thinning, self.__thinner will be none.
if self.__thinner is not None:
if self._thinner is not None:
return self.__thinner.thin(objects, keep_objects, datetime_now(self.utc).timestamp())
return self._thinner.thin(snapshots, keep_snapshots, datetime_now(self.utc).timestamp())
else:
return (keep_objects, [])
return (keep_snapshots, [])
@CachedProperty
@property
def supported_send_options(self):
"""list of supported options, for optimizing sends"""
# not every zfs implementation supports them all
ret = []
for option in ["-L", "-e", "-c"]:
if self.valid_command(["zfs", "send", option, "zfs_autobackup_option_test"]):
ret.append(option)
return ret
if self._supported_send_options is None:
self._supported_send_options = []
for option in ["-L", "-e", "-c"]:
if self.valid_command(["zfs", "send", option, "zfs_autobackup_option_test"]):
self._supported_send_options.append(option)
return self._supported_send_options
@CachedProperty
@property
def supported_recv_options(self):
"""list of supported options"""
# not every zfs implementation supports them all
ret = []
for option in ["-s"]:
if self.valid_command(["zfs", "recv", option, "zfs_autobackup_option_test"]):
ret.append(option)
return ret
if self._supported_recv_options is None:
self._supported_recv_options = []
for option in ["-s"]:
if self.valid_command(["zfs", "recv", option, "zfs_autobackup_option_test"]):
self._supported_recv_options.append(option)
return self._supported_recv_options
def valid_command(self, cmd):
"""test if a specified zfs options are valid exit code. use this to determine support options"""
@@ -108,12 +127,67 @@ class ZfsNode(ExecuteNode):
zpool_name = dataset.name.split("/")[0]
return self.__pools.setdefault(zpool_name, ZfsPool(self, zpool_name))
return self._pools.setdefault(zpool_name, ZfsPool(self, zpool_name))
def get_dataset(self, name, force_exists=None):
"""get a ZfsDataset() object from name. stores objects internally to enable caching"""
"""get a ZfsDataset() object from name. stores objects internally to enable caching
:type name: str
:rtype: ZfsContainer | ZfsBookmark | ZfsSnapshot
"""
return self.__datasets.setdefault(name, ZfsDataset(self, name, force_exists))
if name in self._datasets:
return self._datasets[name]
if '@' in name:
self._datasets[name] = ZfsSnapshot(self, name, force_exists=force_exists)
elif '#' in name:
self._datasets[name] = ZfsBookmark(self, name, force_exists=force_exists)
else:
self._datasets[name] = ZfsContainer(self, name, force_exists=force_exists)
return self._datasets[name]
def get_container(self, name, force_exists=None):
"""get a ZfsContainer from name, asserting the name contains no @ or #.
:type name: str
:rtype: ZfsContainer
"""
dataset = self.get_dataset(name, force_exists=force_exists)
assert isinstance(dataset, ZfsContainer)
return dataset
def get_snapshot(self, name, force_exists=None):
"""get a ZfsSnapshot from name, asserting the name contains @.
:type name: str
:rtype: ZfsSnapshot
"""
dataset = self.get_dataset(name, force_exists=force_exists)
assert isinstance(dataset, ZfsSnapshot)
return dataset
def get_bookmark(self, name, force_exists=None):
"""get a ZfsBookmark from name, asserting the name contains #.
:type name: str
:rtype: ZfsBookmark
"""
dataset = self.get_dataset(name, force_exists=force_exists)
assert isinstance(dataset, ZfsBookmark)
return dataset
def get_datasets(self, names, force_exists=None):
"""get a list of ZfsDataset() object from names. stores objects internally to enable caching
:rtype: list[ZfsContainer | ZfsBookmark | ZfsSnapshot]
"""
ret = []
for name in names:
ret.append(self.get_dataset(name, force_exists))
return ret
# def reset_progress(self):
# """reset progress output counters"""
@@ -149,14 +223,15 @@ class ZfsNode(ExecuteNode):
self._progress_start_time = time.time()
elif progress_fields[1].isnumeric():
bytes_ = int(progress_fields[1])
if self._progress_total_bytes:
elapsed = time.time() - self._progress_start_time
if self._progress_total_bytes and elapsed > 0 and bytes_ > 0:
percentage = min(100, int(bytes_ * 100 / self._progress_total_bytes))
speed = int(bytes_ / (time.time() - self._progress_start_time) / (1024 * 1024))
speed = int(bytes_ / elapsed / (1024 * 1024))
bytes_left = self._progress_total_bytes - bytes_
minutes_left = int((bytes_left / (bytes_ / (time.time() - self._progress_start_time))) / 60)
minutes_left = int(bytes_left / (bytes_ / elapsed) / 60)
self.logger.progress(
"Transfer {}% {}MB/s (total {}MB, {} minutes left)".format(percentage, speed, int(
"Transfering {}% {}MB/s (total {}MB, {} minutes left)".format(percentage, speed, int(
self._progress_total_bytes / (1024 * 1024)), minutes_left))
return
@@ -188,19 +263,25 @@ class ZfsNode(ExecuteNode):
def consistent_snapshot(self, datasets, snapshot_name, min_changed_bytes, pre_snapshot_cmds=[],
post_snapshot_cmds=[], set_snapshot_properties=[]):
"""create a consistent (atomic) snapshot of specified datasets, per pool.
Args:
:type datasets: list[ZfsContainer]
"""
pools = {}
pools = {} # type: dict[str, list[ZfsSnapshot]]
# collect snapshots that we want to make, per pool
# self.debug(datasets)
for dataset in datasets:
if not dataset.is_changed_ours(min_changed_bytes):
dataset.verbose("No changes since {}".format(dataset.our_snapshots[-1].snapshot_name))
dataset.verbose("No changes since {}".format(dataset.our_snapshots[-1].suffix))
continue
# force_exist, since we're making it
snapshot = self.get_dataset(dataset.name + "@" + snapshot_name, force_exists=True)
snapshot = self.get_snapshot(dataset.name + "@" + snapshot_name, force_exists=True)
if self.readonly:
snapshot.simulate_properties()
pool = dataset.split_path()[0]
if pool not in pools:
@@ -208,9 +289,7 @@ class ZfsNode(ExecuteNode):
pools[pool].append(snapshot)
# update cache, but try to prevent an unneeded zfs list
if self.readonly or CachedProperty.is_cached(dataset, 'snapshots'):
dataset.snapshots.append(snapshot) # NOTE: this will trigger zfs list if its not cached
dataset.cache_snapshot_bookmark(snapshot, force=self.readonly)
if not pools:
self.verbose("No changes anywhere: not creating snapshots.")
@@ -227,7 +306,7 @@ class ZfsNode(ExecuteNode):
for snapshot_property in set_snapshot_properties:
cmd += ['-o', snapshot_property]
cmd.extend(map(lambda snapshot_: str(snapshot_), snapshots))
cmd.extend(map(lambda snapshot_: snapshot_.name, snapshots))
self.verbose("Creating snapshots {} in pool {}".format(snapshot_name, pool_name))
self.run(cmd, readonly=False)
@@ -240,22 +319,21 @@ class ZfsNode(ExecuteNode):
except Exception as e:
pass
def selected_datasets(self, property_name, exclude_received, exclude_paths, exclude_unchanged):
def selected_datasets(self, property_name, exclude_paths, exclude_unchanged):
"""determine filesystems that should be backed up by looking at the special autobackup-property, systemwide
returns: (list of selected ZfsDataset sorted by createtxg, list of excluded ZfsDataset)
returns: ( list of selected ZfsContainer, list of excluded ZfsContainer)
"""
self.debug("Getting selected datasets")
# get all source filesystems that have the backup property
lines = self.run(tab_split=True, readonly=True, cmd=[
"zfs", "get", "-t", "volume,filesystem", "-Hp",
property_name + ",createtxg"
"zfs", "get", "-t", "volume,filesystem", "-o", "name,value,source", "-H",
property_name
])
# The returnlist of selected ZfsDataset's:
# The returnlist of selected ZfsContainer's:
selected_filesystems = []
excluded_filesystems = []
@@ -263,14 +341,8 @@ class ZfsNode(ExecuteNode):
sources = {}
for line in lines:
(name, prop_name, value, raw_source) = line
if prop_name == "createtxg":
# createtxg for the last dataset
if selected_filesystems and selected_filesystems[-1].name == name:
selected_filesystems[-1].createtxg = int(value)
if prop_name != property_name:
continue
dataset = self.get_dataset(name, force_exists=True)
(name, value, raw_source) = line
dataset = self.get_container(name, force_exists=True)
# "resolve" inherited sources
sources[name] = raw_source
@@ -283,14 +355,38 @@ class ZfsNode(ExecuteNode):
source = raw_source
# determine it
selected=dataset.is_selected(value=value, source=source, inherited=inherited, exclude_received=exclude_received,
exclude_paths=exclude_paths, exclude_unchanged=exclude_unchanged)
selected = dataset.is_selected(value=value, source=source, inherited=inherited,
exclude_paths=exclude_paths, exclude_unchanged=exclude_unchanged)
if selected==True:
if selected == True:
selected_filesystems.append(dataset)
elif selected==False:
elif selected == False:
excluded_filesystems.append(dataset)
#returns None when no property is set.
# returns None when no property is set.
return (sorted(selected_filesystems, key=attrgetter("createtxg")),
excluded_filesystems)
return (selected_filesystems, excluded_filesystems)
def get_resume_snapshot(self, target_resume_token):
"""returns snapshot that will be resumed by this resume token (run this
on source node with target-token)
Args:
:type target_resume_token: str
:rtype: ZfsSnapshot|None
"""
# use zfs send -n option to determine this
# NOTE: on smartos stderr, on linux stdout
(stdout, stderr) = self.run(["zfs", "send", "-t", target_resume_token, "-n", "-v"], valid_exitcodes=[0, 255],
readonly=True, return_stderr=True)
if stdout:
lines = stdout
else:
lines = stderr
for line in lines:
matches = re.findall("toname = (.*@.*)", line)
if matches:
snapshot = self.get_snapshot(matches[0])
snapshot.debug("resume token belongs to this snapshot")
return snapshot
return None
+103
View File
@@ -0,0 +1,103 @@
import sys
from .ZfsDataset import ZfsDataset
from datetime import datetime
import time
class ZfsPointInTime(ZfsDataset):
"""contains stuff that is common to snapshots and bookmarks"""
def __init__(self, zfs_node, name, force_exists=None):
if type(self) is ZfsPointInTime:
raise TypeError(
"should not be instantiated directly.")
super().__init__(zfs_node, name, force_exists)
#implemented in subclass
@property
def suffix(self):
return ""
#implemented in subclass
@property
def prefix(self):
return ""
@property
def parent(self):
"""get parent dataset
:rtype: ZfsContainer
"""
return self.zfs_node.get_container(self.prefix)
@property
def timestamp(self):
"""get timestamp from snapshot/bookmark name. Only works for our own snapshots
with the correct format. Snapshots that are not ours always return None
Note that the tag-part in the name is ignored, so snapsnots are ours regardless of their tag.
:rtype: int|None
"""
#note: cant use self.tagless_suffix here, because it depends on us
if self.zfs_node.tag_seperator in self.suffix:
tagless_suffix = self.suffix.split(self.zfs_node.tag_seperator)[0]
else:
tagless_suffix = self.suffix
try:
dt = datetime.strptime(tagless_suffix, self.zfs_node.snapshot_time_format)
except ValueError:
return None
if sys.version_info[0] >= 3:
from datetime import timezone
if self.zfs_node.utc:
dt = dt.replace(tzinfo=timezone.utc)
seconds = dt.timestamp()
else:
# python2 has no good functions to deal with UTC. Yet the unix timestamp
# must be in UTC to allow comparison against `time.time()` in on other parts
# of this project (e.g. Thinner.py). If we are handling UTC timestamps,
# we must adjust for that here.
if self.zfs_node.utc:
seconds = (dt - datetime(1970, 1, 1)).total_seconds()
else:
seconds = time.mktime(dt.timetuple())
return seconds
@property
def is_ours(self):
"""return true if this snapshot name belong to the current backup_name and snapshot formatting"""
return self.timestamp is not None
@property
def tagless_suffix(self):
"""snapshot or bookmark part of the name, but without the tag. (if its our snapshot we remove the tag)"""
if self.is_ours and self.zfs_node.tag_seperator in self.suffix:
return self.suffix.split(self.zfs_node.tag_seperator)[0]
else:
return self.suffix
@property
def tag(self):
"""the tag-part of a snapshot or bookmark, if any. can return None"""
if self.is_ours and self.zfs_node.tag_seperator in self.suffix:
return self.suffix.split(self.zfs_node.tag_seperator)[1]
else:
return None
+20 -11
View File
@@ -1,4 +1,4 @@
from .CachedProperty import CachedProperty
from logging import warning
class ZfsPool():
@@ -10,6 +10,10 @@ class ZfsPool():
self.zfs_node = zfs_node
self.name = name
self._properties = None
def invalidate_cache(self):
self._properties = None
def __repr__(self):
return "{}: {}".format(self.zfs_node, self.name)
@@ -26,28 +30,33 @@ class ZfsPool():
def verbose(self, txt):
self.zfs_node.verbose("zpool {}: {}".format(self.name, txt))
def warning(self, txt):
self.zfs_node.warning("zpool {}: {}".format(self.name, txt))
def error(self, txt):
self.zfs_node.error("zpool {}: {}".format(self.name, txt))
def debug(self, txt):
self.zfs_node.debug("zpool {}: {}".format(self.name, txt))
@CachedProperty
@property
def properties(self):
"""all zpool properties"""
self.debug("Getting zpool properties")
if self._properties is None:
cmd = [
"zpool", "get", "-H", "-p", "all", self.name
]
self.debug("Getting zpool properties")
ret = {}
cmd = [
"zpool", "get", "-H", "-p", "all", self.name
]
for pair in self.zfs_node.run(tab_split=True, cmd=cmd, readonly=True, valid_exitcodes=[0]):
ret[pair[1]] = pair[2]
self._properties = {}
return ret
for pair in self.zfs_node.run(tab_split=True, cmd=cmd, readonly=True, valid_exitcodes=[0]):
self._properties[pair[1]] = pair[2]
return self._properties
@property
def features(self):
@@ -60,4 +69,4 @@ class ZfsPool():
if value == 'enabled' or value == 'active':
ret.append(feature)
return ret
return ret
+330
View File
@@ -0,0 +1,330 @@
from typing import cast
from .ZfsPointInTime import ZfsPointInTime
class ZfsSnapshot(ZfsPointInTime):
"""A ZFS snapshot"""
@property
def prefix(self):
(filesystem, snapshot) = self.name.split("@")
return filesystem
@property
def suffix(self):
(filesystem, snapshot_name) = self.name.split("@")
return snapshot_name
@property
def typed_suffix(self):
return "@" + self.suffix
@property
def is_snapshot_excluded(self):
"""true if this is a snapshot and matches the exclude pattern"""
suffix = self.suffix
for pattern in self.zfs_node.exclude_snapshot_patterns:
if pattern.search(suffix) is not None:
self.debug("Excluded (path matches snapshot exclude pattern)")
return True
return False
def find_snapshot_by_suffix(self, snapshot_bookmarks):
"""return ZfsSnapshot|ZfsBookmark from the list of snapshot_bookmarks, if it matches our snapshot_name. Otherwise None
Args:
:type snapshot_bookmarks: list[ZfsSnapshot|ZfsBookmark]
:rtype: ZfsSnapshot|ZfsBookmark|None
"""
for snapshot_bookmark in snapshot_bookmarks:
if snapshot_bookmark.tagless_suffix == self.tagless_suffix:
return snapshot_bookmark
return None
def send_pipe(self, features, prev_snapshot, resume_token, show_progress, raw, send_properties, write_embedded,
send_pipes, zfs_compressed):
"""returns a pipe with zfs send output for this snapshot
resume_token: resume sending from this token. (in that case we don't
need to know snapshot names)
If prev_snapshot is on a different parent dataset (clone-of-origin case),
its full name is passed to zfs send -i; otherwise the short @suffix form is used.
Args:
:param send_pipes: output cmd array that will be added to actual zfs send command. (e.g. mbuffer or compression program)
:type send_pipes: list[str]
:type features: list[str]
:type prev_snapshot: ZfsSnapshot|ZfsBookmark|None
:type resume_token: str|None
:type show_progress: bool
:type raw: bool
:type send_properties: bool
:type write_embedded: bool
:type zfs_compressed: bool
"""
# build source command
cmd = []
cmd.extend(["zfs", "send", ])
# all kind of performance options:
if 'large_blocks' in features and "-L" in self.zfs_node.supported_send_options:
# large block support (only if recordsize>128k which is seldomly used)
cmd.append("-L") # --large-block
if write_embedded and 'embedded_data' in features and "-e" in self.zfs_node.supported_send_options:
cmd.append("-e") # --embed; WRITE_EMBEDDED, more compact stream
if zfs_compressed and "-c" in self.zfs_node.supported_send_options:
cmd.append("-c") # --compressed; use compressed WRITE records
# raw? (send over encrypted data in its original encrypted form without decrypting)
if raw:
cmd.append("--raw")
# progress output
if show_progress:
cmd.append("-v") # --verbose
cmd.append("-P") # --parsable
# resume a previous send? (don't need more parameters in that case)
if resume_token:
cmd.extend(["-t", resume_token])
else:
# send properties
if send_properties:
cmd.append("-p") # --props
# incremental?
if prev_snapshot:
if prev_snapshot.prefix != self.prefix:
# cross-dataset (clone replicated from its origin snapshot): full path required
cmd.extend(["-i", prev_snapshot.name])
else:
cmd.extend(["-i", prev_snapshot.typed_suffix])
cmd.append(self.name)
cmd.extend(send_pipes)
output_pipe = self.zfs_node.run(cmd, pipe=True, readonly=True)
return output_pipe
def recv_pipe(self, pipe, features, recv_pipes, filter_properties=None, set_properties=None, ignore_exit_code=False,
force=False):
"""starts a zfs recv for this snapshot and uses pipe as input
note: you can it both on a snapshot or filesystem object. The
resulting zfs command is the same, only our object cache is invalidated
differently.
Args:
:param recv_pipes: input cmd array that will be prepended to actual zfs recv command. (e.g. mbuffer or decompression program)
:type pipe: subprocess.pOpen
:type features: list[str]
:type filter_properties: list[str]
:type set_properties: list[str]
:type ignore_exit_code: bool
:type force: bool
"""
if set_properties is None:
set_properties = []
if filter_properties is None:
filter_properties = []
# build target command
cmd = []
cmd.extend(recv_pipes)
cmd.extend(["zfs", "recv"])
# don't let zfs recv mount everything thats received (even with canmount=noauto!)
cmd.append("-u")
for property_ in filter_properties:
cmd.extend(["-x", property_])
for property_ in set_properties:
cmd.extend(["-o", property_])
# verbose output
cmd.append("-v")
if force:
cmd.append("-F")
if 'extensible_dataset' in features and "-s" in self.zfs_node.supported_recv_options:
# support resuming
self.debug("Enabled resume support")
cmd.append("-s")
cmd.append(self.prefix)
if ignore_exit_code:
valid_exitcodes = []
else:
valid_exitcodes = [0]
# self.zfs_node.reset_progress()
self.zfs_node.run(cmd, inp=pipe, valid_exitcodes=valid_exitcodes)
# invalidate cache
self.invalidate_cache()
# in test mode we pretend everything was ok and it exists
if self.zfs_node.readonly:
self.force_exists = True
self.simulate_properties()
# check if transfer was really ok (exit codes have been wrong before due to bugs in zfs-utils and some
# errors should be ignored, thats where the ignore_exitcodes is for.)
if not self.exists:
self.error("error during transfer")
raise (Exception("Target doesn't exist after transfer, something went wrong."))
# at this point we're sure the actual dataset exists
self.parent.force_exists = True
def transfer_snapshot(self, target_snapshot, features, prev_snapshot, show_progress,
filter_properties, set_properties, ignore_recv_exit_code, resume_token,
raw, send_properties, write_embedded, send_pipes, recv_pipes, zfs_compressed, force):
"""transfer this snapshot to target_snapshot. specify prev_snapshot for
incremental transfer
connects a send_pipe() to recv_pipe()
Args:
:type send_pipes: list[str]
:type recv_pipes: list[str]
:type target_snapshot: ZfsSnapshot
:type features: list[str]
:type prev_snapshot: ZfsSnapshot|ZfsBookmark|None
:type show_progress: bool
:type filter_properties: list[str]
:type set_properties: list[str]
:type ignore_recv_exit_code: bool
:type resume_token: str|None
:type raw: bool
:type send_properties: bool
:type write_embedded: bool
:type zfs_compressed: bool
:type force: bool
"""
if set_properties is None:
set_properties = []
if filter_properties is None:
filter_properties = []
self.debug("Transfer snapshot to {}".format(target_snapshot.prefix))
if resume_token:
self.verbose("resuming")
# initial or increment
if not prev_snapshot:
self.verbose("-> {} (new)".format(target_snapshot.prefix))
else:
# incremental
self.verbose("-> {}".format(target_snapshot.prefix))
if show_progress:
self.zfs_node.logger.progress("Transferring...")
# do it
pipe = self.send_pipe(features=features, show_progress=show_progress, prev_snapshot=prev_snapshot,
resume_token=resume_token, raw=raw, send_properties=send_properties,
write_embedded=write_embedded, send_pipes=send_pipes, zfs_compressed=zfs_compressed)
target_snapshot.recv_pipe(pipe, features=features, filter_properties=filter_properties,
set_properties=set_properties, ignore_exit_code=ignore_recv_exit_code,
recv_pipes=recv_pipes, force=force)
# try to automount it, if its the initial transfer
if not prev_snapshot:
# in test mode it doesnt actually exist, so dont try to mount it/read properties, and simulate properties.
if not target_snapshot.zfs_node.readonly:
target_snapshot.parent.automount()
def bookmark(self, tag):
"""Bookmark this snapshot, and return the bookmark. If the bookmark already exist, it returns it."""
# NOTE: we use the tag to add the target_path GUID, so that we can have multiple bookmarks for the same snapshot, but for different targets.
# This is to make sure you can send a backup to two locations, without them interfering with eachothers bookmarks.
bookmark_name = self.prefix + "#" + self.tagless_suffix + self.zfs_node.tag_seperator + tag
# does this exact bookmark (including tag) already exists?
existing_bookmark = self.parent.find_exact_bookmark(bookmark_name)
if existing_bookmark is not None:
return existing_bookmark
self.debug("Bookmarking {}".format(bookmark_name))
cmd = [
"zfs", "bookmark", self.name, bookmark_name
]
self.zfs_node.run(cmd=cmd)
bookmark = self.zfs_node.get_bookmark(bookmark_name, force_exists=True)
self.parent.cache_snapshot_bookmark(bookmark)
return bookmark
@property
def holds(self):
output = cast("list[list[str]]", cast(object, self.zfs_node.run(
["zfs", "holds", "-H", self.name], valid_exitcodes=[0], tab_split=True, readonly=True)))
return map(lambda fields: fields[1], output)
def is_hold(self):
"""did we hold this snapshot?"""
return self.zfs_node.hold_name in self.holds
def hold(self):
"""hold snapshot"""
self.debug("holding")
self.zfs_node.run(["zfs", "hold", self.zfs_node.hold_name, self.name], valid_exitcodes=[0, 1])
def release(self):
"""release snapshot"""
if self.zfs_node.readonly or self.is_hold():
self.debug("releasing")
self.zfs_node.run(["zfs", "release", self.zfs_node.hold_name, self.name], valid_exitcodes=[0, 1])
def rollback(self):
self.debug("Rolling back")
self.zfs_node.run(["zfs", "rollback", self.name])
def destroy(self, fail_exception=False, deferred=False):
self.verbose("Destroying")
self.release()
return super().destroy(fail_exception=fail_exception, deferred=deferred)
def clone(self, name):
"""clones this snapshot and returns ZfsContainer of the clone
:rtype: ZfsContainer
"""
self.debug("Cloning to {}".format(name))
cmd = [
"zfs", "clone", self.name, name
]
self.zfs_node.run(cmd=cmd, valid_exitcodes=[0])
return self.zfs_node.get_container(name, force_exists=True)
+20 -17
View File
@@ -7,69 +7,72 @@
COMPRESS_CMDS = {
'gzip': {
'cmd': 'gzip',
'args': [ '-3' ],
'args': ['-3'],
'dcmd': 'zcat',
'dargs': [],
},
'pigz-fast': {
'cmd': 'pigz',
'args': [ '-3' ],
'args': ['-3'],
'dcmd': 'pigz',
'dargs': [ '-dc' ],
'dargs': ['-dc'],
},
'pigz-slow': {
'cmd': 'pigz',
'args': [ '-9' ],
'args': ['-9'],
'dcmd': 'pigz',
'dargs': [ '-dc' ],
'dargs': ['-dc'],
},
'zstd-fast': {
'cmd': 'zstdmt',
'args': [ '-3' ],
'args': ['-3'],
'dcmd': 'zstdmt',
'dargs': [ '-dc' ],
'dargs': ['-dc'],
},
'zstd-slow': {
'cmd': 'zstdmt',
'args': [ '-19' ],
'args': ['-19'],
'dcmd': 'zstdmt',
'dargs': [ '-dc' ],
'dargs': ['-dc'],
},
'zstd-adapt': {
'cmd': 'zstdmt',
'args': [ '--adapt' ],
'args': ['--adapt'],
'dcmd': 'zstdmt',
'dargs': [ '-dc' ],
'dargs': ['-dc'],
},
'xz': {
'cmd': 'xz',
'args': [],
'dcmd': 'xz',
'dargs': [ '-d' ],
'dargs': ['-d'],
},
'lzo': {
'cmd': 'lzop',
'args': [],
'dcmd': 'lzop',
'dargs': [ '-dfc' ],
'dargs': ['-dfc'],
},
'lz4': {
'cmd': 'lz4',
'args': [],
'dcmd': 'lz4',
'dargs': [ '-dc' ],
'dargs': ['-dc'],
},
}
def compress_cmd(compressor):
ret=[ COMPRESS_CMDS[compressor]['cmd'] ]
ret.extend( COMPRESS_CMDS[compressor]['args'])
ret = [COMPRESS_CMDS[compressor]['cmd']]
ret.extend(COMPRESS_CMDS[compressor]['args'])
return ret
def decompress_cmd(compressor):
ret= [ COMPRESS_CMDS[compressor]['dcmd'] ]
ret = [COMPRESS_CMDS[compressor]['dcmd']]
ret.extend(COMPRESS_CMDS[compressor]['dargs'])
return ret
def choices():
return COMPRESS_CMDS.keys()
+297
View File
@@ -0,0 +1,297 @@
"""Dataset-level replication: sync a selection of source datasets to a target node,
and thin or destroy target datasets that are missing on the source."""
from collections import deque
from .ThinnerRule import ThinnerRule
from .ZfsContainer import ZfsContainer
from .sync_snapshots import sync_snapshots
from .util import datetime_now
def _missing_datasets(target_dataset, used_target_datasets):
"""list the datasets under target_dataset that are not used as a backup target (anymore)
:type target_dataset: ZfsContainer
:type used_target_datasets: list[ZfsContainer]
:rtype: list[ZfsContainer]
"""
return [dataset for dataset in target_dataset.recursive_datasets if dataset not in used_target_datasets]
def thin_missing_targets(logger, target_dataset, used_target_datasets):
"""thin target datasets that are missing on the source.
:type logger: LogConsole
:type target_dataset: ZfsContainer
:type used_target_datasets: list[ZfsContainer]
"""
logger.debug("Thinning obsolete datasets")
missing_datasets = _missing_datasets(target_dataset, used_target_datasets)
for count, dataset in enumerate(missing_datasets, start=1):
logger.debug("analyse missing {}".format(dataset))
logger.progress("Analysing missing..", count, len(missing_datasets), 0)
try:
dataset.debug("Missing on source, thinning")
dataset.thin()
except Exception as e:
dataset.error("Error during thinning of missing datasets ({})".format(str(e)))
def _destroy_missing_dataset(dataset, destroy_missing, utc):
"""destroy a single missing dataset if it meets all the requirements, otherwise tell the user why not.
:type dataset: ZfsContainer
:type destroy_missing: str
:type utc: bool
"""
# cant do anything without our own snapshots
if not dataset.our_snapshots:
if dataset.datasets:
# its not a leaf, just ignore
dataset.debug("Destroy missing: ignoring")
else:
dataset.verbose("Destroy missing: has no snapshots made by us (please destroy manually).")
return
# past the deadline?
deadline_ttl = ThinnerRule("0s" + destroy_missing).ttl
now = datetime_now(utc).timestamp()
if dataset.our_snapshots[-1].timestamp + deadline_ttl > now:
dataset.verbose("Destroy missing: Waiting for deadline.")
return
dataset.debug("Destroy missing: Removing our snapshots.")
# remove all our snaphots, except last, to safe space in case we fail later on
for snapshot in dataset.our_snapshots[:-1]:
snapshot.destroy(fail_exception=True)
# does it have other snapshots?
if any(not snapshot.is_ours for snapshot in dataset.snapshots):
dataset.verbose("Destroy missing: Still in use by other snapshots")
return
if dataset.datasets:
dataset.verbose("Destroy missing: Still has children here.")
return
dataset.verbose("Destroy missing.")
dataset.our_snapshots[-1].destroy(fail_exception=True)
dataset.destroy(fail_exception=True)
def destroy_missing_targets(logger, target_dataset, used_target_datasets, destroy_missing, utc):
"""destroy target datasets that are missing on the source and that meet the requirements
:type logger: LogConsole
:type target_dataset: ZfsContainer
:type used_target_datasets: list[ZfsContainer]
:type destroy_missing: str
:type utc: bool
"""
logger.debug("Destroying obsolete datasets")
missing_datasets = _missing_datasets(target_dataset, used_target_datasets)
for count, dataset in enumerate(missing_datasets, start=1):
logger.progress("Analysing destroy missing...", count, len(missing_datasets), 0)
try:
_destroy_missing_dataset(dataset, destroy_missing, utc)
except Exception as e:
dataset.error("Error during --destroy-missing: {}".format(str(e)))
def _clone_dependencies(source_datasets):
"""yield (clone_dataset, origin_snapshot, origin_dataset) for every selected dataset that is a
clone of another selected dataset.
:type source_datasets: list[ZfsContainer]
"""
for dataset in source_datasets:
origin_snapshot = dataset.origin
if origin_snapshot is None:
continue
origin_dataset = origin_snapshot.parent
# only a dependency if the origin dataset is itself in the selection
if origin_dataset in source_datasets and origin_dataset is not dataset:
yield dataset, origin_snapshot, origin_dataset
def _topological_sort_for_clones(logger, source_datasets):
"""Reorder source_datasets so an origin is processed before any selected clone of it.
Datasets whose origin is not in the selection (or who are not clones) are independent
and keep their relative order. On a detected cycle (should not happen with valid ZFS)
the original list is returned with a warning.
:type logger: LogConsole
:type source_datasets: list[ZfsContainer]
:rtype: list[ZfsContainer]
"""
# build edge map: for each dataset that is a clone of another selected dataset,
# record that dependency so the origin is emitted first.
indegree = {dataset: 0 for dataset in source_datasets}
children = {dataset: [] for dataset in source_datasets}
for clone_dataset, origin_snapshot, origin_dataset in _clone_dependencies(source_datasets):
indegree[clone_dataset] += 1
children[origin_dataset].append(clone_dataset)
# Kahn's algorithm: start with datasets that have no in-selection origin
queue = deque(dataset for dataset in source_datasets if indegree[dataset] == 0)
result = []
while queue:
dataset = queue.popleft()
result.append(dataset)
for child in children[dataset]:
indegree[child] -= 1
if indegree[child] == 0:
queue.append(child)
# a cycle means ZFS state is inconsistent; fall back to original order
if len(result) != len(source_datasets):
logger.warning("Clone topological sort: cycle detected, keeping original dataset order.")
return source_datasets
return result
def _required_origin_snapshots(source_datasets):
"""Build a map origin_dataset -> set of origin snapshots that downstream clones (also in the
selection) need pinned on the target. Used to force-include those specific snapshots even
without --other-snapshots.
:type source_datasets: list[ZfsContainer]
:rtype: dict[ZfsContainer, set[ZfsSnapshot]]
"""
required = {}
for clone_dataset, origin_snapshot, origin_dataset in _clone_dependencies(source_datasets):
required.setdefault(origin_dataset, set()).add(origin_snapshot)
return required
def sync_datasets(logger, source_node, source_datasets, target_node, bookmark_tag,
send_pipes, recv_pipes, target_path, strip_path,
no_clone, no_send, no_bookmarks, no_thinning,
filter_properties, set_properties,
ignore_transfer_errors, holds, rollback, other_snapshots,
destroy_incompatible, decrypt, encrypt, zfs_compressed, force,
guid_check, property_format, debug,
target_dataset_base, destroy_missing, utc):
"""Sync datasets, or thin-only on both sides.
:type logger: LogConsole
:type source_node: ZfsNode
:type source_datasets: list[ZfsContainer]
:type target_node: ZfsNode
:type bookmark_tag: str
:type send_pipes: list[str]
:type recv_pipes: list[str]
:type target_path: str
:type strip_path: int
:type no_clone: bool
:type no_send: bool
:type no_bookmarks: bool
:type no_thinning: bool
:type filter_properties: list[str]
:type set_properties: list[str]
:type ignore_transfer_errors: bool
:type holds: bool
:type rollback: bool
:type other_snapshots: bool
:type destroy_incompatible: bool
:type decrypt: bool
:type encrypt: bool
:type zfs_compressed: bool
:type force: bool
:type guid_check: bool
:type property_format: str
:type debug: bool
:type target_dataset_base: ZfsContainer
:type destroy_missing: str|None
:type utc: bool
:rtype: int
"""
if not no_clone:
source_datasets = _topological_sort_for_clones(logger, source_datasets)
required_origin_snapshots = _required_origin_snapshots(source_datasets)
else:
required_origin_snapshots = {}
fail_count = 0
target_datasets = []
bookmarks_unsupported_pools = set() # source pools we've already warned about
for count, source_dataset in enumerate(source_datasets, start=1):
logger.progress("Analysing dataset...", count, len(source_datasets), fail_count)
try:
# determine corresponding target_dataset
target_name = source_dataset.map_to_target_path(target_path, strip_path)
target_dataset = target_node.get_container(target_name)
target_datasets.append(target_dataset)
# ensure parents exists
target_parent = target_dataset.parent
if not no_send \
and target_parent is not None \
and target_parent not in target_datasets \
and not target_parent.exists:
target_dataset.debug("Creating unmountable parents")
target_parent.create_filesystem(parents=True)
# determine common zpool features
source_pool = source_node.get_pool(source_dataset)
source_features = source_pool.features
target_features = target_node.get_pool(target_dataset).features
common_features = [feature for feature in source_features if feature in target_features]
use_bookmarks = not no_bookmarks
if use_bookmarks and 'bookmarks' not in common_features:
if source_pool.name not in bookmarks_unsupported_pools:
source_pool.warning("Disabling bookmarks, not supported on both pools.")
bookmarks_unsupported_pools.add(source_pool.name)
use_bookmarks = False
# sync the snapshots of this dataset
sync_snapshots(source_dataset, target_dataset, show_progress=True,
features=common_features, filter_properties=filter_properties,
set_properties=set_properties,
ignore_recv_exit_code=ignore_transfer_errors,
holds=holds, rollback=rollback,
also_other_snapshots=other_snapshots,
no_send=no_send,
destroy_incompatible=destroy_incompatible,
send_pipes=send_pipes, recv_pipes=recv_pipes,
decrypt=decrypt, encrypt=encrypt,
zfs_compressed=zfs_compressed, force=force,
guid_check=guid_check, use_bookmarks=use_bookmarks,
bookmark_tag=bookmark_tag,
property_format=property_format,
no_clone=no_clone,
target_path=target_path, strip_path=strip_path,
required_snapshots=required_origin_snapshots.get(source_dataset))
except Exception as e:
fail_count = fail_count + 1
source_dataset.error("FAILED: " + str(e))
if debug:
logger.verbose("Debug mode, aborting on first error")
raise
if not no_thinning:
thin_missing_targets(logger, target_dataset=target_dataset_base, used_target_datasets=target_datasets)
if destroy_missing is not None:
destroy_missing_targets(logger, target_dataset=target_dataset_base, used_target_datasets=target_datasets,
destroy_missing=destroy_missing, utc=utc)
return fail_count
+483
View File
@@ -0,0 +1,483 @@
"""Snapshot-level replication: sync the snapshots of one dataset to a target dataset,
thinning out obsolete snapshots on both sides along the way."""
from .ZfsBookmark import ZfsBookmark
from .ZfsContainer import ZfsContainer
from .ZfsPointInTime import ZfsPointInTime
from .ZfsSnapshot import ZfsSnapshot
def _resolve_clone_origin(source_dataset, target_node, target_path, strip_path, guid_check):
"""Return the source-side origin snapshot to use as zfs send -i base for a clone,
or None if the clone relationship cannot be preserved on the target.
:type source_dataset: ZfsContainer
:type target_node: ZfsNode
:type target_path: str
:type strip_path: int
:type guid_check: bool
:rtype: ZfsSnapshot|None
"""
source_origin_snap = source_dataset.origin
if source_origin_snap is None:
return None
# Reverse-clone topology (after 'zfs promote' of a child): origin lives on a namespace descendant
# of this dataset. zfs recv cannot land a clone-creating stream on top of the placeholder that has
# to exist for the descendant, so replication can't preserve the relationship.
origin_parent_path = source_origin_snap.name.split('@', 1)[0]
if origin_parent_path == source_dataset.name or origin_parent_path.startswith(source_dataset.name + "/"):
source_dataset.warning(
"Cannot replicate as clone: origin '{}' lives on a namespace descendant of this dataset. "
"Falling back to full send.".format(source_origin_snap.name))
return None
try:
target_origin_parent = source_origin_snap.parent.map_to_target_path(target_path, strip_path)
except Exception as e:
source_dataset.warning(
"Cannot replicate as clone: cannot map origin '{}' to target ({}). "
"Falling back to full send.".format(source_origin_snap.name, str(e)))
return None
target_origin_snap = target_node.get_snapshot(target_origin_parent + '@' + source_origin_snap.suffix)
if not target_origin_snap.exists:
source_dataset.warning(
"Cannot replicate as clone: origin '{}' not available on target. "
"Falling back to full send.".format(source_origin_snap.name))
return None
if guid_check and not source_origin_snap.guid_matches(target_origin_snap):
source_dataset.warning(
"Cannot replicate as clone: origin guid mismatch between source and target. "
"Falling back to full send.")
return None
return source_origin_snap
def _pre_clean(source_dataset, target_dataset, source_common_snapshot, source_obsoletes, target_obsoletes,
target_transfers):
"""cleanup old stuff on source and target before starting snapshot syncing
:type source_dataset: ZfsContainer
:type target_dataset: ZfsContainer
:type source_common_snapshot: ZfsSnapshot|ZfsBookmark|None
:type source_obsoletes: list[ZfsSnapshot]
:type target_obsoletes: list[ZfsSnapshot]
:type target_transfers: list[ZfsSnapshot]
"""
# on source: delete all obsoletes that are not in target_transfers (except common snapshot, if its not a bookmark)
for source_snapshot in source_dataset.snapshots:
if (source_snapshot in source_obsoletes
and source_common_snapshot != source_snapshot
and source_snapshot.find_snapshot_by_suffix(target_transfers) is None):
source_snapshot.destroy()
# on target: destroy everything thats obsolete, except the common snapshot
if not target_dataset.exists:
return
for target_snapshot in target_dataset.snapshots:
if (target_snapshot in target_obsoletes
and (not source_common_snapshot
or target_snapshot.tagless_suffix != source_common_snapshot.tagless_suffix)
and target_snapshot.exists):
target_snapshot.destroy()
def _handle_incompatible_target(target_dataset, incompatible_target_snapshots, destroy_incompatible, force,
source_common_snapshot, is_clone_origin, is_resume):
"""destroy incompatible snapshots on target before sync, or inform user what to do
:type target_dataset: ZfsContainer
:type incompatible_target_snapshots: list[ZfsSnapshot]
:type destroy_incompatible: bool
:type force: bool
:type source_common_snapshot: ZfsPointInTime|None
:type is_clone_origin: bool
:type is_resume: bool
"""
# a clone origin is not a real common snapshot of this dataset: it lives on a different
# target dataset, so it doesnt make existing content of this target dataset compatible.
if is_clone_origin:
source_common_snapshot = None
if not target_dataset.exists:
# no target yet, so everything ok
return
if not incompatible_target_snapshots and source_common_snapshot is not None:
# nice existing target with compatible common snapshot.
return
if not incompatible_target_snapshots and is_resume and len(target_dataset.snapshots) == 0:
# its a dataset from an incremental send we probably can resume, so its fine
return
# from this point on things get progressively worse..
if source_common_snapshot:
# we just need to delete some incompatibles:
for snapshot in incompatible_target_snapshots:
snapshot.warning("Incompatible snapshot")
if not destroy_incompatible:
raise Exception("Use --destroy-incompatible to get rid of these.")
else:
# no common snapshot, so the whole dataset is incompatible
if incompatible_target_snapshots:
if not force or not destroy_incompatible:
target_dataset.error("Incompatible dataset!")
raise Exception("Use --destroy-incompatible -F to overwrite the target dataset and start over.")
elif not force:
target_dataset.error("Incompatible dataset!")
raise Exception("Use -F to overwrite the target dataset and start over.")
target_dataset.warning("Overwriting incompatible dataset.")
# remove incompatibles and rollback
for snapshot in incompatible_target_snapshots:
snapshot.destroy(fail_exception=True)
target_dataset.rollback()
def _validate_resume_token(source_dataset, target_dataset, start_snapshot):
"""validate and return the resume token, or abort (destroy) the resume state if its no longer valid
:type source_dataset: ZfsContainer
:type target_dataset: ZfsContainer
:type start_snapshot: ZfsSnapshot|None
:rtype: str|None
"""
if not target_dataset.exists or 'receive_resume_token' not in target_dataset.properties:
return None
if start_snapshot is None:
target_dataset.verbose("Aborting resume, its obsolete.")
target_dataset.abort_resume()
return None
resume_token = target_dataset.properties['receive_resume_token']
# not valid anymore?
resume_snapshot = source_dataset.zfs_node.get_resume_snapshot(resume_token)
if not resume_snapshot or start_snapshot.suffix != resume_snapshot.suffix:
target_dataset.verbose("Aborting resume, its no longer valid.")
target_dataset.abort_resume()
return None
return resume_token
def _plan_sync(source_dataset, target_dataset, also_other_snapshots, guid_check, raw, bookmark_tag,
no_clone, target_path, strip_path, required_snapshots=None):
"""Determine at what snapshot to start syncing to target_dataset and what to sync and what to keep.
:type source_dataset: ZfsContainer
:type target_dataset: ZfsContainer
:type also_other_snapshots: bool
:type guid_check: bool
:type raw: bool
:type bookmark_tag: str
:type no_clone: bool
:type target_path: str
:type strip_path: int
:type required_snapshots: set[ZfsSnapshot]|None
:rtype: ( ZfsSnapshot|ZfsBookmark|None, bool, list[ZfsSnapshot], list[ZfsSnapshot], list[ZfsSnapshot], list[ZfsSnapshot] )
Returns:
tuple: A tuple containing:
- ZfsSnapshot|ZfsBookmark|None: The source common snapshot. (the clone origin snapshot when is_clone_origin is True)
- bool: is_clone_origin: the common snapshot is actually the clone origin, which lives on a different dataset.
- list[ZfsSnapshot]: Our obsolete source snapshots, after transfer is done. (will be thinned asap)
- list[ZfsSnapshot]: Our obsolete target snapshots, after transfer is done. (will be thinned asap)
- list[ZfsSnapshot]: Transfer target snapshots. These need to be transferred.
- list[ZfsSnapshot]: Incompatible target snapshots. Target snapshots that are in the way, after the common snapshot. (need to be destroyed to continue)
"""
### 1: determine common and start snapshot if target already exists:
if target_dataset.exists:
target_dataset.debug("Determining start snapshot")
(source_common_snapshot, target_common_snapshot) = source_dataset.find_common_snapshot(
target_dataset, guid_check=guid_check, bookmark_tag=bookmark_tag)
incompatible_target_snapshots = target_dataset.find_incompatible_snapshots(target_common_snapshot, raw)
else:
source_common_snapshot = None
incompatible_target_snapshots = []
# let thinner decide whats obsolete on source after the transfer is done
if source_dataset.our_snapshots:
source_obsoletes = source_dataset.thin_list()[1]
else:
source_obsoletes = []
### 2: determine possible target snapshots
# start with snapshots that already exist, minus incompatibles
if target_dataset.exists:
possible_target_snapshots = [snapshot for snapshot in target_dataset.snapshots
if snapshot not in incompatible_target_snapshots]
else:
possible_target_snapshots = []
# add all snapshots from the source, starting after the common snapshot if it exists
if source_common_snapshot is not None:
source_snapshot = source_dataset.find_next_snapshot(source_common_snapshot)
elif source_dataset.snapshots:
source_snapshot = source_dataset.snapshots[0]
else:
source_snapshot = None
while source_snapshot:
# we want it?
is_required = required_snapshots is not None and source_snapshot in required_snapshots
if is_required and not source_snapshot.is_ours and not also_other_snapshots:
source_snapshot.verbose("Including as clone origin for a selected clone")
if (also_other_snapshots or source_snapshot.is_ours or is_required) \
and not source_snapshot.is_snapshot_excluded:
# create virtual target snapshot
target_snapshot = target_dataset.zfs_node.get_snapshot(
target_dataset.name + source_snapshot.typed_suffix, force_exists=False)
possible_target_snapshots.append(target_snapshot)
source_snapshot = source_dataset.find_next_snapshot(source_snapshot)
### 3: let the thinner decide what it wants by looking at all the possible target snapshots at once.
# always keep the last target snapshot as common snapshot.
if possible_target_snapshots:
(target_keeps, target_obsoletes) = target_dataset.zfs_node.thin_list(
possible_target_snapshots, keep_snapshots=[possible_target_snapshots[-1]])
else:
target_keeps = []
target_obsoletes = []
### 4: look at what the thinner wants to keep and create a list of snapshots we still need to transfer
target_transfers = [target_keep for target_keep in target_keeps if not target_keep.exists]
### 5: no common snapshot, but the source is a clone? then the origin acts as the common snapshot:
# the first send becomes an incremental from the origin, so zfs recv reconstructs the clone
# relationship on the target.
is_clone_origin = False
if source_common_snapshot is None and not no_clone and target_transfers:
source_common_snapshot = _resolve_clone_origin(source_dataset, target_dataset.zfs_node, target_path,
strip_path, guid_check)
is_clone_origin = source_common_snapshot is not None
return (source_common_snapshot, is_clone_origin, source_obsoletes, target_obsoletes, target_transfers,
incompatible_target_snapshots)
def _active_properties(source_dataset, filter_properties, set_properties, property_format):
"""determine the property filter/set lists to use for this transfer
:type source_dataset: ZfsContainer
:type filter_properties: list[str]
:type set_properties: list[str]
:type property_format: str
:rtype: (list[str], list[str])
"""
(active_filter_properties, active_set_properties) = source_dataset.get_allowed_properties(filter_properties,
set_properties)
# always filter properties that start with the property-format prefix (https://github.com/psy0rz/zfs_autobackup/issues/221)
property_prefix = property_format.split(':')[0]
for prop in source_dataset.properties:
if prop.startswith(property_prefix):
active_filter_properties.append(prop)
return active_filter_properties, active_set_properties
def _post_transfer_source_cleanup(source_snapshot, prev_base, source_obsoletes, holds, use_bookmarks, bookmark_tag):
"""after source_snapshot was transferred: hold or bookmark it as the new common snapshot, and
release/destroy the previous one if its obsolete. Returns the source-side point-in-time to use
as base for the next incremental send.
:type source_snapshot: ZfsSnapshot
:type prev_base: ZfsPointInTime|None
:type source_obsoletes: list[ZfsSnapshot]
:type holds: bool
:type use_bookmarks: bool
:type bookmark_tag: str
:rtype: ZfsPointInTime
"""
if use_bookmarks:
# bookmark the new common snapshot, so the snapshot itself can be destroyed when obsolete
source_bookmark = source_snapshot.bookmark(bookmark_tag)
if source_snapshot in source_obsoletes:
source_snapshot.destroy()
# TODO: make a better is_ours specially for bookmarks, as part of the next refactoring splitting in more classes
# delete the previous bookmark if its ours (ends in our tag_separator + tag)
if isinstance(prev_base, ZfsBookmark) and prev_base.name.endswith(
source_snapshot.zfs_node.tag_seperator + bookmark_tag):
prev_base.destroy()
return source_bookmark
# not using bookmarks: hold the new common snapshot instead
if holds:
source_snapshot.hold()
# release hold and cleanup the previous common snapshot if its obsolete
if isinstance(prev_base, ZfsSnapshot):
prev_base.release()
if prev_base in source_obsoletes:
prev_base.destroy()
return source_snapshot
def sync_snapshots(source_dataset, target_dataset, features, show_progress, filter_properties, set_properties,
ignore_recv_exit_code, holds, rollback, decrypt, encrypt, also_other_snapshots,
no_send, destroy_incompatible, send_pipes, recv_pipes, zfs_compressed, force, guid_check,
use_bookmarks, bookmark_tag, property_format, no_clone=True,
target_path=None, strip_path=0, required_snapshots=None):
"""sync source_dataset's snapshots to target_dataset, while also thinning out old snapshots along the way.
:type source_dataset: ZfsContainer
:type target_dataset: ZfsContainer
:type features: list[str]
:type show_progress: bool
:type filter_properties: list[str]
:type set_properties: list[str]
:type ignore_recv_exit_code: bool
:type holds: bool
:type rollback: bool
:type decrypt: bool
:type encrypt: bool
:type also_other_snapshots: bool
:type no_send: bool
:type destroy_incompatible: bool
:type send_pipes: list[str]
:type recv_pipes: list[str]
:type zfs_compressed: bool
:type force: bool
:type guid_check: bool
:type use_bookmarks: bool
:type bookmark_tag: str
:type property_format: str
:type no_clone: bool
:type target_path: str
:type strip_path: int
:type required_snapshots: set[ZfsSnapshot]|None
"""
# determine how to handle encryption during the transfer:
send_properties = True
raw = False
# source dataset encrypted?
if source_dataset.properties.get('encryption', 'off') != 'off':
# user wants to send it over decrypted?
if decrypt:
# when decrypting, zfs can't send properties
send_properties = False
else:
# keep data encrypted by sending it raw (including properties)
raw = True
(source_common_snapshot, is_clone_origin, source_obsoletes, target_obsoletes, target_transfers,
incompatible_target_snapshots) = \
_plan_sync(source_dataset, target_dataset=target_dataset, also_other_snapshots=also_other_snapshots,
guid_check=guid_check, raw=raw, bookmark_tag=bookmark_tag,
no_clone=no_clone, target_path=target_path, strip_path=strip_path,
required_snapshots=required_snapshots)
source_dataset.zfs_node.logger.progress("Pre-cleaning..")
# NOTE: we do a pre-clean because we dont want filesystems to fillup when backups keep failing.
# Also usefull with no_send to still cleanup stuff.
_pre_clean(
source_dataset, target_dataset=target_dataset, source_common_snapshot=source_common_snapshot,
source_obsoletes=source_obsoletes, target_obsoletes=target_obsoletes, target_transfers=target_transfers)
# check if we can resume
if len(target_transfers) > 0:
source_dataset.zfs_node.logger.progress("Verifying resume token...")
resume_token = _validate_resume_token(source_dataset, target_dataset, target_transfers[0])
else:
resume_token = None
source_dataset.zfs_node.logger.progress("Preparing...")
# handle incompatible stuff on target
_handle_incompatible_target(target_dataset, incompatible_target_snapshots, destroy_incompatible, force,
source_common_snapshot, is_clone_origin, resume_token is not None)
# now actually transfer the snapshots, if we want
if no_send or len(target_transfers) == 0:
return
(active_filter_properties, active_set_properties) = _active_properties(source_dataset, filter_properties,
set_properties, property_format)
# encrypt at target?
write_embedded = True
if encrypt and not raw:
# filter out encryption properties to let encryption on the target take place
active_filter_properties.extend(["keylocation", "pbkdf2iters", "keyformat", "encryption"])
write_embedded = False
# the source-side point-in-time (snapshot, bookmark or clone origin) to use as base for the
# next incremental send
incremental_base = source_common_snapshot
# the target-side counterpart of the common snapshot. (a clone origin has no counterpart here:
# it lives on the origin's target dataset)
if is_clone_origin:
prev_target_snapshot = None
else:
prev_target_snapshot = target_dataset.find_snapshot(source_common_snapshot)
do_rollback = rollback
for target_snapshot in target_transfers:
source_snapshot = source_dataset.find_snapshot(target_snapshot)
assert source_snapshot is not None
# do the rollback, one time at first transfer
if do_rollback:
source_dataset.zfs_node.logger.progress("Rolling back {}...".format(target_dataset))
target_dataset.rollback()
do_rollback = False
source_snapshot.transfer_snapshot(target_snapshot, features=features,
prev_snapshot=incremental_base, show_progress=show_progress,
filter_properties=active_filter_properties,
set_properties=active_set_properties,
ignore_recv_exit_code=ignore_recv_exit_code,
resume_token=resume_token, write_embedded=write_embedded, raw=raw,
send_properties=send_properties, send_pipes=send_pipes,
recv_pipes=recv_pipes, zfs_compressed=zfs_compressed, force=force)
# the resume token is only valid for the first transfer
resume_token = None
# hold/release common snapshot on the target.
if holds:
target_snapshot.hold()
if prev_target_snapshot:
prev_target_snapshot.release()
# cleanup on the source and determine the base for the next incremental send
incremental_base = _post_transfer_source_cleanup(source_snapshot, incremental_base, source_obsoletes,
holds, use_bookmarks, bookmark_tag)
# destroy the previous target snapshot if obsolete (usually this is only the common_snapshot,
# the rest was already destroyed or will not be send)
if prev_target_snapshot is not None and prev_target_snapshot in target_obsoletes:
prev_target_snapshot.destroy()
prev_target_snapshot = target_snapshot
+17 -12
View File
@@ -1,4 +1,3 @@
# NOTE: surprisingly sha1 in via python3 is faster than the native sha1sum utility, even in the way we use below!
import os
import platform
@@ -9,19 +8,21 @@ from datetime import datetime
def tmp_name(suffix=""):
"""create temporary name unique to this process and node. always retruns the same result during the same execution"""
#we could use uuids but those are ugly and confusing
name="{}-{}-{}".format(
os.path.basename(sys.argv[0]).replace(" ","_"),
# we could use uuids but those are ugly and confusing
name = "{}-{}-{}".format(
os.path.basename(sys.argv[0]).replace(" ", "_"),
platform.node(),
os.getpid())
name=name+suffix
name = name + suffix
return name
def get_tmp_clone_name(snapshot):
pool=snapshot.zfs_node.get_pool(snapshot)
return pool.name+"/"+tmp_name()
"""
:type snapshot: ZfsSnapshot
"""
pool = snapshot.zfs_node.get_pool(snapshot)
return pool.name + "/" + tmp_name()
def output_redir():
@@ -33,10 +34,12 @@ def output_redir():
os.dup2(devnull, sys.stdout.fileno())
os.dup2(devnull, sys.stderr.fileno())
def sigpipe_handler(sig, stack):
#redir output so we dont get more SIGPIPES during cleanup. (which my try to write to stdout)
# redir output so we dont get more SIGPIPES during cleanup. (which my try to write to stdout)
output_redir()
#deb('redir')
# deb('redir')
# def check_output():
# """make sure stdout still functions. if its broken, this will trigger a SIGPIPE which will be handled by the sigpipe_handler."""
@@ -55,9 +58,11 @@ def sigpipe_handler(sig, stack):
# This function will be mocked during unit testing.
datetime_now_mock=None
datetime_now_mock = None
def datetime_now(utc):
if datetime_now_mock is None:
return( datetime.utcnow() if utc else datetime.now())
return (datetime.utcnow() if utc else datetime.now())
else:
return datetime_now_mock