2022-03-03T12:00:18.224 INFO:root:teuthology version: 1.1.0-eeaeee38 2022-03-03T12:00:18.226 DEBUG:teuthology.run:Teuthology command: teuthology --owner scheduled_mrfreezeex@teuthology --archive /home/teuthworker/archive/mrfreezeex-2022-03-03_11:17:22-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6717908 --verbose --description rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-low-osd-mem-target supported-random-distro$/{ubuntu_latest} workloads/rbd-mirror-snapshot-workunit-minimum} --name mrfreezeex-2022-03-03_11:17:22-rbd-wip-mrfreezeex-multipeer-distro-default-smithi -- /home/teuthworker/archive/mrfreezeex-2022-03-03_11:17:22-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6717908/orig.config.yaml 2022-03-03T12:00:18.277 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2022-03-03T12:00:18.347 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/mrfreezeex-2022-03-03_11:17:22-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6717908 branch: wip-mrfreezeex-multipeer description: rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-low-osd-mem-target supported-random-distro$/{ubuntu_latest} workloads/rbd-mirror-snapshot-workunit-minimum} email: arthur.outhenin-chalandre@cern.ch first_in_suite: false job_id: '6717908' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi meta: - desc: run two ceph clusters and install rbd-mirror - desc: configure the permissions for client.mirror - desc: configure the permissions for client.mirror - desc: 2 ceph clusters with 1 mon and 3 osds each - desc: run the rbd_mirror_snapshot.sh workunit to test the rbd-mirror daemon name: mrfreezeex-2022-03-03_11:17:22-rbd-wip-mrfreezeex-multipeer-distro-default-smithi nuke-on-error: true openstack: - volumes: count: 3 size: 30 os_type: ubuntu os_version: '20.04' overrides: admin_socket: branch: wip-mrfreezeex-multipeer ceph: conf: client: debug rbd: 20 debug rbd_mirror: 15 log to stderr: false rbd default features: 125 client.mirror.0: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid client.mirror.1: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid client.mirror.2: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid client.mirror.3: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid client.mirror.4: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid client.mirror.5: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid client.mirror.6: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid global: mon client directed command retry: 5 ms inject socket failures: 5000 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd memory target: 1610612736 osd objectstore: bluestore flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME log-whitelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: 501cde21dbba697e001f934aba397a4bf89fefdd ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 install: ceph: flavor: default sha1: 501cde21dbba697e001f934aba397a4bf89fefdd thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-mrfreezeex-multipeer sha1: 501cde21dbba697e001f934aba397a4bf89fefdd owner: scheduled_mrfreezeex@teuthology priority: 160 repo: git://git.ceph.com/ceph-ci.git roles: - - cluster1.mon.a - cluster1.mgr.x - cluster2.mgr.x - cluster1.osd.0 - cluster1.osd.1 - cluster1.osd.2 - cluster1.client.0 - cluster2.client.0 - - cluster2.mon.a - cluster2.osd.0 - cluster2.osd.1 - cluster2.osd.2 - cluster1.client.mirror - cluster1.client.mirror.0 - cluster1.client.mirror.1 - cluster1.client.mirror.2 - cluster1.client.mirror.3 - cluster1.client.mirror.4 - cluster1.client.mirror.5 - cluster1.client.mirror.6 - cluster2.client.mirror - cluster2.client.mirror.0 - cluster2.client.mirror.1 - cluster2.client.mirror.2 - cluster2.client.mirror.3 - cluster2.client.mirror.4 - cluster2.client.mirror.5 - cluster2.client.mirror.6 sha1: 501cde21dbba697e001f934aba397a4bf89fefdd sleep_before_teardown: 0 suite: rbd suite_branch: wip-mrfreezeex-multipeer suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_501cde21dbba697e001f934aba397a4bf89fefdd/qa suite_relpath: qa suite_repo: git://git.ceph.com/ceph-ci.git suite_sha1: 501cde21dbba697e001f934aba397a4bf89fefdd targets: smithi072.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDJgJ/wwPo5MSXasKkImELL5bcTho1xYIe7xwsmpCA+gksF8aPav6GQ8OeqPTKiIRFGvgCP3ANl8C2e1EPq6J27NF8ysFMLV874P6xtKeWalcxB/egUithCn/ZbyYo9If8HmDPNdDR6GKTvVZMPJd7mTgWN+Ekil0SVzBgcRtXAdrtQ2gJZuVexWQujMUa/xudOPrmU/YVx4NlGwWQQkZ33QWL/I103k3GqCZCDp+zhvkxXMP82/YNi/6j1C/4o3fr/2IEh40ULVOYtbwnD9LNIntJagA0RL6oPzUF/383kZGYEmv8u1Lg2QDek539zguAtdCAGU0GpO2bh8/zHHwr+raetH6Qxt+szGq65ZUVMnJxMlCENb8AMD7OSja3q6kraJE4ejVQTTlJDTd1YceF6giA7jYsTbt67hIedZrr39qwfwwEfSL+uEHufEwSO0dpwf+dBUyg64l7V4OwN8tsf1BZ4pjfhtKrnCsZWRQ3j8y24lh+8ogKb3PJWwRERVss= smithi179.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCVzp+v0nJUnFCWs17hJ2IM8aAfRqyFnjEcZDzW6hbi934c/G6JK3Tkspt/j7Z44u/uXDPwkBKWx+fnEuiudBhGvGnVYcBjpgePtL3aPnfa1uYYVLOqukUDfA3fhB3lBOg/wMCFgfJNb6tGrtYxV56QrckCiDHGT1A73stvaD0ACWVETDg3/sw9GUJrTWyEAKaEz61Xm+MzpUwiUufv7nk+0hTExT2sHZhlLGX7NZ4uRnofN+8+Zj32Z4ZhVb7JUGdVkIMzG+t+3vgc6i3D2+0wivh1irEChGCQGnsiM221huganGltKhQbIiX7WRxVWae9Vz/nQdPm44Mdj4QtSt5/XaX84JSZiKse148+UU4DfI2AJEYQeW78EYbgEVJVMUbD0Ec8WQiTkUdNSGGtt70sG7NBF5tUjFzMtam9/fTtObH47WAKIM8XWk6KGeU1klgggteBGXAvMscxnwfpWRbF6itYqEkTryniQ/BOHV1DkOk3Ij5BvshLQtlavauA00E= tasks: - install: extra_packages: - rbd-mirror - ceph: cluster: cluster1 - ceph: cluster: cluster2 - exec: cluster1.client.mirror.0: - sudo ceph --cluster cluster1 auth caps client.mirror.4 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.5 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.6 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.4 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.5 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.6 mon 'profile rbd-mirror' osd 'profile rbd' - exec: cluster1.client.mirror.0: - sudo ceph --cluster cluster1 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd' - workunit: clients: cluster1.client.mirror: - rbd/rbd_mirror_snapshot.sh env: CEPH_ARGS: '' RBD_IMAGE_FEATURES: layering RBD_MIRROR_CONFIG_KEY: '1' RBD_MIRROR_INSTANCES: '4' RBD_MIRROR_USE_EXISTING_CLUSTER: '1' teuthology_branch: master teuthology_sha1: eeaeee38264fad8a01fd1ce912d8908676ed64bc timestamp: 2022-03-03_11:17:22 tube: smithi user: mrfreezeex verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.22301 2022-03-03T12:00:18.348 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_501cde21dbba697e001f934aba397a4bf89fefdd/qa; will attempt to use it 2022-03-03T12:00:18.348 INFO:teuthology.run:Tasks not found; will attempt to fetch 2022-03-03T12:00:18.349 INFO:teuthology.repo_utils:Cloning git://git.ceph.com/ceph-ci.git wip-mrfreezeex-multipeer from upstream 2022-03-03T12:00:38.795 ERROR:teuthology.repo_utils:Git error encountered; retrying Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_eeaeee38264fad8a01fd1ce912d8908676ed64bc/teuthology/repo_utils.py", line 341, in fetch_repo enforce_repo_state(url, dest_path, branch, commit) File "/home/teuthworker/src/git.ceph.com_git_teuthology_eeaeee38264fad8a01fd1ce912d8908676ed64bc/teuthology/repo_utils.py", line 94, in enforce_repo_state clone_repo(repo_url, dest_path, branch, shallow=commit is None) File "/home/teuthworker/src/git.ceph.com_git_teuthology_eeaeee38264fad8a01fd1ce912d8908676ed64bc/teuthology/repo_utils.py", line 153, in clone_repo raise GitError("git clone failed!") teuthology.exceptions.GitError: git clone failed! 2022-03-03T12:00:48.806 INFO:teuthology.repo_utils:/home/teuthworker/src/git.ceph.com_ceph-c_501cde21dbba697e001f934aba397a4bf89fefdd was just updated or references a specific commit; assuming it is current 2022-03-03T12:00:48.807 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-c_501cde21dbba697e001f934aba397a4bf89fefdd to 501cde21dbba697e001f934aba397a4bf89fefdd 2022-03-03T12:00:48.818 CRITICAL:teuthology:Uncaught exception Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_eeaeee38264fad8a01fd1ce912d8908676ed64bc/teuthology/repo_utils.py", line 297, in reset_repo cwd=dest_path, File "/usr/lib/python3.6/subprocess.py", line 356, in check_output **kwargs).stdout File "/usr/lib/python3.6/subprocess.py", line 438, in run output=stdout, stderr=stderr) subprocess.CalledProcessError: Command '('git', 'reset', '--hard', '501cde21dbba697e001f934aba397a4bf89fefdd')' returned non-zero exit status 128. During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_eeaeee38264fad8a01fd1ce912d8908676ed64bc/virtualenv/bin/teuthology", line 33, in sys.exit(load_entry_point('teuthology', 'console_scripts', 'teuthology')()) File "/home/teuthworker/src/git.ceph.com_git_teuthology_eeaeee38264fad8a01fd1ce912d8908676ed64bc/scripts/run.py", line 38, in main teuthology.run.main(args) File "/home/teuthworker/src/git.ceph.com_git_teuthology_eeaeee38264fad8a01fd1ce912d8908676ed64bc/teuthology/run.py", line 388, in main config["suite_path"] = fetch_tasks_if_needed(config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_eeaeee38264fad8a01fd1ce912d8908676ed64bc/teuthology/run.py", line 90, in fetch_tasks_if_needed fetch_qa_suite(suite_branch, commit=suite_sha1), File "/home/teuthworker/src/git.ceph.com_git_teuthology_eeaeee38264fad8a01fd1ce912d8908676ed64bc/teuthology/repo_utils.py", line 403, in fetch_qa_suite branch, commit, lock=lock) File "/home/teuthworker/src/git.ceph.com_git_teuthology_eeaeee38264fad8a01fd1ce912d8908676ed64bc/teuthology/repo_utils.py", line 341, in fetch_repo enforce_repo_state(url, dest_path, branch, commit) File "/home/teuthworker/src/git.ceph.com_git_teuthology_eeaeee38264fad8a01fd1ce912d8908676ed64bc/teuthology/repo_utils.py", line 105, in enforce_repo_state reset_repo(repo_url, dest_path, branch, commit) File "/home/teuthworker/src/git.ceph.com_git_teuthology_eeaeee38264fad8a01fd1ce912d8908676ed64bc/teuthology/repo_utils.py", line 301, in reset_repo raise CommitNotFoundError(commit, repo_url) teuthology.exceptions.CommitNotFoundError: '501cde21dbba697e001f934aba397a4bf89fefdd' not found in repo: git://git.ceph.com/ceph-ci.git!