2022-03-03T12:00:27.099 INFO:root:teuthology version: 1.1.0-eeaeee38 2022-03-03T12:00:27.101 DEBUG:teuthology.run:Teuthology command: teuthology --name mrfreezeex-2022-03-03_11:17:22-rbd-wip-mrfreezeex-multipeer-distro-default-smithi --owner scheduled_mrfreezeex@teuthology --archive /home/teuthworker/archive/mrfreezeex-2022-03-03_11:17:22-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6717912 --description rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-comp-lz4 policy/simple rbd-mirror/four-per-cluster supported-random-distro$/{centos_8} workloads/rbd-mirror-journal-workunit-multipeer} --verbose -- /home/teuthworker/archive/mrfreezeex-2022-03-03_11:17:22-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6717912/orig.config.yaml 2022-03-03T12:00:27.156 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2022-03-03T12:00:27.227 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/mrfreezeex-2022-03-03_11:17:22-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6717912 branch: wip-mrfreezeex-multipeer description: rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-comp-lz4 policy/simple rbd-mirror/four-per-cluster supported-random-distro$/{centos_8} workloads/rbd-mirror-journal-workunit-multipeer} email: arthur.outhenin-chalandre@cern.ch first_in_suite: false job_id: '6717912' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi meta: - desc: run two ceph clusters and install rbd-mirror - desc: configure the permissions for client.mirror - desc: 2 ceph clusters with 1 mon and 3 osds each - desc: run four rbd-mirror daemons per cluster - desc: run the rbd_mirror_journal.sh workunit to test the rbd-mirror daemon name: mrfreezeex-2022-03-03_11:17:22-rbd-wip-mrfreezeex-multipeer-distro-default-smithi nuke-on-error: true openstack: - volumes: count: 3 size: 30 os_type: centos os_version: 8.stream overrides: admin_socket: branch: wip-mrfreezeex-multipeer ceph: conf: client: debug rbd: 20 debug rbd_mirror: 15 log to stderr: false rbd default features: 125 rbd mirror image policy type: simple client.mirror.0: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid client.mirror.1: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid client.mirror.2: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid client.mirror.3: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid global: mon client directed command retry: 5 ms inject socket failures: 5000 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: bluestore block size: 96636764160 bluestore compression algorithm: lz4 bluestore compression mode: aggressive bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME log-whitelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: 501cde21dbba697e001f934aba397a4bf89fefdd ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 install: ceph: flavor: default sha1: 501cde21dbba697e001f934aba397a4bf89fefdd selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-mrfreezeex-multipeer sha1: 501cde21dbba697e001f934aba397a4bf89fefdd owner: scheduled_mrfreezeex@teuthology priority: 160 repo: git://git.ceph.com/ceph-ci.git roles: - - cluster1.mon.a - cluster1.mgr.x - cluster2.mgr.x - cluster1.osd.0 - cluster1.osd.1 - cluster1.osd.2 - cluster1.client.0 - cluster2.client.0 - - cluster2.mon.a - cluster2.osd.0 - cluster2.osd.1 - cluster2.osd.2 - cluster1.client.mirror - cluster1.client.mirror.0 - cluster1.client.mirror.1 - cluster1.client.mirror.2 - cluster1.client.mirror.3 - cluster1.client.mirror.4 - cluster1.client.mirror.5 - cluster1.client.mirror.6 - cluster2.client.mirror - cluster2.client.mirror.0 - cluster2.client.mirror.1 - cluster2.client.mirror.2 - cluster2.client.mirror.3 - cluster2.client.mirror.4 - cluster2.client.mirror.5 - cluster2.client.mirror.6 sha1: 501cde21dbba697e001f934aba397a4bf89fefdd sleep_before_teardown: 0 suite: rbd suite_branch: wip-mrfreezeex-multipeer suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_501cde21dbba697e001f934aba397a4bf89fefdd/qa suite_relpath: qa suite_repo: git://git.ceph.com/ceph-ci.git suite_sha1: 501cde21dbba697e001f934aba397a4bf89fefdd targets: smithi058.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCeNYL9hiOh/47PNaWj9SpTxX5x8qbPZKs3WX54i1UUZJZ0LV1JwKarMRJXxICHklRF3+j54NsPr8LST2dEjLKomgr8WgeVNXi7aIVq0rkfg3Z+BjRpSKPFS0ojXiCq+/5Cc+hVUtSmDx5HjhafuzSFBFvxl5L+gEUMWP4pm6p0CbD76y3hNBs1agpT5q3eUSSCeY3aEV492H5RpBzCYeMIXvDO4HC2PWvpFYw0ErdmOv0Qfa3CrPHHShGLVLDuJRF+GA9/QS5sO9v/hXfSxLR9hbR7EibqKOgHg/91jMkSiFY0Rvu5mFtlrRqWkGZBXPK6YHmmGm7txVDyjtgVfAGNZJ5+3FYxqAYGQjL8Eqk79bS2dFK7HT/peHu4kxkyj3rAo/COAgpGlyUtI2AwGb3x+iOcwQMwPk24DZHy01BKqfhN38fDEg2F6TV/G/5ZhUUi7mYLRYt8PkjJ7dMgorqu36la5Um+NRYl+YOX1zKgT3l4gwodqNevQaeaSKdKnAM= smithi083.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCqxKr5qXHIcswquPIBbdbkhd+8tOUoyRUr6NryrNGDkBmDs6Dfs9sODpP8AH7fUmXoTqVrAnsjtLccPjcGXEjGWPxd531+5RrdxfWnAd83SoJ+Dp+gaIN+gTUXzyjFYQTvD0/ldzwlAz2ju3bWGvwAvO8NPkQXou2KQtMZwL05PZpJEDORgU8cU2TomnLbLOAEZVzjRPNv4/J8bv8AtZBY1n+SVS5ueWdbUJhSkwjJVqOGBG7xvEiIyJJl0rSY99STQ5hjeWi+wzClh+pyhTLOKy54uiAveX8KV6iWE9qK+LRGlnCfK1rxsuHX6PDD2YQQLhTHLD+V9pekQ07c0nLUjQRlTIkqAcD5rDae9v0hJUBInVB5ojtDS2bEldkWRdoQWKjcTM+13D6ZqGZUF8hyqMV7tspOxvTFDLjFErIvmt2JzWvmRNAcbNONny1NVgHD5M8nLYpoJ7UYwTO19eP8AoaGFsz1F6zUs5hqO2rtx7zVvl6x54W15ogNQkSdty0= tasks: - install: extra_packages: - rbd-mirror - ceph: cluster: cluster1 - ceph: cluster: cluster2 - exec: cluster1.client.mirror.0: - sudo ceph --cluster cluster1 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd' - rbd-mirror: client: cluster1.client.mirror.0 thrash: true - rbd-mirror: client: cluster1.client.mirror.1 thrash: true - rbd-mirror: client: cluster1.client.mirror.2 thrash: true - rbd-mirror: client: cluster1.client.mirror.3 thrash: true - rbd-mirror: client: cluster2.client.mirror.0 thrash: true - rbd-mirror: client: cluster2.client.mirror.1 thrash: true - rbd-mirror: client: cluster2.client.mirror.2 thrash: true - rbd-mirror: client: cluster2.client.mirror.3 thrash: true - rbd-mirror-thrash: cluster: cluster1 - rbd-mirror-thrash: cluster: cluster2 - ceph: cluster: cluster3 - rbd-mirror: client: cluster3.client.mirror.0 thrash: true - rbd-mirror: client: cluster3.client.mirror.1 thrash: true - rbd-mirror: client: cluster3.client.mirror.2 thrash: true - rbd-mirror: client: cluster3.client.mirror.3 thrash: true - rbd-mirror-thrash: cluster: cluster3 - workunit: clients: cluster1.client.mirror: - rbd/rbd_mirror_journal.sh env: CEPH_ARGS: '' RBD_MIRROR_INSTANCES: '4' RBD_MIRROR_USE_EXISTING_CLUSTER: '1' RBD_MIRROR_USE_RBD_MIRROR: '1' USE_CLUSTER3: '1' teuthology_branch: master teuthology_sha1: eeaeee38264fad8a01fd1ce912d8908676ed64bc timestamp: 2022-03-03_11:17:22 tube: smithi user: mrfreezeex verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.22301 2022-03-03T12:00:27.227 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_501cde21dbba697e001f934aba397a4bf89fefdd/qa; will attempt to use it 2022-03-03T12:00:27.228 INFO:teuthology.run:Tasks not found; will attempt to fetch 2022-03-03T12:00:27.229 INFO:teuthology.repo_utils:/home/teuthworker/src/git.ceph.com_ceph-c_501cde21dbba697e001f934aba397a4bf89fefdd was just updated or references a specific commit; assuming it is current 2022-03-03T12:00:27.229 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-c_501cde21dbba697e001f934aba397a4bf89fefdd to 501cde21dbba697e001f934aba397a4bf89fefdd 2022-03-03T12:00:27.236 CRITICAL:teuthology:Uncaught exception Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_eeaeee38264fad8a01fd1ce912d8908676ed64bc/teuthology/repo_utils.py", line 297, in reset_repo cwd=dest_path, File "/usr/lib/python3.6/subprocess.py", line 356, in check_output **kwargs).stdout File "/usr/lib/python3.6/subprocess.py", line 438, in run output=stdout, stderr=stderr) subprocess.CalledProcessError: Command '('git', 'reset', '--hard', '501cde21dbba697e001f934aba397a4bf89fefdd')' returned non-zero exit status 128. During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_eeaeee38264fad8a01fd1ce912d8908676ed64bc/virtualenv/bin/teuthology", line 33, in sys.exit(load_entry_point('teuthology', 'console_scripts', 'teuthology')()) File "/home/teuthworker/src/git.ceph.com_git_teuthology_eeaeee38264fad8a01fd1ce912d8908676ed64bc/scripts/run.py", line 38, in main teuthology.run.main(args) File "/home/teuthworker/src/git.ceph.com_git_teuthology_eeaeee38264fad8a01fd1ce912d8908676ed64bc/teuthology/run.py", line 388, in main config["suite_path"] = fetch_tasks_if_needed(config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_eeaeee38264fad8a01fd1ce912d8908676ed64bc/teuthology/run.py", line 90, in fetch_tasks_if_needed fetch_qa_suite(suite_branch, commit=suite_sha1), File "/home/teuthworker/src/git.ceph.com_git_teuthology_eeaeee38264fad8a01fd1ce912d8908676ed64bc/teuthology/repo_utils.py", line 403, in fetch_qa_suite branch, commit, lock=lock) File "/home/teuthworker/src/git.ceph.com_git_teuthology_eeaeee38264fad8a01fd1ce912d8908676ed64bc/teuthology/repo_utils.py", line 341, in fetch_repo enforce_repo_state(url, dest_path, branch, commit) File "/home/teuthworker/src/git.ceph.com_git_teuthology_eeaeee38264fad8a01fd1ce912d8908676ed64bc/teuthology/repo_utils.py", line 105, in enforce_repo_state reset_repo(repo_url, dest_path, branch, commit) File "/home/teuthworker/src/git.ceph.com_git_teuthology_eeaeee38264fad8a01fd1ce912d8908676ed64bc/teuthology/repo_utils.py", line 301, in reset_repo raise CommitNotFoundError(commit, repo_url) teuthology.exceptions.CommitNotFoundError: '501cde21dbba697e001f934aba397a4bf89fefdd' not found in repo: git://git.ceph.com/ceph-ci.git!