2022-01-19T16:42:42.860 INFO:root:teuthology version: 1.1.0-3ff86a5d 2022-01-19T16:42:42.861 INFO:teuthology.results:subset: '1/3' 2022-01-19T16:42:42.862 INFO:teuthology.results:seed: '2' 2022-01-19T18:41:17.436 INFO:root:teuthology version: 1.1.0-3ff86a5d 2022-01-19T18:41:17.485 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2022-01-19T19:49:22.987 INFO:teuthology.results:Tests finished! gathering results... 2022-01-19T19:49:29.720 INFO:scrape:Found 84 jobs 2022-01-19T19:49:37.916 INFO:scrape:Found 6 distinct failure reasons 2022-01-19T19:49:37.917 INFO:scrape:Failure: 2022-01-19T19:49:37.917 INFO:scrape:9 jobs: ['6627235', '6627230', '6627245', '6627174', '6627207', '6627189', '6627179', '6627217', '6627202'] 2022-01-19T19:49:37.918 INFO:scrape:suites intersection: ['cluster/{2-node', 'msgr-failures/few', 'openstack}'] 2022-01-19T19:49:37.918 INFO:scrape:suites union: ['clients/mirror', 'clients/{mirror-extra', 'cluster/{2-node', 'mirror}', 'msgr-failures/few', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-comp-snappy', 'objectstore/bluestore-comp-zstd', 'objectstore/bluestore-low-osd-mem-target', 'objectstore/bluestore-stupid', 'objectstore/filestore-xfs', 'openstack}', 'policy/none', 'policy/simple', 'rbd-mirror/four-per-cluster', 'rbd/mirror-thrash/{base/install', 'rbd/mirror/{base/install', 'supported-random-distro$/{centos_8.stream}', 'supported-random-distro$/{centos_8}', 'supported-random-distro$/{rhel_8}', 'supported-random-distro$/{ubuntu_latest}', 'workloads/rbd-mirror-journal-workunit-multipeer}', 'workloads/rbd-mirror-snapshot-workunit-fast-diff-multipeer}'] 2022-01-19T19:49:37.919 INFO:scrape: 2022-01-19T19:49:37.919 INFO:scrape:Failure: Command failed (workunit test rbd/rbd_mirror_journal.sh) on smithi163 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=d238de862197118eb4dfe9e422168d942c06f08c TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\' RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_journal.sh' 2022-01-19T19:49:37.920 INFO:scrape:14 jobs: ['6627225', '6627204', '6627248', '6627176', '6627247', '6627246', '6627218', '6627191', '6627192', '6627190', '6627219', '6627232', '6627220', '6627194'] 2022-01-19T19:49:37.920 INFO:scrape:suites intersection: ['cluster/{2-node', 'msgr-failures/few', 'openstack}'] 2022-01-19T19:49:37.921 INFO:scrape:suites union: ['clients/mirror', 'clients/{mirror-extra', 'cluster/{2-node', 'mirror}', 'msgr-failures/few', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-comp-lz4', 'objectstore/bluestore-comp-snappy', 'objectstore/bluestore-comp-zlib', 'objectstore/bluestore-comp-zstd', 'objectstore/bluestore-hybrid', 'objectstore/bluestore-stupid', 'objectstore/filestore-xfs', 'openstack}', 'policy/none', 'policy/simple', 'rbd-mirror/four-per-cluster', 'rbd/mirror-thrash/{base/install', 'rbd/mirror/{base/install', 'supported-random-distro$/{centos_8.stream}', 'supported-random-distro$/{centos_8}', 'supported-random-distro$/{rhel_8}', 'supported-random-distro$/{ubuntu_latest}', 'workloads/rbd-mirror-journal-workunit}', 'workloads/rbd-mirror-workunit-config-key}', 'workloads/rbd-mirror-workunit-min-compat-client-octopus}', 'workloads/rbd-mirror-workunit-policy-none}', 'workloads/rbd-mirror-workunit-policy-simple}'] 2022-01-19T19:49:37.921 INFO:scrape: 2022-01-19T19:49:37.922 INFO:scrape:Failure: Command failed (workunit test rbd/rbd_mirror_snapshot.sh) on smithi116 with status 16: 'mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=d238de862197118eb4dfe9e422168d942c06f08c TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\' RBD_IMAGE_FEATURES=layering RBD_MIRROR_CONFIG_KEY=1 RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_snapshot.sh' 2022-01-19T19:49:37.922 INFO:scrape:2 jobs: ['6627185', '6627205'] 2022-01-19T19:49:37.923 INFO:scrape:suites intersection: ['clients/{mirror-extra', 'cluster/{2-node', 'mirror}', 'msgr-failures/few', 'objectstore/bluestore-comp-lz4', 'openstack}', 'rbd/mirror/{base/install'] 2022-01-19T19:49:37.923 INFO:scrape:suites union: ['clients/{mirror-extra', 'cluster/{2-node', 'mirror}', 'msgr-failures/few', 'objectstore/bluestore-comp-lz4', 'openstack}', 'rbd/mirror/{base/install', 'supported-random-distro$/{centos_8}', 'supported-random-distro$/{ubuntu_latest}', 'workloads/rbd-mirror-snapshot-workunit-exclusive-lock}', 'workloads/rbd-mirror-snapshot-workunit-minimum}'] 2022-01-19T19:49:37.924 INFO:scrape: 2022-01-19T19:49:37.924 INFO:scrape:Failure: Command failed (workunit test rbd/rbd_mirror_stress.sh) on smithi154 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=d238de862197118eb4dfe9e422168d942c06f08c TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\' RBD_MIRROR_MIN_COMPAT_CLIENT=octopus RBD_MIRROR_USE_EXISTING_CLUSTER=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_stress.sh' 2022-01-19T19:49:37.925 INFO:scrape:3 jobs: ['6627188', '6627244', '6627216'] 2022-01-19T19:49:37.925 INFO:scrape:suites intersection: ['clients/{mirror-extra', 'cluster/{2-node', 'mirror}', 'msgr-failures/few', 'openstack}', 'rbd/mirror/{base/install', 'workloads/rbd-mirror-stress-workunit-min-compat-client-octopus}'] 2022-01-19T19:49:37.926 INFO:scrape:suites union: ['clients/{mirror-extra', 'cluster/{2-node', 'mirror}', 'msgr-failures/few', 'objectstore/bluestore-comp-lz4', 'objectstore/bluestore-comp-snappy', 'objectstore/bluestore-low-osd-mem-target', 'openstack}', 'rbd/mirror/{base/install', 'supported-random-distro$/{centos_8.stream}', 'supported-random-distro$/{rhel_8}', 'supported-random-distro$/{ubuntu_latest}', 'workloads/rbd-mirror-stress-workunit-min-compat-client-octopus}'] 2022-01-19T19:49:37.926 INFO:scrape: 2022-01-19T19:49:37.927 INFO:scrape:Failure: "2022-01-19T17:37:04.680043+0000 mon.a (mon.0) 201 : cluster [WRN] Health check failed: Degraded data redundancy: 2/1962 objects degraded (0.102%), 1 pg degraded (PG_DEGRADED)" in cluster log 2022-01-19T19:49:37.927 INFO:scrape:1 jobs: ['6627212'] 2022-01-19T19:49:37.927 INFO:scrape:suites: ['clients/mirror', 'cluster/{2-node', 'msgr-failures/few', 'objectstore/bluestore-bitmap', 'openstack}', 'policy/simple', 'rbd-mirror/four-per-cluster', 'rbd/mirror-thrash/{base/install', 'supported-random-distro$/{ubuntu_latest}', 'workloads/rbd-mirror-fsx-workunit}'] 2022-01-19T19:49:37.928 INFO:scrape: 2022-01-19T19:49:37.928 INFO:scrape:Crash: "2022-01-19T17:58:33.126930+0000 mon.a (mon.0) 1574 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log 2022-01-19T19:49:37.929 INFO:scrape:ceph version 17.0.0-10251-gd238de86 (d238de862197118eb4dfe9e422168d942c06f08c) quincy (dev) 1: /lib64/libpthread.so.0(+0x12c20) [0x7f4b9aeccc20] 2: gsignal() 3: abort() 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x1b8) [0x556e00a7a3ac] 5: (KernelDevice::_aio_thread()+0xf9e) [0x556e016b8f4e] 6: (KernelDevice::AioCompletionThread::entry()+0x11) [0x556e016c6781] 7: /lib64/libpthread.so.0(+0x817a) [0x7f4b9aec217a] 8: clone() 2022-01-19T19:49:37.929 INFO:scrape:1 jobs: ['6627206'] 2022-01-19T19:49:37.930 INFO:scrape:suites: ['clients/mirror', 'cluster/{2-node', 'msgr-failures/few', 'objectstore/bluestore-low-osd-mem-target', 'openstack}', 'policy/none', 'rbd-mirror/four-per-cluster', 'rbd/mirror-thrash/{base/install', 'supported-random-distro$/{centos_8.stream}', 'workloads/rbd-mirror-snapshot-stress-workunit-exclusive-lock}'] 2022-01-19T19:49:37.930 INFO:scrape: 2022-01-19T19:49:37.934 INFO:teuthology.results:Sending results to arthur.outhenin-chalandre@cern.ch: Test Run: mrfreezeex-2022-01-19_12:26:12-rbd-wip-mrfreezeex-multipeer-distro-default-smithi ================================================================= info: http://pulpito.ceph.com/mrfreezeex-2022-01-19_12:26:12-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-01-19_12:26:12-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/ failed: 30 dead: 0 running: 0 waiting: 0 queued: 0 passed: 54 Fail ================================================================= [6627174] rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} msgr-failures/few objectstore/filestore-xfs policy/none rbd-mirror/four-per-cluster supported-random-distro$/{centos_8.stream} workloads/rbd-mirror-journal-workunit-multipeer} ----------------------------------------------------------------- time: 00:11:01 info: http://pulpito.ceph.com/mrfreezeex-2022-01-19_12:26:12-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6627174/ log: http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-01-19_12:26:12-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6627174/ sentry: https://sentry.ceph.com/organizations/ceph/?query=13ef945391ed405884cebbb5698ab33d [6627176] rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-bitmap policy/simple rbd-mirror/four-per-cluster supported-random-distro$/{ubuntu_latest} workloads/rbd-mirror-journal-workunit} ----------------------------------------------------------------- time: 00:09:58 info: http://pulpito.ceph.com/mrfreezeex-2022-01-19_12:26:12-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6627176/ log: http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-01-19_12:26:12-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6627176/ sentry: https://sentry.ceph.com/organizations/ceph/?query=5843429bc7cc4a288184cb061838a719 Command failed (workunit test rbd/rbd_mirror_journal.sh) on smithi161 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=d238de862197118eb4dfe9e422168d942c06f08c TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\' RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1 RBD_MIRROR_USE_RBD_MIRROR=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo ne.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_journal.sh' [6627179] rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-stupid supported-random-distro$/{centos_8} workloads/rbd-mirror-snapshot-workunit-fast-diff-multipeer} ----------------------------------------------------------------- time: 00:11:53 info: http://pulpito.ceph.com/mrfreezeex-2022-01-19_12:26:12-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6627179/ log: http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-01-19_12:26:12-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6627179/ sentry: https://sentry.ceph.com/organizations/ceph/?query=f0de2c4b78f24bc9bd9cae4d07f5b490 [6627185] rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-comp-lz4 supported-random-distro$/{ubuntu_latest} workloads/rbd-mirror-snapshot-workunit-minimum} ----------------------------------------------------------------- time: 00:13:03 info: http://pulpito.ceph.com/mrfreezeex-2022-01-19_12:26:12-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6627185/ log: http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-01-19_12:26:12-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6627185/ sentry: https://sentry.ceph.com/organizations/ceph/?query=9dbd80dd178742d6bec5e6c17c1a0dcb Command failed (workunit test rbd/rbd_mirror_snapshot.sh) on smithi116 with status 16: 'mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=d238de862197118eb4dfe9e422168d942c06f08c TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\' RBD_IMAGE_FEATURES=layering RBD_MIRROR_CONFIG_KEY=1 RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo ne.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_snapshot.sh' [6627188] rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-comp-snappy supported-random-distro$/{rhel_8} workloads/rbd-mirror-stress-workunit-min-compat-client-octopus} ----------------------------------------------------------------- time: 00:23:11 info: http://pulpito.ceph.com/mrfreezeex-2022-01-19_12:26:12-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6627188/ log: http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-01-19_12:26:12-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6627188/ sentry: https://sentry.ceph.com/organizations/ceph/?query=e332d5f91bcb48eaa718eb9b564f5d5f Command failed (workunit test rbd/rbd_mirror_stress.sh) on smithi154 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=d238de862197118eb4dfe9e422168d942c06f08c TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\' RBD_MIRROR_MIN_COMPAT_CLIENT=octopus RBD_MIRROR_USE_EXISTING_CLUSTER=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_ mirror_stress.sh' [6627189] rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-low-osd-mem-target policy/simple rbd-mirror/four-per-cluster supported-random-distro$/{rhel_8} workloads/rbd-mirror-journal-workunit-multipeer} ----------------------------------------------------------------- time: 00:21:13 info: http://pulpito.ceph.com/mrfreezeex-2022-01-19_12:26:12-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6627189/ log: http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-01-19_12:26:12-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6627189/ sentry: https://sentry.ceph.com/organizations/ceph/?query=cd89ea7d156d4a5ba0646bafcfae59bd [6627190] rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-comp-zlib supported-random-distro$/{centos_8} workloads/rbd-mirror-workunit-config-key} ----------------------------------------------------------------- time: 00:17:09 info: http://pulpito.ceph.com/mrfreezeex-2022-01-19_12:26:12-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6627190/ log: http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-01-19_12:26:12-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6627190/ sentry: https://sentry.ceph.com/organizations/ceph/?query=71c62a36687947b59cc34b17871e91bc Command failed (workunit test rbd/rbd_mirror_journal.sh) on smithi162 with status 2: 'mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=d238de862197118eb4dfe9e422168d942c06f08c TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\' RBD_MIRROR_CONFIG_KEY=1 RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo ne.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_journal.sh' [6627191] rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-stupid policy/none rbd-mirror/four-per-cluster supported-random-distro$/{centos_8} workloads/rbd-mirror-journal-workunit} ----------------------------------------------------------------- time: 00:14:15 info: http://pulpito.ceph.com/mrfreezeex-2022-01-19_12:26:12-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6627191/ log: http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-01-19_12:26:12-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6627191/ sentry: https://sentry.ceph.com/organizations/ceph/?query=d9c0777181a44d0a8a714dedc0fb18a5 Command failed (workunit test rbd/rbd_mirror_journal.sh) on smithi105 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=d238de862197118eb4dfe9e422168d942c06f08c TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\' RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1 RBD_MIRROR_USE_RBD_MIRROR=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo ne.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_journal.sh' [6627192] rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-comp-zstd supported-random-distro$/{centos_8.stream} workloads/rbd-mirror-workunit-min-compat-client-octopus} ----------------------------------------------------------------- time: 00:11:26 info: http://pulpito.ceph.com/mrfreezeex-2022-01-19_12:26:12-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6627192/ log: http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-01-19_12:26:12-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6627192/ sentry: https://sentry.ceph.com/organizations/ceph/?query=244c63d5c40a42a3b7c10472d9a123a5 Command failed (workunit test rbd/rbd_mirror_journal.sh) on smithi055 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=d238de862197118eb4dfe9e422168d942c06f08c TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\' RBD_MIRROR_MIN_COMPAT_CLIENT=octopus RBD_MIRROR_USE_EXISTING_CLUSTER=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_ mirror_journal.sh' [6627194] rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-hybrid supported-random-distro$/{centos_8} workloads/rbd-mirror-workunit-policy-none} ----------------------------------------------------------------- time: 00:17:42 info: http://pulpito.ceph.com/mrfreezeex-2022-01-19_12:26:12-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6627194/ log: http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-01-19_12:26:12-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6627194/ sentry: https://sentry.ceph.com/organizations/ceph/?query=db627cf273de47a894f709b6838caf98 Command failed (workunit test rbd/rbd_mirror_journal.sh) on smithi090 with status 2: 'mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=d238de862197118eb4dfe9e422168d942c06f08c TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\' RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubunt u/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_journal .sh' [6627202] rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-comp-zstd policy/none rbd-mirror/four-per-cluster supported-random-distro$/{centos_8.stream} workloads/rbd-mirror-journal-workunit-multipeer} ----------------------------------------------------------------- time: 00:10:08 info: http://pulpito.ceph.com/mrfreezeex-2022-01-19_12:26:12-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6627202/ log: http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-01-19_12:26:12-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6627202/ sentry: https://sentry.ceph.com/organizations/ceph/?query=2602aba26835456b861bfe284c154af9 [6627204] rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-hybrid policy/simple rbd-mirror/four-per-cluster supported-random-distro$/{ubuntu_latest} workloads/rbd-mirror-journal-workunit} ----------------------------------------------------------------- time: 00:09:58 info: http://pulpito.ceph.com/mrfreezeex-2022-01-19_12:26:12-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6627204/ log: http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-01-19_12:26:12-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6627204/ sentry: https://sentry.ceph.com/organizations/ceph/?query=a2f00bd3d8b247459cb8122629bb7ce0 Command failed (workunit test rbd/rbd_mirror_journal.sh) on smithi181 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=d238de862197118eb4dfe9e422168d942c06f08c TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\' RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1 RBD_MIRROR_USE_RBD_MIRROR=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo ne.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_journal.sh' [6627205] rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-comp-lz4 supported-random-distro$/{centos_8} workloads/rbd-mirror-snapshot-workunit-exclusive-lock} ----------------------------------------------------------------- time: 00:16:23 info: http://pulpito.ceph.com/mrfreezeex-2022-01-19_12:26:12-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6627205/ log: http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-01-19_12:26:12-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6627205/ sentry: https://sentry.ceph.com/organizations/ceph/?query=71944a7f0fc944debf7f70f5939d7b36 Command failed (workunit test rbd/rbd_mirror_snapshot.sh) on smithi087 with status 16: 'mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=d238de862197118eb4dfe9e422168d942c06f08c TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\' RBD_IMAGE_FEATURES=layering,exclusive-lock RBD_MIRROR_CONFIG_KEY=1 RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubunt u/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_snapsho t.sh' [6627206] rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-low-osd-mem-target policy/none rbd-mirror/four-per-cluster supported-random-distro$/{centos_8.stream} workloads/rbd-mirror-snapshot-stress-workunit-exclusive-lock} ----------------------------------------------------------------- time: 00:52:38 info: http://pulpito.ceph.com/mrfreezeex-2022-01-19_12:26:12-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6627206/ log: http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-01-19_12:26:12-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6627206/ "2022-01-19T17:58:33.126930+0000 mon.a (mon.0) 1574 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log [6627207] rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-comp-snappy supported-random-distro$/{ubuntu_latest} workloads/rbd-mirror-snapshot-workunit-fast-diff-multipeer} ----------------------------------------------------------------- time: 00:07:41 info: http://pulpito.ceph.com/mrfreezeex-2022-01-19_12:26:12-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6627207/ log: http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-01-19_12:26:12-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6627207/ sentry: https://sentry.ceph.com/organizations/ceph/?query=b97b7b10da0f425e8fcd3a698f749627 [6627212] rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-bitmap policy/simple rbd-mirror/four-per-cluster supported-random-distro$/{ubuntu_latest} workloads/rbd-mirror-fsx-workunit} ----------------------------------------------------------------- time: 00:33:51 info: http://pulpito.ceph.com/mrfreezeex-2022-01-19_12:26:12-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6627212/ log: http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-01-19_12:26:12-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6627212/ "2022-01-19T17:37:04.680043+0000 mon.a (mon.0) 201 : cluster [WRN] Health check failed: Degraded data redundancy: 2/1962 objects degraded (0.102%), 1 pg degraded (PG_DEGRADED)" in cluster log [6627216] rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-low-osd-mem-target supported-random-distro$/{centos_8.stream} workloads/rbd-mirror-stress-workunit-min-compat-client-octopus} ----------------------------------------------------------------- time: 00:11:01 info: http://pulpito.ceph.com/mrfreezeex-2022-01-19_12:26:12-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6627216/ log: http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-01-19_12:26:12-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6627216/ sentry: https://sentry.ceph.com/organizations/ceph/?query=dc0a328fd0a7421cb62a5f9b68a979d4 Command failed (workunit test rbd/rbd_mirror_stress.sh) on smithi198 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=d238de862197118eb4dfe9e422168d942c06f08c TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\' RBD_MIRROR_MIN_COMPAT_CLIENT=octopus RBD_MIRROR_USE_EXISTING_CLUSTER=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_ mirror_stress.sh' [6627217] rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-comp-snappy policy/simple rbd-mirror/four-per-cluster supported-random-distro$/{rhel_8} workloads/rbd-mirror-journal-workunit-multipeer} ----------------------------------------------------------------- time: 00:21:40 info: http://pulpito.ceph.com/mrfreezeex-2022-01-19_12:26:12-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6627217/ log: http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-01-19_12:26:12-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6627217/ sentry: https://sentry.ceph.com/organizations/ceph/?query=987b0d20328644a7bd1c39469da6e59b [6627218] rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-stupid supported-random-distro$/{centos_8.stream} workloads/rbd-mirror-workunit-config-key} ----------------------------------------------------------------- time: 00:23:36 info: http://pulpito.ceph.com/mrfreezeex-2022-01-19_12:26:12-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6627218/ log: http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-01-19_12:26:12-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6627218/ sentry: https://sentry.ceph.com/organizations/ceph/?query=1345f3ea7bb74f5aa52e6149eb5591ae Command failed (workunit test rbd/rbd_mirror_journal.sh) on smithi173 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=d238de862197118eb4dfe9e422168d942c06f08c TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\' RBD_MIRROR_CONFIG_KEY=1 RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo ne.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_journal.sh' [6627219] rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-comp-zlib policy/none rbd-mirror/four-per-cluster supported-random-distro$/{rhel_8} workloads/rbd-mirror-journal-workunit} ----------------------------------------------------------------- time: 00:25:17 info: http://pulpito.ceph.com/mrfreezeex-2022-01-19_12:26:12-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6627219/ log: http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-01-19_12:26:12-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6627219/ sentry: https://sentry.ceph.com/organizations/ceph/?query=d1f9524be52842e1951f0d3d99f60edc Command failed (workunit test rbd/rbd_mirror_journal.sh) on smithi070 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=d238de862197118eb4dfe9e422168d942c06f08c TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\' RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1 RBD_MIRROR_USE_RBD_MIRROR=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo ne.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_journal.sh' [6627220] rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} msgr-failures/few objectstore/filestore-xfs supported-random-distro$/{centos_8.stream} workloads/rbd-mirror-workunit-min-compat-client-octopus} ----------------------------------------------------------------- time: 00:11:23 info: http://pulpito.ceph.com/mrfreezeex-2022-01-19_12:26:12-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6627220/ log: http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-01-19_12:26:12-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6627220/ sentry: https://sentry.ceph.com/organizations/ceph/?query=bc87e508011a4a9e93724791ca228b0a Command failed (workunit test rbd/rbd_mirror_journal.sh) on smithi079 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=d238de862197118eb4dfe9e422168d942c06f08c TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\' RBD_MIRROR_MIN_COMPAT_CLIENT=octopus RBD_MIRROR_USE_EXISTING_CLUSTER=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_ mirror_journal.sh' [6627225] rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-comp-lz4 supported-random-distro$/{centos_8.stream} workloads/rbd-mirror-workunit-policy-simple} ----------------------------------------------------------------- time: 00:25:35 info: http://pulpito.ceph.com/mrfreezeex-2022-01-19_12:26:12-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6627225/ log: http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-01-19_12:26:12-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6627225/ sentry: https://sentry.ceph.com/organizations/ceph/?query=ecf1fe7a68664648846eb6422c6d7987 Command failed (workunit test rbd/rbd_mirror_journal.sh) on smithi163 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=d238de862197118eb4dfe9e422168d942c06f08c TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\' RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubunt u/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_journal .sh' [6627230] rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-bitmap policy/none rbd-mirror/four-per-cluster supported-random-distro$/{rhel_8} workloads/rbd-mirror-journal-workunit-multipeer} ----------------------------------------------------------------- time: 00:21:36 info: http://pulpito.ceph.com/mrfreezeex-2022-01-19_12:26:12-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6627230/ log: http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-01-19_12:26:12-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6627230/ sentry: https://sentry.ceph.com/organizations/ceph/?query=aead16d8b14b402db771774b1dff7807 [6627232] rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-comp-lz4 policy/simple rbd-mirror/four-per-cluster supported-random-distro$/{ubuntu_latest} workloads/rbd-mirror-journal-workunit} ----------------------------------------------------------------- time: 00:10:30 info: http://pulpito.ceph.com/mrfreezeex-2022-01-19_12:26:12-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6627232/ log: http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-01-19_12:26:12-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6627232/ sentry: https://sentry.ceph.com/organizations/ceph/?query=a180239c34014afd968c59859d1324f4 Command failed (workunit test rbd/rbd_mirror_journal.sh) on smithi111 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=d238de862197118eb4dfe9e422168d942c06f08c TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\' RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1 RBD_MIRROR_USE_RBD_MIRROR=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo ne.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_journal.sh' [6627235] rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-low-osd-mem-target supported-random-distro$/{centos_8} workloads/rbd-mirror-snapshot-workunit-fast-diff-multipeer} ----------------------------------------------------------------- time: 00:11:33 info: http://pulpito.ceph.com/mrfreezeex-2022-01-19_12:26:12-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6627235/ log: http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-01-19_12:26:12-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6627235/ sentry: https://sentry.ceph.com/organizations/ceph/?query=419ec222978747938e8bd9f36ba85bcd [6627244] rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-comp-lz4 supported-random-distro$/{ubuntu_latest} workloads/rbd-mirror-stress-workunit-min-compat-client-octopus} ----------------------------------------------------------------- time: 00:08:57 info: http://pulpito.ceph.com/mrfreezeex-2022-01-19_12:26:12-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6627244/ log: http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-01-19_12:26:12-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6627244/ sentry: https://sentry.ceph.com/organizations/ceph/?query=21ef8971f4334a7eae94ab5660730a93 Command failed (workunit test rbd/rbd_mirror_stress.sh) on smithi078 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=d238de862197118eb4dfe9e422168d942c06f08c TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\' RBD_MIRROR_MIN_COMPAT_CLIENT=octopus RBD_MIRROR_USE_EXISTING_CLUSTER=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_ mirror_stress.sh' [6627245] rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-stupid policy/simple rbd-mirror/four-per-cluster supported-random-distro$/{ubuntu_latest} workloads/rbd-mirror-journal-workunit-multipeer} ----------------------------------------------------------------- time: 00:07:38 info: http://pulpito.ceph.com/mrfreezeex-2022-01-19_12:26:12-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6627245/ log: http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-01-19_12:26:12-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6627245/ sentry: https://sentry.ceph.com/organizations/ceph/?query=329045614c694b239412a6d7cfab8542 [6627246] rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-comp-snappy supported-random-distro$/{rhel_8} workloads/rbd-mirror-workunit-config-key} ----------------------------------------------------------------- time: 00:28:01 info: http://pulpito.ceph.com/mrfreezeex-2022-01-19_12:26:12-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6627246/ log: http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-01-19_12:26:12-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6627246/ sentry: https://sentry.ceph.com/organizations/ceph/?query=78426d89158348d48bc18f775e3c6570 Command failed (workunit test rbd/rbd_mirror_journal.sh) on smithi196 with status 2: 'mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=d238de862197118eb4dfe9e422168d942c06f08c TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\' RBD_MIRROR_CONFIG_KEY=1 RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo ne.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_journal.sh' [6627247] rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} msgr-failures/few objectstore/filestore-xfs policy/none rbd-mirror/four-per-cluster supported-random-distro$/{centos_8.stream} workloads/rbd-mirror-journal-workunit} ----------------------------------------------------------------- time: 00:12:27 info: http://pulpito.ceph.com/mrfreezeex-2022-01-19_12:26:12-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6627247/ log: http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-01-19_12:26:12-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6627247/ sentry: https://sentry.ceph.com/organizations/ceph/?query=831c3d611f3a4f5598f9ead5cd511390 Command failed (workunit test rbd/rbd_mirror_journal.sh) on smithi114 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=d238de862197118eb4dfe9e422168d942c06f08c TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\' RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1 RBD_MIRROR_USE_RBD_MIRROR=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo ne.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_journal.sh' [6627248] rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-comp-zlib supported-random-distro$/{rhel_8} workloads/rbd-mirror-workunit-min-compat-client-octopus} ----------------------------------------------------------------- time: 00:24:32 info: http://pulpito.ceph.com/mrfreezeex-2022-01-19_12:26:12-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6627248/ log: http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-01-19_12:26:12-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6627248/ sentry: https://sentry.ceph.com/organizations/ceph/?query=0d8c7e4e616e46eb888d76e6b16c8f19 Command failed (workunit test rbd/rbd_mirror_journal.sh) on smithi057 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=d238de862197118eb4dfe9e422168d942c06f08c TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\' RBD_MIRROR_MIN_COMPAT_CLIENT=octopus RBD_MIRROR_USE_EXISTING_CLUSTER=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_ mirror_journal.sh' Pass ================================================================= [6627171] rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-low-osd-mem-target policy/none rbd-mirror/four-per-cluster supported-random-distro$/{ubuntu_latest} workloads/rbd-mirror-fsx-workunit} time: 00:49:19 info: http://pulpito.ceph.com/mrfreezeex-2022-01-19_12:26:12-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6627171/ [6627172] rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-stupid policy/simple rbd-mirror/four-per-cluster supported-random-distro$/{centos_8.stream} workloads/rbd-mirror-journal-stress-workunit} time: 00:57:54 info: http://pulpito.ceph.com/mrfreezeex-2022-01-19_12:26:12-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6627172/ [6627173] rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-comp-zstd supported-random-distro$/{centos_8.stream} workloads/rbd-mirror-journal-bootstrap-workunit} time: 00:11:23 info: http://pulpito.ceph.com/mrfreezeex-2022-01-19_12:26:12-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6627173/ [6627175] rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-hybrid supported-random-distro$/{ubuntu_latest} workloads/rbd-mirror-snapshot-bootstrap-workunit} time: 00:09:39 info: http://pulpito.ceph.com/mrfreezeex-2022-01-19_12:26:12-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6627175/ [6627177] rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-low-osd-mem-target supported-random-distro$/{ubuntu_latest} workloads/rbd-mirror-snapshot-workunit-exclusive-lock} time: 00:20:27 info: http://pulpito.ceph.com/mrfreezeex-2022-01-19_12:26:12-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6627177/ [6627178] rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-comp-lz4 policy/none rbd-mirror/four-per-cluster supported-random-distro$/{rhel_8} workloads/rbd-mirror-snapshot-stress-workunit-exclusive-lock} time: 01:04:30 info: http://pulpito.ceph.com/mrfreezeex-2022-01-19_12:26:12-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6627178/ [6627180] rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-comp-snappy policy/simple rbd-mirror/four-per-cluster supported-random-distro$/{ubuntu_latest} workloads/rbd-mirror-snapshot-stress-workunit-fast-diff} time: 00:53:50 info: http://pulpito.ceph.com/mrfreezeex-2022-01-19_12:26:12-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6627180/ [6627181] rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} msgr-failures/few objectstore/filestore-xfs supported-random-distro$/{ubuntu_latest} workloads/rbd-mirror-snapshot-workunit-fast-diff} time: 00:20:17 info: http://pulpito.ceph.com/mrfreezeex-2022-01-19_12:26:12-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6627181/ [6627182] rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-comp-zlib policy/none rbd-mirror/four-per-cluster supported-random-distro$/{ubuntu_latest} workloads/rbd-mirror-snapshot-stress-workunit-minimum} time: 00:49:06 info: http://pulpito.ceph.com/mrfreezeex-2022-01-19_12:26:12-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6627182/ [6627183] rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-bitmap supported-random-distro$/{centos_8} workloads/rbd-mirror-snapshot-workunit-journaling} time: 00:24:04 info: http://pulpito.ceph.com/mrfreezeex-2022-01-19_12:26:12-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6627183/ [6627184] rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-comp-zstd policy/simple rbd-mirror/four-per-cluster supported-random-distro$/{ubuntu_latest} workloads/rbd-mirror-fsx-workunit} time: 00:39:17 info: http://pulpito.ceph.com/mrfreezeex-2022-01-19_12:26:12-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6627184/ [6627186] rbd/singleton/{all/rbd_mirror objectstore/bluestore-stupid openstack supported-random-distro$/{centos_8}} time: 00:48:00 info: http://pulpito.ceph.com/mrfreezeex-2022-01-19_12:26:12-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6627186/ [6627187] rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-hybrid policy/none rbd-mirror/four-per-cluster supported-random-distro$/{ubuntu_latest} workloads/rbd-mirror-journal-stress-workunit} time: 00:55:03 info: http://pulpito.ceph.com/mrfreezeex-2022-01-19_12:26:12-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6627187/ [6627193] rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} msgr-failures/few objectstore/filestore-xfs policy/simple rbd-mirror/four-per-cluster supported-random-distro$/{rhel_8} workloads/rbd-mirror-snapshot-stress-workunit-exclusive-lock} time: 01:03:18 info: http://pulpito.ceph.com/mrfreezeex-2022-01-19_12:26:12-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6627193/ [6627195] rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-bitmap policy/none rbd-mirror/four-per-cluster supported-random-distro$/{rhel_8} workloads/rbd-mirror-snapshot-stress-workunit-fast-diff} time: 01:04:04 info: http://pulpito.ceph.com/mrfreezeex-2022-01-19_12:26:12-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6627195/ [6627196] rbd/valgrind/{base/install centos_latest clusters/{fixed-1 openstack} objectstore/bluestore-comp-zstd validator/memcheck workloads/rbd_mirror} time: 00:57:21 info: http://pulpito.ceph.com/mrfreezeex-2022-01-19_12:26:12-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6627196/ [6627197] rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-low-osd-mem-target supported-random-distro$/{rhel_8} workloads/rbd-mirror-workunit-policy-simple} time: 00:36:08 info: http://pulpito.ceph.com/mrfreezeex-2022-01-19_12:26:12-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6627197/ [6627198] rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-comp-lz4 policy/simple rbd-mirror/four-per-cluster supported-random-distro$/{centos_8} workloads/rbd-mirror-snapshot-stress-workunit-minimum} time: 00:53:16 info: http://pulpito.ceph.com/mrfreezeex-2022-01-19_12:26:12-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6627198/ [6627199] rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-comp-snappy policy/none rbd-mirror/four-per-cluster supported-random-distro$/{ubuntu_latest} workloads/rbd-mirror-fsx-workunit} time: 00:38:58 info: http://pulpito.ceph.com/mrfreezeex-2022-01-19_12:26:12-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6627199/ [6627200] rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-comp-zlib policy/simple rbd-mirror/four-per-cluster supported-random-distro$/{rhel_8} workloads/rbd-mirror-journal-stress-workunit} time: 01:09:23 info: http://pulpito.ceph.com/mrfreezeex-2022-01-19_12:26:12-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6627200/ [6627201] rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} msgr-failures/few objectstore/filestore-xfs supported-random-distro$/{centos_8} workloads/rbd-mirror-journal-bootstrap-workunit} time: 00:12:10 info: http://pulpito.ceph.com/mrfreezeex-2022-01-19_12:26:12-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6627201/ [6627203] rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-bitmap supported-random-distro$/{centos_8.stream} workloads/rbd-mirror-snapshot-bootstrap-workunit} time: 00:11:02 info: http://pulpito.ceph.com/mrfreezeex-2022-01-19_12:26:12-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6627203/ [6627208] rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-stupid policy/simple rbd-mirror/four-per-cluster supported-random-distro$/{ubuntu_latest} workloads/rbd-mirror-snapshot-stress-workunit-fast-diff} time: 00:52:55 info: http://pulpito.ceph.com/mrfreezeex-2022-01-19_12:26:12-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6627208/ [6627209] rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-comp-zlib supported-random-distro$/{centos_8.stream} workloads/rbd-mirror-snapshot-workunit-fast-diff} time: 00:22:52 info: http://pulpito.ceph.com/mrfreezeex-2022-01-19_12:26:12-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6627209/ [6627210] rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} msgr-failures/few objectstore/filestore-xfs policy/none rbd-mirror/four-per-cluster supported-random-distro$/{centos_8} workloads/rbd-mirror-snapshot-stress-workunit-minimum} time: 00:52:37 info: http://pulpito.ceph.com/mrfreezeex-2022-01-19_12:26:12-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6627210/ [6627211] rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-comp-zstd supported-random-distro$/{rhel_8} workloads/rbd-mirror-snapshot-workunit-journaling} time: 00:36:06 info: http://pulpito.ceph.com/mrfreezeex-2022-01-19_12:26:12-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6627211/ [6627213] rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-hybrid supported-random-distro$/{centos_8.stream} workloads/rbd-mirror-snapshot-workunit-minimum} time: 00:22:17 info: http://pulpito.ceph.com/mrfreezeex-2022-01-19_12:26:12-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6627213/ [6627214] rbd/singleton/{all/rbd_mirror objectstore/bluestore-low-osd-mem-target openstack supported-random-distro$/{ubuntu_latest}} time: 00:45:18 info: http://pulpito.ceph.com/mrfreezeex-2022-01-19_12:26:12-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6627214/ [6627215] rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-comp-lz4 policy/none rbd-mirror/four-per-cluster supported-random-distro$/{rhel_8} workloads/rbd-mirror-journal-stress-workunit} time: 01:06:54 info: http://pulpito.ceph.com/mrfreezeex-2022-01-19_12:26:12-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6627215/ [6627221] rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-comp-zstd policy/simple rbd-mirror/four-per-cluster supported-random-distro$/{ubuntu_latest} workloads/rbd-mirror-snapshot-stress-workunit-exclusive-lock} time: 00:52:11 info: http://pulpito.ceph.com/mrfreezeex-2022-01-19_12:26:12-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6627221/ [6627222] rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-bitmap supported-random-distro$/{rhel_8} workloads/rbd-mirror-workunit-policy-none} time: 00:36:54 info: http://pulpito.ceph.com/mrfreezeex-2022-01-19_12:26:12-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6627222/ [6627223] rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-hybrid policy/none rbd-mirror/four-per-cluster supported-random-distro$/{centos_8.stream} workloads/rbd-mirror-snapshot-stress-workunit-fast-diff} time: 00:54:05 info: http://pulpito.ceph.com/mrfreezeex-2022-01-19_12:26:12-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6627223/ [6627224] rbd/valgrind/{base/install centos_latest clusters/{fixed-1 openstack} objectstore/bluestore-comp-zlib validator/memcheck workloads/rbd_mirror} time: 01:02:15 info: http://pulpito.ceph.com/mrfreezeex-2022-01-19_12:26:12-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6627224/ [6627226] rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-low-osd-mem-target policy/simple rbd-mirror/four-per-cluster supported-random-distro$/{centos_8.stream} workloads/rbd-mirror-snapshot-stress-workunit-minimum} time: 00:51:13 info: http://pulpito.ceph.com/mrfreezeex-2022-01-19_12:26:12-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6627226/ [6627227] rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-stupid policy/none rbd-mirror/four-per-cluster supported-random-distro$/{rhel_8} workloads/rbd-mirror-fsx-workunit} time: 01:00:03 info: http://pulpito.ceph.com/mrfreezeex-2022-01-19_12:26:12-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6627227/ [6627228] rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} msgr-failures/few objectstore/filestore-xfs policy/simple rbd-mirror/four-per-cluster supported-random-distro$/{ubuntu_latest} workloads/rbd-mirror-journal-stress-workunit} time: 00:57:28 info: http://pulpito.ceph.com/mrfreezeex-2022-01-19_12:26:12-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6627228/ [6627229] rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-comp-zlib supported-random-distro$/{rhel_8} workloads/rbd-mirror-journal-bootstrap-workunit} time: 00:22:42 info: http://pulpito.ceph.com/mrfreezeex-2022-01-19_12:26:12-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6627229/ [6627231] rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-comp-zstd supported-random-distro$/{rhel_8} workloads/rbd-mirror-snapshot-bootstrap-workunit} time: 00:24:02 info: http://pulpito.ceph.com/mrfreezeex-2022-01-19_12:26:12-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6627231/ [6627233] rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-hybrid supported-random-distro$/{centos_8.stream} workloads/rbd-mirror-snapshot-workunit-exclusive-lock} time: 00:23:08 info: http://pulpito.ceph.com/mrfreezeex-2022-01-19_12:26:12-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6627233/ [6627234] rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-comp-snappy policy/none rbd-mirror/four-per-cluster supported-random-distro$/{centos_8.stream} workloads/rbd-mirror-snapshot-stress-workunit-exclusive-lock} time: 00:49:41 info: http://pulpito.ceph.com/mrfreezeex-2022-01-19_12:26:12-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6627234/ [6627236] rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-comp-zlib policy/simple rbd-mirror/four-per-cluster supported-random-distro$/{centos_8.stream} workloads/rbd-mirror-snapshot-stress-workunit-fast-diff} time: 00:52:49 info: http://pulpito.ceph.com/mrfreezeex-2022-01-19_12:26:12-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6627236/ [6627237] rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-stupid supported-random-distro$/{centos_8.stream} workloads/rbd-mirror-snapshot-workunit-fast-diff} time: 00:22:45 info: http://pulpito.ceph.com/mrfreezeex-2022-01-19_12:26:12-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6627237/ [6627238] rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-comp-zstd policy/none rbd-mirror/four-per-cluster supported-random-distro$/{centos_8.stream} workloads/rbd-mirror-snapshot-stress-workunit-minimum} time: 00:51:31 info: http://pulpito.ceph.com/mrfreezeex-2022-01-19_12:26:12-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6627238/ [6627239] rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} msgr-failures/few objectstore/filestore-xfs supported-random-distro$/{ubuntu_latest} workloads/rbd-mirror-snapshot-workunit-journaling} time: 00:20:41 info: http://pulpito.ceph.com/mrfreezeex-2022-01-19_12:26:12-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6627239/ [6627240] rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-hybrid policy/simple rbd-mirror/four-per-cluster supported-random-distro$/{ubuntu_latest} workloads/rbd-mirror-fsx-workunit} time: 00:40:47 info: http://pulpito.ceph.com/mrfreezeex-2022-01-19_12:26:12-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6627240/ [6627241] rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-bitmap supported-random-distro$/{ubuntu_latest} workloads/rbd-mirror-snapshot-workunit-minimum} time: 00:21:11 info: http://pulpito.ceph.com/mrfreezeex-2022-01-19_12:26:12-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6627241/ [6627242] rbd/singleton/{all/rbd_mirror objectstore/bluestore-hybrid openstack supported-random-distro$/{rhel_8}} time: 00:57:11 info: http://pulpito.ceph.com/mrfreezeex-2022-01-19_12:26:12-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6627242/ [6627243] rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-low-osd-mem-target policy/none rbd-mirror/four-per-cluster supported-random-distro$/{centos_8.stream} workloads/rbd-mirror-journal-stress-workunit} time: 00:56:50 info: http://pulpito.ceph.com/mrfreezeex-2022-01-19_12:26:12-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6627243/ [6627249] rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-bitmap policy/simple rbd-mirror/four-per-cluster supported-random-distro$/{rhel_8} workloads/rbd-mirror-snapshot-stress-workunit-exclusive-lock} time: 01:03:36 info: http://pulpito.ceph.com/mrfreezeex-2022-01-19_12:26:12-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6627249/ [6627250] rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-comp-zstd supported-random-distro$/{ubuntu_latest} workloads/rbd-mirror-workunit-policy-none} time: 00:22:31 info: http://pulpito.ceph.com/mrfreezeex-2022-01-19_12:26:12-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6627250/ [6627251] rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-comp-lz4 policy/none rbd-mirror/four-per-cluster supported-random-distro$/{centos_8.stream} workloads/rbd-mirror-snapshot-stress-workunit-fast-diff} time: 00:54:12 info: http://pulpito.ceph.com/mrfreezeex-2022-01-19_12:26:12-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6627251/ [6627252] rbd/valgrind/{base/install centos_latest clusters/{fixed-1 openstack} objectstore/bluestore-comp-snappy validator/memcheck workloads/rbd_mirror} time: 01:01:20 info: http://pulpito.ceph.com/mrfreezeex-2022-01-19_12:26:12-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6627252/ [6627253] rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-hybrid supported-random-distro$/{centos_8} workloads/rbd-mirror-workunit-policy-simple} time: 00:26:02 info: http://pulpito.ceph.com/mrfreezeex-2022-01-19_12:26:12-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6627253/ [6627254] rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-comp-snappy policy/simple rbd-mirror/four-per-cluster supported-random-distro$/{centos_8} workloads/rbd-mirror-snapshot-stress-workunit-minimum} time: 00:53:32 info: http://pulpito.ceph.com/mrfreezeex-2022-01-19_12:26:12-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6627254/