2021-12-17T03:21:05.024 INFO:root:teuthology version: 1.1.0-95a7d479 2021-12-17T03:21:05.026 INFO:teuthology.results:seed: '4325' 2021-12-17T17:14:12.128 INFO:root:teuthology version: 1.1.0-95a7d479 2021-12-17T17:14:12.164 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2021-12-17T18:20:17.871 INFO:teuthology.results:Tests finished! gathering results... 2021-12-17T18:20:37.937 INFO:scrape:Found 220 jobs 2021-12-17T18:20:47.735 INFO:scrape:Found 8 distinct failure reasons 2021-12-17T18:20:47.736 INFO:scrape:Failure: "2021-12-17T16:26:01.876167+0000 mon.a (mon.0) 317 : cluster [WRN] Health check failed: Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded (PG_DEGRADED)" in cluster log 2021-12-17T18:20:47.737 INFO:scrape:36 jobs 2021-12-17T18:20:47.737 INFO:scrape:suites intersection: ['conf'] 2021-12-17T18:20:47.738 INFO:scrape:suites union: ['clusters/3-node', 'clusters/fixed-3', 'conf', 'features/no-object-map', 'features/object-map', 'install/ceph', 'krbd/fsx/{ceph/ceph', 'krbd/rbd-nomount/{bluestore-bitmap', 'krbd/rbd/{bluestore-bitmap', 'krbd/singleton/{bluestore-bitmap', 'ms_mode$/{crc}', 'ms_mode$/{legacy}', 'ms_mode$/{secure}', 'ms_mode/crc', 'ms_mode/legacy', 'ms_mode/secure', 'msgr-failures/few', 'msgr-failures/many', 'objectstore/bluestore-bitmap', 'objectstore/filestore-xfs', 'randomized-striping-off}', 'randomized-striping-on}', 'striping/default/{msgr-failures/few', 'striping/default/{msgr-failures/many', 'striping/fancy/{msgr-failures/few', 'tasks/fsx-1-client}', 'tasks/fsx-3-client}', 'tasks/krbd_data_pool}', 'tasks/krbd_exclusive_option}', 'tasks/krbd_huge_osdmap}', 'tasks/krbd_latest_osdmap_on_map}', 'tasks/krbd_namespaces}', 'tasks/krbd_udev_enumerate}', 'tasks/krbd_udev_netlink_enobufs}', 'tasks/krbd_udev_symlinks}', 'tasks/rbd_concurrent}', 'tasks/rbd_fio}', 'tasks/rbd_huge_tickets}', 'tasks/rbd_image_read}', 'tasks/rbd_kernel}', 'tasks/rbd_map_unmap}', 'tasks/rbd_simple_big}', 'tasks/rbd_workunit_kernel_untar_build}', 'tasks/rbd_workunit_suites_dbench}', 'tasks/rbd_workunit_suites_ffsb}', 'tasks/rbd_workunit_suites_fsx}', 'tasks/rbd_workunit_trivial_sync}', 'tasks/rbd_xfstests}'] 2021-12-17T18:20:47.739 INFO:scrape: 2021-12-17T18:20:47.739 INFO:scrape:Failure: Command failed on smithi122 with status 128: 'cd /lib/firmware/updates && sudo git fetch origin && sudo git reset --hard origin/master' 2021-12-17T18:20:47.740 INFO:scrape:1 jobs: ['6566908'] 2021-12-17T18:20:47.740 INFO:scrape:suites: ['clusters/fixed-3', 'conf', 'krbd/rbd/{bluestore-bitmap', 'ms_mode/secure', 'msgr-failures/many', 'tasks/rbd_fio}'] 2021-12-17T18:20:47.741 INFO:scrape: 2021-12-17T18:20:47.741 INFO:scrape:Failure: Command failed on smithi175 with status 128: 'rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/ceph/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 1db238840efd3668e5fb3b829cf206334b1dbe4c' 2021-12-17T18:20:47.742 INFO:scrape:1 jobs: ['6566961'] 2021-12-17T18:20:47.742 INFO:scrape:suites: ['clusters/fixed-3', 'conf', 'install/ceph', 'krbd/rbd-nomount/{bluestore-bitmap', 'ms_mode/crc', 'msgr-failures/few', 'tasks/krbd_udev_netns}'] 2021-12-17T18:20:47.743 INFO:scrape: 2021-12-17T18:20:47.743 INFO:scrape:Failure: {'Failure object was': {'smithi058.front.sepia.ceph.com': {'msg': 'Failed to update apt cache: ', 'invocation': {'module_args': {'dpkg_options': 'force-confdef,force-confold', 'autoremove': False, 'force': False, 'force_apt_get': False, 'policy_rc_d': 'None', 'package': 'None', 'autoclean': False, 'install_recommends': 'None', 'purge': False, 'allow_unauthenticated': False, 'state': 'present', 'upgrade': 'None', 'update_cache': True, 'default_release': 'None', 'only_upgrade': False, 'deb': 'None', 'cache_valid_time': 0}}, '_ansible_no_log': False, 'attempts': 24, 'changed': False}}, 'Traceback (most recent call last)': 'File "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/callback_plugins/failure_log.py", line 44, in log_failure log.error(yaml.safe_dump(failure)) File "/home/teuthworker/src/git.ceph.com_git_teuthology_95a7d4799b562f3bbb5ec66107094963abd62fa1/virtualenv/lib/python3.6/site-packages/yaml/__init__.py", line 306, in safe_dump return dump_all([data], stream, Dumper=SafeDumper, **kwds) File "/home/teuthworker/src/git.ceph.com_git_teuthology_95a7d4799b562f3bbb5ec66107094963abd62fa1/virtualenv/lib/python3.6/site-packages/yaml/__init__.py", line 278, in dump_all dumper.represent(data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_95a7d4799b562f3bbb5ec66107094963abd62fa1/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 27, in represent node = self.represent_data(data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_95a7d4799b562f3bbb5ec66107094963abd62fa1/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 48, in represent_data node = self.yaml_representers[data_types[0]](self, data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_95a7d4799b562f3bbb5ec66107094963abd62fa1/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 207, in represent_dict return self.represent_mapping(\'tag:yaml.org,2002:map\', data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_95a7d4799b562f3bbb5ec66107094963abd62fa1/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 118, in represent_mapping node_value = self.represent_data(item_value) File "/home/teuthworker/src/git.ceph.com_git_teuthology_95a7d4799b562f3bbb5ec66107094963abd62fa1/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 48, in represent_data node = self.yaml_representers[data_types[0]](self, data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_95a7d4799b562f3bbb5ec66107094963abd62fa1/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 207, in represent_dict return self.represent_mapping(\'tag:yaml.org,2002:map\', data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_95a7d4799b562f3bbb5ec66107094963abd62fa1/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 117, in represent_mapping node_key = self.represent_data(item_key) File "/home/teuthworker/src/git.ceph.com_git_teuthology_95a7d4799b562f3bbb5ec66107094963abd62fa1/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 58, in represent_data node = self.yaml_representers[None](self, data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_95a7d4799b562f3bbb5ec66107094963abd62fa1/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 231, in represent_undefined raise RepresenterError("cannot represent an object", data)', 'yaml.representer.RepresenterError': "('cannot represent an object', '_ansible_no_log')"} 2021-12-17T18:20:47.744 INFO:scrape:2 jobs: ['6566985', '6567014'] 2021-12-17T18:20:47.744 INFO:scrape:suites intersection: ['clusters/fixed-3', 'conf', 'ms_mode/legacy'] 2021-12-17T18:20:47.745 INFO:scrape:suites union: ['clusters/fixed-3', 'conf', 'install/ceph', 'krbd/rbd-nomount/{bluestore-bitmap', 'krbd/rbd/{bluestore-bitmap', 'ms_mode/legacy', 'msgr-failures/few', 'msgr-failures/many', 'tasks/rbd_simple_big}', 'tasks/rbd_workunit_kernel_untar_build}'] 2021-12-17T18:20:47.745 INFO:scrape: 2021-12-17T18:20:47.748 INFO:scrape:Failure: Failed to fetch package version from https://shaman.ceph.com/api/search/?status=ready&project=kernel&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&ref=nightly_pre-single-major 2021-12-17T18:20:47.749 INFO:scrape:1 jobs: ['6566873'] 2021-12-17T18:20:47.749 INFO:scrape:suites: ['clusters/separate-client', 'conf', 'filestore-xfs', 'kernels/pre-single-major', 'krbd/unmap/{ceph/ceph', 'tasks/unmap}'] 2021-12-17T18:20:47.750 INFO:scrape: 2021-12-17T18:20:47.751 INFO:scrape:Timeout 3h running clone.client.0/qa/workunits/rbd/concurrent.sh 2021-12-17T18:20:47.751 INFO:scrape:2 jobs: ['6567002', '6566929'] 2021-12-17T18:20:47.752 INFO:scrape:suites intersection: ['clusters/fixed-3', 'conf', 'install/ceph', 'krbd/rbd-nomount/{bluestore-bitmap', 'msgr-failures/many', 'tasks/rbd_concurrent}'] 2021-12-17T18:20:47.752 INFO:scrape:suites union: ['clusters/fixed-3', 'conf', 'install/ceph', 'krbd/rbd-nomount/{bluestore-bitmap', 'ms_mode/crc', 'ms_mode/legacy', 'msgr-failures/many', 'tasks/rbd_concurrent}'] 2021-12-17T18:20:47.753 INFO:scrape: 2021-12-17T18:20:47.753 INFO:scrape:Failure: Command failed on smithi194 with status 1: "/usr/bin/sudo TESTDIR=/home/ubuntu/cephtest adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/bash /home/ubuntu/cephtest/run_xfstests.sh -c 1 -f ext4 -t /dev/rbd0 -s /dev/rbd1 -x /tmp/excludep_ahkfn7 -r -- '-g auto -g blockdev -x clone'" 2021-12-17T18:20:47.754 INFO:scrape:1 jobs: ['6566871'] 2021-12-17T18:20:47.754 INFO:scrape:suites: ['conf', 'krbd/singleton/{bluestore-bitmap', 'ms_mode$/{prefer-crc}', 'msgr-failures/few', 'tasks/rbd_xfstests}'] 2021-12-17T18:20:47.755 INFO:scrape: 2021-12-17T18:20:47.755 INFO:scrape:Failure: "2021-12-17T04:57:52.144585+0000 osd.3 (osd.3) 3 : cluster [WRN] slow request osd_op(client.4512.0:9726 2.2 2:5cc62631:::rbd_data.1183163d8235.0000000000000005:head [zero 3354624~4096] snapc 0=[] ondisk+write+known_if_redirected e24) initiated 2021-12-17T04:57:21.372768+0000 currently started" in cluster log 2021-12-17T18:20:47.756 INFO:scrape:1 jobs: ['6566973'] 2021-12-17T18:20:47.756 INFO:scrape:suites: ['clusters/fixed-3', 'conf', 'krbd/rbd/{bluestore-bitmap', 'ms_mode/secure', 'msgr-failures/many', 'tasks/rbd_workunit_suites_iozone}'] 2021-12-17T18:20:47.757 INFO:scrape: 2021-12-17T18:20:47.765 INFO:teuthology.results:Sending results to ceph-qa@ceph.io: Test Run: teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi ================================================================= info: http://pulpito.ceph.com/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/ failed: 43 dead: 2 running: 0 waiting: 0 queued: 0 passed: 175 Fail ================================================================= [6566871] krbd/singleton/{bluestore-bitmap conf ms_mode$/{prefer-crc} msgr-failures/few tasks/rbd_xfstests} ----------------------------------------------------------------- time: 04:42:58 info: http://pulpito.ceph.com/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6566871/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6566871/ sentry: https://sentry.ceph.com/organizations/ceph/?query=8ef7b97810ef415d929e785efd02999e Command failed on smithi194 with status 1: "/usr/bin/sudo TESTDIR=/home/ubuntu/cephtest adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/bash /home/ubuntu/cephtest/run_xfstests.sh -c 1 -f ext4 -t /dev/rbd0 -s /dev/rbd1 -x /tmp/excludep_ahkfn7 -r -- '-g auto -g blockdev -x clone'" [6566873] krbd/unmap/{ceph/ceph clusters/separate-client conf filestore-xfs kernels/pre-single-major tasks/unmap} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6566873/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6566873/ sentry: https://sentry.ceph.com/organizations/ceph/?query=8f09448b3d164371afd5e1e5f6254a40 Failed to fetch package version from https://shaman.ceph.com/api/search/?st atus=ready&project=kernel&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&re f=nightly_pre-single-major [6566875] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy msgr-failures/many tasks/krbd_exclusive_option} ----------------------------------------------------------------- time: 00:24:28 info: http://pulpito.ceph.com/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6566875/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6566875/ sentry: https://sentry.ceph.com/organizations/ceph/?query=be04ddd720174a5cbe5a242e78a0564c "2021-12-17T04:07:03.011987+0000 mon.a (mon.0) 257 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log [6566883] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/many tasks/krbd_namespaces} ----------------------------------------------------------------- time: 00:24:36 info: http://pulpito.ceph.com/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6566883/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6566883/ sentry: https://sentry.ceph.com/organizations/ceph/?query=20ea85ac076345eb9e7dc4a9630878b9 "2021-12-17T04:06:10.594756+0000 mon.a (mon.0) 204 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log [6566884] krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/crc msgr-failures/many tasks/rbd_workunit_suites_ffsb} ----------------------------------------------------------------- time: 00:19:54 info: http://pulpito.ceph.com/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6566884/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6566884/ sentry: https://sentry.ceph.com/organizations/ceph/?query=3bf4e1fddbfb4bacb44e4b90b8236714 "2021-12-17T03:54:05.590486+0000 mon.a (mon.0) 241 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log [6566890] krbd/fsx/{ceph/ceph clusters/3-node conf features/no-object-map ms_mode$/{legacy} objectstore/bluestore-bitmap striping/default/{msgr-failures/many randomized-striping-off} tasks/fsx-1-client} ----------------------------------------------------------------- time: 00:19:59 info: http://pulpito.ceph.com/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6566890/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6566890/ sentry: https://sentry.ceph.com/organizations/ceph/?query=00a66df29e9e4ab4acc60f7ae327d720 "2021-12-17T03:59:56.170915+0000 mon.a (mon.0) 270 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log [6566891] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc msgr-failures/many tasks/krbd_udev_symlinks} ----------------------------------------------------------------- time: 00:24:19 info: http://pulpito.ceph.com/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6566891/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6566891/ sentry: https://sentry.ceph.com/organizations/ceph/?query=23e2d22819424e0c8ffaf60e4b7f24f7 "2021-12-17T03:56:00.350049+0000 mon.a (mon.0) 181 : cluster [WRN] Health check failed: Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded (PG_DEGRADED)" in cluster log [6566893] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy msgr-failures/few tasks/rbd_concurrent} ----------------------------------------------------------------- time: 00:24:40 info: http://pulpito.ceph.com/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6566893/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6566893/ sentry: https://sentry.ceph.com/organizations/ceph/?query=f53e674f6b064003b706cb8e6bacbd64 "2021-12-17T04:06:53.620798+0000 mon.a (mon.0) 204 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log [6566894] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/many tasks/rbd_huge_tickets} ----------------------------------------------------------------- time: 00:24:16 info: http://pulpito.ceph.com/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6566894/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6566894/ sentry: https://sentry.ceph.com/organizations/ceph/?query=de2ead2dea8144c18f7c121f75ba5db1 "2021-12-17T04:06:43.387508+0000 mon.a (mon.0) 206 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log [6566903] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc msgr-failures/many tasks/rbd_map_unmap} ----------------------------------------------------------------- time: 00:20:06 info: http://pulpito.ceph.com/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6566903/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6566903/ sentry: https://sentry.ceph.com/organizations/ceph/?query=ab46c0a1e7c64cea9a0fc7d2be249341 "2021-12-17T03:54:44.663930+0000 mon.a (mon.0) 323 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log [6566908] krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/secure msgr-failures/many tasks/rbd_fio} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6566908/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6566908/ sentry: https://sentry.ceph.com/organizations/ceph/?query=5453a27f9e334f49a3d6dfd96811365e Command failed on smithi122 with status 128: 'cd /lib/firmware/updates && sudo git fetch origin && sudo git reset --hard origin/master' [6566909] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/many tasks/krbd_data_pool} ----------------------------------------------------------------- time: 00:23:24 info: http://pulpito.ceph.com/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6566909/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6566909/ sentry: https://sentry.ceph.com/organizations/ceph/?query=6d05e2b51f754d1e92d6010d7b3c1911 "2021-12-17T03:56:24.158765+0000 mon.a (mon.0) 180 : cluster [WRN] Health check failed: Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded (PG_DEGRADED)" in cluster log [6566914] krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/legacy msgr-failures/many tasks/rbd_workunit_suites_dbench} ----------------------------------------------------------------- time: 00:19:34 info: http://pulpito.ceph.com/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6566914/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6566914/ sentry: https://sentry.ceph.com/organizations/ceph/?query=c749982d8bea44adbcc9f74510dcf86a "2021-12-17T04:24:40.940324+0000 mon.a (mon.0) 260 : cluster [WRN] Health check failed: 2 osds down (OSD_DOWN)" in cluster log [6566929] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc msgr-failures/many tasks/rbd_concurrent} ----------------------------------------------------------------- time: 04:00:07 info: http://pulpito.ceph.com/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6566929/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6566929/ sentry: https://sentry.ceph.com/organizations/ceph/?query=26b0543f8f2f49b48764812e4f7d677a Command failed (workunit test rbd/concurrent.sh) on smithi172 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=1db238840efd3668e5fb3b829cf206334b1dbe4c TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/concurrent.sh' [6566939] krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/legacy msgr-failures/many tasks/rbd_workunit_trivial_sync} ----------------------------------------------------------------- time: 00:20:02 info: http://pulpito.ceph.com/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6566939/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6566939/ sentry: https://sentry.ceph.com/organizations/ceph/?query=75f4be006792488090be74ec7fb23dda "2021-12-17T04:29:53.708415+0000 mon.a (mon.0) 173 : cluster [WRN] Health check failed: Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded (PG_DEGRADED)" in cluster log [6566956] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc msgr-failures/many tasks/krbd_namespaces} ----------------------------------------------------------------- time: 00:19:44 info: http://pulpito.ceph.com/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6566956/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6566956/ sentry: https://sentry.ceph.com/organizations/ceph/?query=482a391613344e01b02947eccb232e6c "2021-12-17T04:52:41.075418+0000 mon.a (mon.0) 294 : cluster [WRN] Health check failed: 2 osds down (OSD_DOWN)" in cluster log [6566959] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/many tasks/krbd_udev_netlink_enobufs} ----------------------------------------------------------------- time: 00:19:44 info: http://pulpito.ceph.com/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6566959/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6566959/ sentry: https://sentry.ceph.com/organizations/ceph/?query=1b6cbb96122e4e678a33eb1e320fd260 "2021-12-17T04:45:08.446391+0000 mon.a (mon.0) 116 : cluster [WRN] Health check failed: Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded (PG_DEGRADED)" in cluster log [6566961] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc msgr-failures/few tasks/krbd_udev_netns} ----------------------------------------------------------------- time: 00:10:10 info: http://pulpito.ceph.com/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6566961/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6566961/ sentry: https://sentry.ceph.com/organizations/ceph/?query=231969958fbe4a39a105e04ad63ee8aa Command failed on smithi175 with status 128: 'rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/ceph/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 1db238840efd3668e5fb3b829cf206334b1dbe4c' [6566972] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/many tasks/rbd_kernel} ----------------------------------------------------------------- time: 00:19:34 info: http://pulpito.ceph.com/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6566972/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6566972/ sentry: https://sentry.ceph.com/organizations/ceph/?query=5792f0939868490eac737075cd961f83 "2021-12-17T05:05:58.247865+0000 mon.a (mon.0) 257 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log [6566973] krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/secure msgr-failures/many tasks/rbd_workunit_suites_iozone} ----------------------------------------------------------------- time: 00:20:16 info: http://pulpito.ceph.com/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6566973/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6566973/ "2021-12-17T04:57:52.144585+0000 osd.3 (osd.3) 3 : cluster [WRN] slow request osd_op(client.4512.0:9726 2.2 2:5cc62631:::rbd_data.1183163d8235.0000000000000005:head [zero 3354624~4096] snapc 0=[] ondisk+write+known_if_redirected e24) initiated 2021-12-17T04:57:21.372768+0000 currently started" in cluster log [6566980] krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/crc msgr-failures/many tasks/rbd_fio} ----------------------------------------------------------------- time: 00:19:46 info: http://pulpito.ceph.com/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6566980/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6566980/ sentry: https://sentry.ceph.com/organizations/ceph/?query=58902ffc0df7493c8611e189cf99f791 "2021-12-17T05:12:03.125655+0000 mon.a (mon.0) 228 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log [6566982] krbd/singleton/{bluestore-bitmap conf ms_mode$/{crc} msgr-failures/many tasks/rbd_xfstests} ----------------------------------------------------------------- time: 00:19:40 info: http://pulpito.ceph.com/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6566982/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6566982/ sentry: https://sentry.ceph.com/organizations/ceph/?query=f1fe506502da46588d7bd857e6cec863 "2021-12-17T05:16:21.127345+0000 mon.a (mon.0) 209 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log [6566988] krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/secure msgr-failures/many tasks/rbd_workunit_suites_dbench} ----------------------------------------------------------------- time: 00:20:03 info: http://pulpito.ceph.com/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6566988/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6566988/ sentry: https://sentry.ceph.com/organizations/ceph/?query=e2419c73c4e64b2a84e6226b2177d6d1 "2021-12-17T05:20:49.259288+0000 mon.a (mon.0) 199 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log [6566989] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy msgr-failures/many tasks/krbd_latest_osdmap_on_map} ----------------------------------------------------------------- time: 00:20:39 info: http://pulpito.ceph.com/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6566989/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6566989/ sentry: https://sentry.ceph.com/organizations/ceph/?query=e1e7b67da1e346308ad89356c8599d4a "2021-12-17T05:21:51.820710+0000 mon.a (mon.0) 264 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log [6566994] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc msgr-failures/many tasks/krbd_udev_enumerate} ----------------------------------------------------------------- time: 00:21:24 info: http://pulpito.ceph.com/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6566994/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6566994/ sentry: https://sentry.ceph.com/organizations/ceph/?query=1a8c5370e3d84d7a8ce13708dd8590e6 "2021-12-17T05:13:23.924206+0000 mon.a (mon.0) 131 : cluster [WRN] Health check failed: Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded (PG_DEGRADED)" in cluster log [6566999] krbd/fsx/{ceph/ceph clusters/3-node conf features/object-map ms_mode$/{legacy} objectstore/bluestore-bitmap striping/default/{msgr-failures/many randomized-striping-off} tasks/fsx-1-client} ----------------------------------------------------------------- time: 00:20:25 info: http://pulpito.ceph.com/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6566999/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6566999/ sentry: https://sentry.ceph.com/organizations/ceph/?query=cc483b56f73d4c708036fcd481663003 "2021-12-17T05:28:22.183456+0000 mon.a (mon.0) 342 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log [6567002] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy msgr-failures/many tasks/rbd_concurrent} ----------------------------------------------------------------- time: 03:33:15 info: http://pulpito.ceph.com/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6567002/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6567002/ sentry: https://sentry.ceph.com/organizations/ceph/?query=3e82df3c4e3d4a5ea437eff1627a91d8 Command failed (workunit test rbd/concurrent.sh) on smithi181 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=1db238840efd3668e5fb3b829cf206334b1dbe4c TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/concurrent.sh' [6567004] krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/crc msgr-failures/many tasks/rbd_workunit_suites_fsx} ----------------------------------------------------------------- time: 00:19:51 info: http://pulpito.ceph.com/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6567004/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6567004/ sentry: https://sentry.ceph.com/organizations/ceph/?query=2750bab2b9904b54a048e5547f447c4b "2021-12-17T05:28:41.992041+0000 mon.a (mon.0) 118 : cluster [WRN] Health check failed: Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded (PG_DEGRADED)" in cluster log [6567007] krbd/fsx/{ceph/ceph clusters/3-node conf features/no-object-map ms_mode$/{secure} objectstore/filestore-xfs striping/default/{msgr-failures/few randomized-striping-off} tasks/fsx-3-client} ----------------------------------------------------------------- time: 00:15:30 info: http://pulpito.ceph.com/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6567007/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6567007/ "2021-12-17T05:29:39.048497+0000 mon.a (mon.0) 121 : cluster [WRN] Health check failed: Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded (PG_DEGRADED)" in cluster log [6567021] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc msgr-failures/many tasks/krbd_exclusive_option} ----------------------------------------------------------------- time: 00:20:15 info: http://pulpito.ceph.com/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6567021/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6567021/ sentry: https://sentry.ceph.com/organizations/ceph/?query=9047e4c553ca4e53aea1c3332b9c3817 "2021-12-17T05:48:08.052106+0000 mon.a (mon.0) 292 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log [6567022] krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/crc msgr-failures/many tasks/rbd_workunit_kernel_untar_build} ----------------------------------------------------------------- time: 00:20:08 info: http://pulpito.ceph.com/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6567022/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6567022/ sentry: https://sentry.ceph.com/organizations/ceph/?query=3c3265ea12db4175b53715c62d0003eb "2021-12-17T05:48:39.568805+0000 mon.a (mon.0) 320 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log [6567024] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/many tasks/krbd_huge_osdmap} ----------------------------------------------------------------- time: 00:19:36 info: http://pulpito.ceph.com/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6567024/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6567024/ sentry: https://sentry.ceph.com/organizations/ceph/?query=3b50a88317f847a19b295f4d0d800764 "2021-12-17T16:06:11.477593+0000 mon.a (mon.0) 224 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log [6567032] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc msgr-failures/many tasks/krbd_udev_netlink_enobufs} ----------------------------------------------------------------- time: 00:19:40 info: http://pulpito.ceph.com/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6567032/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6567032/ sentry: https://sentry.ceph.com/organizations/ceph/?query=5397d2c2039d4167b6ef81c9827b7ed8 "2021-12-17T16:17:58.863047+0000 mon.a (mon.0) 141 : cluster [WRN] Health check failed: Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded (PG_DEGRADED)" in cluster log [6567036] krbd/fsx/{ceph/ceph clusters/3-node conf features/object-map ms_mode$/{secure} objectstore/filestore-xfs striping/default/{msgr-failures/few randomized-striping-off} tasks/fsx-1-client} ----------------------------------------------------------------- time: 00:22:08 info: http://pulpito.ceph.com/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6567036/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6567036/ "2021-12-17T16:26:01.876167+0000 mon.a (mon.0) 317 : cluster [WRN] Health check failed: Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded (PG_DEGRADED)" in cluster log [6567043] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/few tasks/rbd_image_read} ----------------------------------------------------------------- time: 00:19:23 info: http://pulpito.ceph.com/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6567043/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6567043/ sentry: https://sentry.ceph.com/organizations/ceph/?query=6a08f8060fec434385536d77ee2493e4 "2021-12-17T16:40:17.887188+0000 mon.a (mon.0) 113 : cluster [WRN] Health check failed: Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded (PG_DEGRADED)" in cluster log [6567060] krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/crc msgr-failures/many tasks/rbd_workunit_suites_dbench} ----------------------------------------------------------------- time: 00:19:40 info: http://pulpito.ceph.com/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6567060/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6567060/ sentry: https://sentry.ceph.com/organizations/ceph/?query=6019c71267b14d6ea93e2631e7713e36 "2021-12-17T17:11:44.574658+0000 mon.a (mon.0) 225 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log [6567061] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/many tasks/krbd_latest_osdmap_on_map} ----------------------------------------------------------------- time: 00:20:13 info: http://pulpito.ceph.com/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6567061/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6567061/ sentry: https://sentry.ceph.com/organizations/ceph/?query=79a748154f5742158a18e12dd10bbed0 "2021-12-17T17:13:10.336346+0000 mon.a (mon.0) 321 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log [6567072] krbd/fsx/{ceph/ceph clusters/3-node conf features/object-map ms_mode$/{secure} objectstore/filestore-xfs striping/fancy/{msgr-failures/few randomized-striping-on} tasks/fsx-1-client} ----------------------------------------------------------------- time: 00:23:13 info: http://pulpito.ceph.com/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6567072/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6567072/ "2021-12-17T17:17:01.696415+0000 mon.a (mon.0) 306 : cluster [WRN] Health check failed: Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded (PG_DEGRADED)" in cluster log [6567075] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/many tasks/rbd_concurrent} ----------------------------------------------------------------- time: 00:20:07 info: http://pulpito.ceph.com/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6567075/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6567075/ sentry: https://sentry.ceph.com/organizations/ceph/?query=f85ed10b3de244a6add557d5bd440397 "2021-12-17T17:28:52.529415+0000 mon.a (mon.0) 219 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log [6567077] krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/legacy msgr-failures/many tasks/rbd_workunit_suites_fsx} ----------------------------------------------------------------- time: 00:19:47 info: http://pulpito.ceph.com/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6567077/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6567077/ sentry: https://sentry.ceph.com/organizations/ceph/?query=d15c2c0b12304390bfcdfb15b7f797e7 "2021-12-17T17:37:27.579170+0000 mon.a (mon.0) 199 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log [6567078] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy msgr-failures/many tasks/rbd_image_read} ----------------------------------------------------------------- time: 00:19:29 info: http://pulpito.ceph.com/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6567078/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6567078/ sentry: https://sentry.ceph.com/organizations/ceph/?query=4310e1f743954181b170b2023988d749 "2021-12-17T17:40:16.365689+0000 mon.a (mon.0) 317 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log [6567081] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/few tasks/rbd_kernel} ----------------------------------------------------------------- time: 00:19:16 info: http://pulpito.ceph.com/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6567081/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6567081/ sentry: https://sentry.ceph.com/organizations/ceph/?query=0a6ea13ac6db450c8ff91ed5ea7ddc8a "2021-12-17T17:39:30.571766+0000 mon.a (mon.0) 284 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log [6567086] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/many tasks/rbd_simple_big} ----------------------------------------------------------------- time: 00:19:49 info: http://pulpito.ceph.com/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6567086/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6567086/ sentry: https://sentry.ceph.com/organizations/ceph/?query=6e236556c47141b39be5c88ee172e90d "2021-12-17T17:43:21.853080+0000 mon.a (mon.0) 371 : cluster [WRN] Health check failed: 3 osds down (OSD_DOWN)" in cluster log Dead ================================================================= [6566985] krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/legacy msgr-failures/few tasks/rbd_workunit_kernel_untar_build} ----------------------------------------------------------------- time: 00:04:54 info: http://pulpito.ceph.com/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6566985/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6566985/ sentry: https://sentry.ceph.com/organizations/ceph/?query=3c03b3889fe145e699b118c815dd2e49 {'Failure object was': {'smithi058.front.sepia.ceph.com': {'msg': 'Failed to update apt cache: ', 'invocation': {'module_args': {'dpkg_options': 'force-confdef,force-confold', 'autoremove': False, 'force': False, 'force_apt_get': False, 'policy_rc_d': 'None', 'package': 'None', 'autoclean': False, 'install_recommends': 'None', 'purge': False, 'allow_unauthenticated': False, 'state': 'present', 'upgrade': 'None', 'update_cache': True, 'default_release': 'None', 'only_upgrade': False, 'deb': 'None', 'cache_valid_time': 0}}, '_ansible_no_log': False, 'attempts': 24, 'changed': False}}, 'Traceback (most recent call last)': 'File "/home/teuthworker/src/git.ceph.com_git_ceph-cm- ansible_master/callback_plugins/failure_log.py", line 44, in log_failure log.error(yaml.safe_dump(failure)) File "/home/teuthworker/src/git.ceph.com _git_teuthology_95a7d4799b562f3bbb5ec66107094963abd62fa1/virtualenv/lib/pyt hon3.6/site-packages/yaml/__init__.py", line 306, in safe_dump return dump_all([data], stream, Dumper=SafeDumper, **kwds) File "/home/teuthworker /src/git.ceph.com_git_teuthology_95a7d4799b562f3bbb5ec66107094963abd62fa1/v irtualenv/lib/python3.6/site-packages/yaml/__init__.py", line 278, in dump_all dumper.represent(data) File "/home/teuthworker/src/git.ceph.com_gi t_teuthology_95a7d4799b562f3bbb5ec66107094963abd62fa1/virtualenv/lib/python 3.6/site-packages/yaml/representer.py", line 27, in represent node = self.represent_data(data) File "/home/teuthworker/src/git.ceph.com_git_teut hology_95a7d4799b562f3bbb5ec66107094963abd62fa1/virtualenv/lib/python3.6/si te-packages/yaml/representer.py", line 48, in represent_data node = self.yaml_representers[data_types[0]](self, data) File "/home/teuthworker/s rc/git.ceph.com_git_teuthology_95a7d4799b562f3bbb5ec66107094963abd62fa1/vir tualenv/lib/python3.6/site-packages/yaml/representer.py", line 207, in represent_dict return self.represent_mapping(\'tag:yaml.org,2002:map\', data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_95a7d4799b562 f3bbb5ec66107094963abd62fa1/virtualenv/lib/python3.6/site- packages/yaml/representer.py", line 118, in represent_mapping node_value = self.represent_data(item_value) File "/home/teuthworker/src/git.ceph.com_gi t_teuthology_95a7d4799b562f3bbb5ec66107094963abd62fa1/virtualenv/lib/python 3.6/site-packages/yaml/representer.py", line 48, in represent_data node = self.yaml_representers[data_types[0]](self, data) File "/home/teuthworker/s rc/git.ceph.com_git_teuthology_95a7d4799b562f3bbb5ec66107094963abd62fa1/vir tualenv/lib/python3.6/site-packages/yaml/representer.py", line 207, in represent_dict return self.represent_mapping(\'tag:yaml.org,2002:map\', data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_95a7d4799b562 f3bbb5ec66107094963abd62fa1/virtualenv/lib/python3.6/site- packages/yaml/representer.py", line 117, in represent_mapping node_key = self.represent_data(item_key) File "/home/teuthworker/src/git.ceph.com_git_ teuthology_95a7d4799b562f3bbb5ec66107094963abd62fa1/virtualenv/lib/python3. 6/site-packages/yaml/representer.py", line 58, in represent_data node = self.yaml_representers[None](self, data) File "/home/teuthworker/src/git.ce ph.com_git_teuthology_95a7d4799b562f3bbb5ec66107094963abd62fa1/virtualenv/l ib/python3.6/site-packages/yaml/representer.py", line 231, in represent_undefined raise RepresenterError("cannot represent an object", data)', 'yaml.representer.RepresenterError': "('cannot represent an object', '_ansible_no_log')"} [6567014] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy msgr-failures/many tasks/rbd_simple_big} ----------------------------------------------------------------- time: 00:04:44 info: http://pulpito.ceph.com/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6567014/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6567014/ sentry: https://sentry.ceph.com/organizations/ceph/?query=1a07591e79f346349e62867f9b99a633 {'Failure object was': {'smithi139.front.sepia.ceph.com': {'stdout': '', 'stderr': 'W: --force-yes is deprecated, use one of the options starting with --allow instead.\\nE: Could not get lock /var/lib/dpkg/lock-frontend. It is held by process 6150 (apt-get)\\nE: Unable to acquire the dpkg frontend lock (/var/lib/dpkg/lock-frontend), is another process using it?\\n', 'rc': 100, 'invocation': {'module_args': {'dpkg_options': 'force- confdef,force-confold', 'autoremove': False, 'force': True, 'force_apt_get': False, 'policy_rc_d': 'None', 'package': ['ceph', 'ceph- common', 'libcephfs1', 'radosgw', 'python-ceph', 'python-rados', 'python- cephfs', 'python-rbd', 'librbd1', 'librados2', 'ceph-fs-common-dbg', 'ceph- fs-common', 'openmpi-common'], 'autoclean': False, 'install_recommends': 'None', 'name': ['ceph', 'ceph-common', 'libcephfs1', 'radosgw', 'python- ceph', 'python-rados', 'python-cephfs', 'python-rbd', 'librbd1', 'librados2', 'ceph-fs-common-dbg', 'ceph-fs-common', 'openmpi-common'], 'purge': False, 'allow_unauthenticated': False, 'state': 'absent', 'upgrade': 'None', 'update_cache': 'None', 'default_release': 'None', 'only_upgrade': False, 'deb': 'None', 'cache_valid_time': 0}}, 'msg': "'apt-get remove 'librbd1' 'librados2'' failed: W: --force-yes is deprecated, use one of the options starting with --allow instead.\nE: Could not get lock /var/lib/dpkg/lock-frontend. It is held by process 6150 (apt- get)\nE: Unable to acquire the dpkg frontend lock (/var/lib/dpkg/lock- frontend), is another process using it?\n", 'stdout_lines': [], 'stderr_lines': ['W: --force-yes is deprecated, use one of the options starting with --allow instead.', 'E: Could not get lock /var/lib/dpkg/lock- frontend. It is held by process 6150 (apt-get)', 'E: Unable to acquire the dpkg frontend lock (/var/lib/dpkg/lock-frontend), is another process using it?'], '_ansible_no_log': False, 'changed': False}}, 'Traceback (most recent call last)': 'File "/home/teuthworker/src/git.ceph.com_git_ceph-cm- ansible_master/callback_plugins/failure_log.py", line 44, in log_failure log.error(yaml.safe_dump(failure)) File "/home/teuthworker/src/git.ceph.com _git_teuthology_95a7d4799b562f3bbb5ec66107094963abd62fa1/virtualenv/lib/pyt hon3.6/site-packages/yaml/__init__.py", line 306, in safe_dump return dump_all([data], stream, Dumper=SafeDumper, **kwds) File "/home/teuthworker /src/git.ceph.com_git_teuthology_95a7d4799b562f3bbb5ec66107094963abd62fa1/v irtualenv/lib/python3.6/site-packages/yaml/__init__.py", line 278, in dump_all dumper.represent(data) File "/home/teuthworker/src/git.ceph.com_gi t_teuthology_95a7d4799b562f3bbb5ec66107094963abd62fa1/virtualenv/lib/python 3.6/site-packages/yaml/representer.py", line 27, in represent node = self.represent_data(data) File "/home/teuthworker/src/git.ceph.com_git_teut hology_95a7d4799b562f3bbb5ec66107094963abd62fa1/virtualenv/lib/python3.6/si te-packages/yaml/representer.py", line 48, in represent_data node = self.yaml_representers[data_types[0]](self, data) File "/home/teuthworker/s rc/git.ceph.com_git_teuthology_95a7d4799b562f3bbb5ec66107094963abd62fa1/vir tualenv/lib/python3.6/site-packages/yaml/representer.py", line 207, in represent_dict return self.represent_mapping(\'tag:yaml.org,2002:map\', data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_95a7d4799b562 f3bbb5ec66107094963abd62fa1/virtualenv/lib/python3.6/site- packages/yaml/representer.py", line 118, in represent_mapping node_value = self.represent_data(item_value) File "/home/teuthworker/src/git.ceph.com_gi t_teuthology_95a7d4799b562f3bbb5ec66107094963abd62fa1/virtualenv/lib/python 3.6/site-packages/yaml/representer.py", line 48, in represent_data node = self.yaml_representers[data_types[0]](self, data) File "/home/teuthworker/s rc/git.ceph.com_git_teuthology_95a7d4799b562f3bbb5ec66107094963abd62fa1/vir tualenv/lib/python3.6/site-packages/yaml/representer.py", line 207, in represent_dict return self.represent_mapping(\'tag:yaml.org,2002:map\', data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_95a7d4799b562 f3bbb5ec66107094963abd62fa1/virtualenv/lib/python3.6/site- packages/yaml/representer.py", line 117, in represent_mapping node_key = self.represent_data(item_key) File "/home/teuthworker/src/git.ceph.com_git_ teuthology_95a7d4799b562f3bbb5ec66107094963abd62fa1/virtualenv/lib/python3. 6/site-packages/yaml/representer.py", line 58, in represent_data node = self.yaml_representers[None](self, data) File "/home/teuthworker/src/git.ce ph.com_git_teuthology_95a7d4799b562f3bbb5ec66107094963abd62fa1/virtualenv/l ib/python3.6/site-packages/yaml/representer.py", line 231, in represent_undefined raise RepresenterError("cannot represent an object", data)', 'yaml.representer.RepresenterError': "('cannot represent an object', 'invocation')"} Pass ================================================================= [6566867] krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/crc tasks/krbd_deep_flatten} time: 00:10:46 info: http://pulpito.ceph.com/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6566867/ [6566868] krbd/fsx/{ceph/ceph clusters/3-node conf features/no-object-map ms_mode$/{crc} objectstore/bluestore-bitmap striping/default/{msgr-failures/few randomized-striping-off} tasks/fsx-1-client} time: 00:20:12 info: http://pulpito.ceph.com/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6566868/ [6566869] krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/crc msgr-failures/few tasks/rbd_fio} time: 00:28:14 info: http://pulpito.ceph.com/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6566869/ [6566870] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc msgr-failures/few tasks/krbd_data_pool} time: 00:24:18 info: http://pulpito.ceph.com/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6566870/ [6566872] krbd/thrash/{bluestore-bitmap ceph/ceph clusters/fixed-3 conf ms_mode$/{crc} thrashers/backoff thrashosds-health workloads/rbd_fio} time: 00:40:05 info: http://pulpito.ceph.com/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6566872/ [6566874] krbd/wac/sysfs/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf tasks/stable_writes} time: 00:13:34 info: http://pulpito.ceph.com/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6566874/ [6566876] krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/legacy msgr-failures/many tasks/rbd_workunit_kernel_untar_build} time: 00:20:23 info: http://pulpito.ceph.com/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6566876/ [6566877] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/few tasks/krbd_fallocate} time: 00:16:58 info: http://pulpito.ceph.com/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6566877/ [6566878] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc msgr-failures/many tasks/krbd_huge_osdmap} time: 00:17:28 info: http://pulpito.ceph.com/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6566878/ [6566879] krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/secure msgr-failures/few tasks/rbd_workunit_suites_dbench} time: 00:38:24 info: http://pulpito.ceph.com/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6566879/ [6566880] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy msgr-failures/few tasks/krbd_latest_osdmap_on_map} time: 00:18:36 info: http://pulpito.ceph.com/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6566880/ [6566881] krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/legacy tasks/krbd_discard} time: 00:10:32 info: http://pulpito.ceph.com/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6566881/ [6566882] krbd/fsx/{ceph/ceph clusters/3-node conf features/object-map ms_mode$/{legacy} objectstore/filestore-xfs striping/fancy/{msgr-failures/few randomized-striping-on} tasks/fsx-3-client} time: 00:18:20 info: http://pulpito.ceph.com/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6566882/ [6566885] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc msgr-failures/few tasks/krbd_udev_enumerate} time: 00:37:42 info: http://pulpito.ceph.com/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6566885/ [6566886] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy msgr-failures/many tasks/krbd_udev_netlink_enobufs} time: 00:15:11 info: http://pulpito.ceph.com/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6566886/ [6566887] krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/legacy msgr-failures/few tasks/rbd_workunit_suites_fsstress} time: 00:16:54 info: http://pulpito.ceph.com/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6566887/ [6566888] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/few tasks/krbd_udev_netns} time: 00:16:13 info: http://pulpito.ceph.com/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6566888/ [6566889] krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/secure tasks/krbd_huge_image} time: 00:09:10 info: http://pulpito.ceph.com/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6566889/ [6566892] krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/secure msgr-failures/many tasks/rbd_workunit_suites_fsstress_ext4} time: 00:16:47 info: http://pulpito.ceph.com/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6566892/ [6566895] krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/crc msgr-failures/few tasks/rbd_workunit_suites_fsx} time: 00:23:50 info: http://pulpito.ceph.com/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6566895/ [6566896] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc msgr-failures/few tasks/rbd_image_read} time: 00:16:09 info: http://pulpito.ceph.com/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6566896/ [6566897] krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/crc tasks/krbd_modprobe} time: 00:09:49 info: http://pulpito.ceph.com/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6566897/ [6566898] krbd/fsx/{ceph/ceph clusters/3-node conf features/object-map ms_mode$/{legacy} objectstore/filestore-xfs striping/default/{msgr-failures/few randomized-striping-off} tasks/fsx-3-client} time: 00:20:33 info: http://pulpito.ceph.com/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6566898/ [6566899] krbd/thrash/{bluestore-bitmap ceph/ceph clusters/fixed-3 conf ms_mode$/{legacy} thrashers/mon-thrasher thrashosds-health workloads/rbd_workunit_suites_ffsb} time: 01:22:20 info: http://pulpito.ceph.com/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6566899/ [6566900] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy msgr-failures/many tasks/rbd_kernel} time: 00:15:21 info: http://pulpito.ceph.com/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6566900/ [6566901] krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/legacy msgr-failures/many tasks/rbd_workunit_suites_iozone} time: 00:22:44 info: http://pulpito.ceph.com/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6566901/ [6566902] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/few tasks/rbd_map_snapshot_io} time: 00:14:07 info: http://pulpito.ceph.com/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6566902/ [6566904] krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/secure msgr-failures/few tasks/rbd_workunit_trivial_sync} time: 00:17:18 info: http://pulpito.ceph.com/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6566904/ [6566905] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy msgr-failures/few tasks/rbd_simple_big} time: 00:26:25 info: http://pulpito.ceph.com/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6566905/ [6566906] krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/legacy tasks/krbd_msgr_segments} time: 00:13:48 info: http://pulpito.ceph.com/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6566906/ [6566907] krbd/fsx/{ceph/ceph clusters/3-node conf features/no-object-map ms_mode$/{legacy} objectstore/bluestore-bitmap striping/fancy/{msgr-failures/few randomized-striping-on} tasks/fsx-1-client} time: 00:27:50 info: http://pulpito.ceph.com/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6566907/ [6566910] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc msgr-failures/few tasks/krbd_exclusive_option} time: 00:14:15 info: http://pulpito.ceph.com/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6566910/ [6566911] krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/crc msgr-failures/few tasks/rbd_workunit_kernel_untar_build} time: 00:18:14 info: http://pulpito.ceph.com/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6566911/ [6566912] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy msgr-failures/many tasks/krbd_fallocate} time: 00:11:46 info: http://pulpito.ceph.com/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6566912/ [6566913] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/few tasks/krbd_huge_osdmap} time: 00:11:43 info: http://pulpito.ceph.com/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6566913/ [6566915] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc msgr-failures/many tasks/krbd_latest_osdmap_on_map} time: 00:14:49 info: http://pulpito.ceph.com/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6566915/ [6566916] krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/secure tasks/krbd_parent_overlap} time: 00:09:23 info: http://pulpito.ceph.com/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6566916/ [6566917] krbd/fsx/{ceph/ceph clusters/3-node conf features/object-map ms_mode$/{crc} objectstore/filestore-xfs striping/default/{msgr-failures/many randomized-striping-off} tasks/fsx-3-client} time: 00:18:26 info: http://pulpito.ceph.com/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6566917/ [6566918] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy msgr-failures/few tasks/krbd_namespaces} time: 00:11:35 info: http://pulpito.ceph.com/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6566918/ [6566919] krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/secure msgr-failures/few tasks/rbd_workunit_suites_ffsb} time: 01:16:19 info: http://pulpito.ceph.com/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6566919/ [6566920] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/many tasks/krbd_udev_enumerate} time: 00:40:15 info: http://pulpito.ceph.com/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6566920/ [6566921] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc msgr-failures/few tasks/krbd_udev_netlink_enobufs} time: 00:16:15 info: http://pulpito.ceph.com/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6566921/ [6566922] krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/crc msgr-failures/many tasks/rbd_workunit_suites_fsstress} time: 00:13:11 info: http://pulpito.ceph.com/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6566922/ [6566923] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy msgr-failures/many tasks/krbd_udev_netns} time: 00:12:29 info: http://pulpito.ceph.com/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6566923/ [6566924] krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/crc tasks/krbd_read_only} time: 00:10:56 info: http://pulpito.ceph.com/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6566924/ [6566925] krbd/fsx/{ceph/ceph clusters/3-node conf features/no-object-map ms_mode$/{prefer-crc} objectstore/filestore-xfs striping/default/{msgr-failures/few randomized-striping-off} tasks/fsx-1-client} time: 00:18:59 info: http://pulpito.ceph.com/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6566925/ [6566926] krbd/thrash/{bluestore-bitmap ceph/ceph clusters/fixed-3 conf ms_mode$/{prefer-crc} thrashers/pggrow thrashosds-health workloads/rbd_fio} time: 00:39:00 info: http://pulpito.ceph.com/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6566926/ [6566927] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/few tasks/krbd_udev_symlinks} time: 00:12:39 info: http://pulpito.ceph.com/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6566927/ [6566928] krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/legacy msgr-failures/few tasks/rbd_workunit_suites_fsstress_ext4} time: 00:11:47 info: http://pulpito.ceph.com/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6566928/ [6566930] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy msgr-failures/few tasks/rbd_huge_tickets} time: 00:12:11 info: http://pulpito.ceph.com/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6566930/ [6566931] krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/secure msgr-failures/many tasks/rbd_workunit_suites_fsx} time: 00:22:30 info: http://pulpito.ceph.com/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6566931/ [6566932] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/many tasks/rbd_image_read} time: 00:12:27 info: http://pulpito.ceph.com/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6566932/ [6566933] krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/legacy tasks/krbd_whole_object_zeroout} time: 00:10:56 info: http://pulpito.ceph.com/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6566933/ [6566934] krbd/fsx/{ceph/ceph clusters/3-node conf features/object-map ms_mode$/{legacy} objectstore/bluestore-bitmap striping/fancy/{msgr-failures/few randomized-striping-on} tasks/fsx-3-client} time: 00:20:34 info: http://pulpito.ceph.com/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6566934/ [6566935] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc msgr-failures/few tasks/rbd_kernel} time: 00:10:52 info: http://pulpito.ceph.com/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6566935/ [6566936] krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/crc msgr-failures/few tasks/rbd_workunit_suites_iozone} time: 00:16:30 info: http://pulpito.ceph.com/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6566936/ [6566937] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy msgr-failures/many tasks/rbd_map_snapshot_io} time: 00:11:03 info: http://pulpito.ceph.com/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6566937/ [6566938] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/few tasks/rbd_map_unmap} time: 00:15:45 info: http://pulpito.ceph.com/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6566938/ [6566940] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc msgr-failures/many tasks/rbd_simple_big} time: 00:27:31 info: http://pulpito.ceph.com/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6566940/ [6566941] krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/secure tasks/krbd_deep_flatten} time: 00:10:21 info: http://pulpito.ceph.com/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6566941/ [6566942] krbd/fsx/{ceph/ceph clusters/3-node conf features/no-object-map ms_mode$/{secure} objectstore/filestore-xfs striping/default/{msgr-failures/many randomized-striping-off} tasks/fsx-1-client} time: 00:21:36 info: http://pulpito.ceph.com/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6566942/ [6566943] krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/legacy msgr-failures/few tasks/rbd_fio} time: 00:29:03 info: http://pulpito.ceph.com/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6566943/ [6566944] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy msgr-failures/few tasks/krbd_data_pool} time: 00:19:18 info: http://pulpito.ceph.com/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6566944/ [6566945] krbd/unmap/{ceph/ceph clusters/separate-client conf filestore-xfs kernels/single-major-off tasks/unmap} time: 00:09:51 info: http://pulpito.ceph.com/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6566945/ [6566946] krbd/wac/wac/{bluestore-bitmap ceph/ceph clusters/fixed-3 conf tasks/wac verify/many-resets} time: 00:22:18 info: http://pulpito.ceph.com/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6566946/ [6566947] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/many tasks/krbd_exclusive_option} time: 00:12:36 info: http://pulpito.ceph.com/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6566947/ [6566948] krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/secure msgr-failures/many tasks/rbd_workunit_kernel_untar_build} time: 00:19:45 info: http://pulpito.ceph.com/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6566948/ [6566949] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc msgr-failures/few tasks/krbd_fallocate} time: 00:11:43 info: http://pulpito.ceph.com/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6566949/ [6566950] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy msgr-failures/many tasks/krbd_huge_osdmap} time: 00:11:45 info: http://pulpito.ceph.com/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6566950/ [6566951] krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/crc msgr-failures/few tasks/rbd_workunit_suites_dbench} time: 00:37:17 info: http://pulpito.ceph.com/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6566951/ [6566952] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/few tasks/krbd_latest_osdmap_on_map} time: 00:14:07 info: http://pulpito.ceph.com/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6566952/ [6566953] krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/crc tasks/krbd_discard} time: 00:09:54 info: http://pulpito.ceph.com/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6566953/ [6566954] krbd/fsx/{ceph/ceph clusters/3-node conf features/object-map ms_mode$/{prefer-crc} objectstore/bluestore-bitmap striping/default/{msgr-failures/few randomized-striping-off} tasks/fsx-3-client} time: 00:19:21 info: http://pulpito.ceph.com/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6566954/ [6566955] krbd/thrash/{bluestore-bitmap ceph/ceph clusters/fixed-3 conf ms_mode$/{crc} thrashers/upmap thrashosds-health workloads/rbd_workunit_suites_ffsb} time: 01:37:07 info: http://pulpito.ceph.com/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6566955/ [6566957] krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/legacy msgr-failures/many tasks/rbd_workunit_suites_ffsb} time: 01:23:27 info: http://pulpito.ceph.com/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6566957/ [6566958] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy msgr-failures/few tasks/krbd_udev_enumerate} time: 00:37:40 info: http://pulpito.ceph.com/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6566958/ [6566960] krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/secure msgr-failures/few tasks/rbd_workunit_suites_fsstress} time: 00:12:28 info: http://pulpito.ceph.com/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6566960/ [6566962] krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/legacy tasks/krbd_huge_image} time: 00:09:34 info: http://pulpito.ceph.com/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6566962/ [6566963] krbd/fsx/{ceph/ceph clusters/3-node conf features/no-object-map ms_mode$/{prefer-crc} objectstore/filestore-xfs striping/fancy/{msgr-failures/few randomized-striping-on} tasks/fsx-1-client} time: 00:21:42 info: http://pulpito.ceph.com/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6566963/ [6566964] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy msgr-failures/many tasks/krbd_udev_symlinks} time: 00:12:49 info: http://pulpito.ceph.com/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6566964/ [6566965] krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/crc msgr-failures/many tasks/rbd_workunit_suites_fsstress_ext4} time: 00:11:20 info: http://pulpito.ceph.com/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6566965/ [6566966] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/few tasks/rbd_concurrent} time: 00:15:42 info: http://pulpito.ceph.com/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6566966/ [6566967] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc msgr-failures/many tasks/rbd_huge_tickets} time: 00:10:59 info: http://pulpito.ceph.com/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6566967/ [6566968] krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/legacy msgr-failures/few tasks/rbd_workunit_suites_fsx} time: 00:19:22 info: http://pulpito.ceph.com/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6566968/ [6566969] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy msgr-failures/few tasks/rbd_image_read} time: 00:11:07 info: http://pulpito.ceph.com/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6566969/ [6566970] krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/secure tasks/krbd_modprobe} time: 00:09:03 info: http://pulpito.ceph.com/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6566970/ [6566971] krbd/fsx/{ceph/ceph clusters/3-node conf features/object-map ms_mode$/{secure} objectstore/bluestore-bitmap striping/default/{msgr-failures/many randomized-striping-off} tasks/fsx-3-client} time: 00:20:03 info: http://pulpito.ceph.com/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6566971/ [6566974] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc msgr-failures/few tasks/rbd_map_snapshot_io} time: 00:11:18 info: http://pulpito.ceph.com/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6566974/ [6566975] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy msgr-failures/many tasks/rbd_map_unmap} time: 00:16:20 info: http://pulpito.ceph.com/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6566975/ [6566976] krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/crc msgr-failures/few tasks/rbd_workunit_trivial_sync} time: 00:11:37 info: http://pulpito.ceph.com/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6566976/ [6566977] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/few tasks/rbd_simple_big} time: 00:27:44 info: http://pulpito.ceph.com/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6566977/ [6566978] krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/crc tasks/krbd_msgr_segments} time: 00:09:25 info: http://pulpito.ceph.com/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6566978/ [6566979] krbd/fsx/{ceph/ceph clusters/3-node conf features/object-map ms_mode$/{legacy} objectstore/bluestore-bitmap striping/default/{msgr-failures/few randomized-striping-off} tasks/fsx-1-client} time: 00:24:06 info: http://pulpito.ceph.com/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6566979/ [6566981] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc msgr-failures/many tasks/krbd_data_pool} time: 00:22:46 info: http://pulpito.ceph.com/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6566981/ [6566983] krbd/thrash/{bluestore-bitmap ceph/ceph clusters/fixed-3 conf ms_mode$/{prefer-crc} thrashers/backoff thrashosds-health workloads/rbd_workunit_suites_ffsb} time: 01:34:55 info: http://pulpito.ceph.com/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6566983/ [6566984] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy msgr-failures/few tasks/krbd_exclusive_option} time: 00:15:31 info: http://pulpito.ceph.com/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6566984/ [6566986] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/many tasks/krbd_fallocate} time: 00:15:21 info: http://pulpito.ceph.com/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6566986/ [6566987] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc msgr-failures/few tasks/krbd_huge_osdmap} time: 00:14:35 info: http://pulpito.ceph.com/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6566987/ [6566990] krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/legacy tasks/krbd_parent_overlap} time: 00:11:43 info: http://pulpito.ceph.com/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6566990/ [6566991] krbd/fsx/{ceph/ceph clusters/3-node conf features/no-object-map ms_mode$/{legacy} objectstore/filestore-xfs striping/fancy/{msgr-failures/few randomized-striping-on} tasks/fsx-3-client} time: 00:18:00 info: http://pulpito.ceph.com/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6566991/ [6566992] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/few tasks/krbd_namespaces} time: 00:14:48 info: http://pulpito.ceph.com/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6566992/ [6566993] krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/crc msgr-failures/few tasks/rbd_workunit_suites_ffsb} time: 00:55:23 info: http://pulpito.ceph.com/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6566993/ [6566995] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy msgr-failures/few tasks/krbd_udev_netlink_enobufs} time: 00:18:32 info: http://pulpito.ceph.com/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6566995/ [6566996] krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/legacy msgr-failures/many tasks/rbd_workunit_suites_fsstress} time: 00:15:10 info: http://pulpito.ceph.com/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6566996/ [6566997] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/many tasks/krbd_udev_netns} time: 00:13:20 info: http://pulpito.ceph.com/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6566997/ [6566998] krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/secure tasks/krbd_read_only} time: 00:11:09 info: http://pulpito.ceph.com/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6566998/ [6567000] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc msgr-failures/few tasks/krbd_udev_symlinks} time: 00:12:56 info: http://pulpito.ceph.com/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6567000/ [6567001] krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/secure msgr-failures/few tasks/rbd_workunit_suites_fsstress_ext4} time: 00:11:35 info: http://pulpito.ceph.com/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6567001/ [6567003] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/few tasks/rbd_huge_tickets} time: 00:11:14 info: http://pulpito.ceph.com/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6567003/ [6567005] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc msgr-failures/many tasks/rbd_image_read} time: 00:14:09 info: http://pulpito.ceph.com/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6567005/ [6567006] krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/crc tasks/krbd_whole_object_zeroout} time: 00:09:50 info: http://pulpito.ceph.com/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6567006/ [6567008] krbd/thrash/{bluestore-bitmap ceph/ceph clusters/fixed-3 conf ms_mode$/{crc} thrashers/mon-thrasher thrashosds-health workloads/rbd_fio} time: 00:37:48 info: http://pulpito.ceph.com/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6567008/ [6567009] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy msgr-failures/few tasks/rbd_kernel} time: 00:13:20 info: http://pulpito.ceph.com/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6567009/ [6567010] krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/legacy msgr-failures/few tasks/rbd_workunit_suites_iozone} time: 00:17:05 info: http://pulpito.ceph.com/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6567010/ [6567011] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/many tasks/rbd_map_snapshot_io} time: 00:12:43 info: http://pulpito.ceph.com/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6567011/ [6567012] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc msgr-failures/few tasks/rbd_map_unmap} time: 00:17:27 info: http://pulpito.ceph.com/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6567012/ [6567013] krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/secure msgr-failures/many tasks/rbd_workunit_trivial_sync} time: 00:13:04 info: http://pulpito.ceph.com/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6567013/ [6567015] krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/legacy tasks/krbd_deep_flatten} time: 00:11:12 info: http://pulpito.ceph.com/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6567015/ [6567016] krbd/fsx/{ceph/ceph clusters/3-node conf features/object-map ms_mode$/{legacy} objectstore/bluestore-bitmap striping/fancy/{msgr-failures/few randomized-striping-on} tasks/fsx-1-client} time: 00:28:00 info: http://pulpito.ceph.com/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6567016/ [6567017] krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/secure msgr-failures/few tasks/rbd_fio} time: 00:29:33 info: http://pulpito.ceph.com/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6567017/ [6567018] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/few tasks/krbd_data_pool} time: 00:21:01 info: http://pulpito.ceph.com/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6567018/ [6567019] krbd/unmap/{ceph/ceph clusters/separate-client conf filestore-xfs kernels/single-major-on tasks/unmap} time: 00:12:21 info: http://pulpito.ceph.com/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6567019/ [6567020] krbd/wac/wac/{bluestore-bitmap ceph/ceph clusters/fixed-3 conf tasks/wac verify/no-resets} time: 00:24:07 info: http://pulpito.ceph.com/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6567020/ [6567023] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy msgr-failures/few tasks/krbd_fallocate} time: 00:11:14 info: http://pulpito.ceph.com/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6567023/ [6567025] krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/legacy msgr-failures/few tasks/rbd_workunit_suites_dbench} time: 00:36:54 info: http://pulpito.ceph.com/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6567025/ [6567026] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc msgr-failures/few tasks/krbd_latest_osdmap_on_map} time: 00:14:07 info: http://pulpito.ceph.com/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6567026/ [6567027] krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/secure tasks/krbd_discard} time: 00:10:12 info: http://pulpito.ceph.com/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6567027/ [6567028] krbd/fsx/{ceph/ceph clusters/3-node conf features/no-object-map ms_mode$/{crc} objectstore/filestore-xfs striping/default/{msgr-failures/many randomized-striping-off} tasks/fsx-3-client} time: 00:16:30 info: http://pulpito.ceph.com/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6567028/ [6567029] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy msgr-failures/many tasks/krbd_namespaces} time: 00:12:41 info: http://pulpito.ceph.com/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6567029/ [6567030] krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/secure msgr-failures/many tasks/rbd_workunit_suites_ffsb} time: 01:24:32 info: http://pulpito.ceph.com/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6567030/ [6567031] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/few tasks/krbd_udev_enumerate} time: 00:37:16 info: http://pulpito.ceph.com/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6567031/ [6567033] krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/crc msgr-failures/few tasks/rbd_workunit_suites_fsstress} time: 00:12:13 info: http://pulpito.ceph.com/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6567033/ [6567034] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy msgr-failures/few tasks/krbd_udev_netns} time: 00:11:38 info: http://pulpito.ceph.com/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6567034/ [6567035] krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/crc tasks/krbd_huge_image} time: 00:09:34 info: http://pulpito.ceph.com/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6567035/ [6567037] krbd/thrash/{bluestore-bitmap ceph/ceph clusters/fixed-3 conf ms_mode$/{prefer-crc} thrashers/pggrow thrashosds-health workloads/rbd_workunit_suites_ffsb} time: 01:32:01 info: http://pulpito.ceph.com/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6567037/ [6567038] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/many tasks/krbd_udev_symlinks} time: 00:12:35 info: http://pulpito.ceph.com/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6567038/ [6567039] krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/legacy msgr-failures/many tasks/rbd_workunit_suites_fsstress_ext4} time: 00:11:21 info: http://pulpito.ceph.com/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6567039/ [6567040] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc msgr-failures/few tasks/rbd_concurrent} time: 00:15:45 info: http://pulpito.ceph.com/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6567040/ [6567041] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy msgr-failures/many tasks/rbd_huge_tickets} time: 00:11:10 info: http://pulpito.ceph.com/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6567041/ [6567042] krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/secure msgr-failures/few tasks/rbd_workunit_suites_fsx} time: 00:18:45 info: http://pulpito.ceph.com/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6567042/ [6567044] krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/legacy tasks/krbd_modprobe} time: 00:09:32 info: http://pulpito.ceph.com/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6567044/ [6567045] krbd/fsx/{ceph/ceph clusters/3-node conf features/no-object-map ms_mode$/{secure} objectstore/bluestore-bitmap striping/fancy/{msgr-failures/few randomized-striping-on} tasks/fsx-3-client} time: 00:19:27 info: http://pulpito.ceph.com/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6567045/ [6567046] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc msgr-failures/many tasks/rbd_kernel} time: 00:11:13 info: http://pulpito.ceph.com/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6567046/ [6567047] krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/crc msgr-failures/many tasks/rbd_workunit_suites_iozone} time: 00:18:33 info: http://pulpito.ceph.com/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6567047/ [6567048] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy msgr-failures/few tasks/rbd_map_snapshot_io} time: 00:10:32 info: http://pulpito.ceph.com/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6567048/ [6567049] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/many tasks/rbd_map_unmap} time: 00:15:41 info: http://pulpito.ceph.com/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6567049/ [6567050] krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/legacy msgr-failures/few tasks/rbd_workunit_trivial_sync} time: 00:11:52 info: http://pulpito.ceph.com/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6567050/ [6567051] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc msgr-failures/few tasks/rbd_simple_big} time: 00:26:25 info: http://pulpito.ceph.com/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6567051/ [6567052] krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/secure tasks/krbd_msgr_segments} time: 00:09:41 info: http://pulpito.ceph.com/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6567052/ [6567053] krbd/fsx/{ceph/ceph clusters/3-node conf features/object-map ms_mode$/{crc} objectstore/filestore-xfs striping/default/{msgr-failures/many randomized-striping-off} tasks/fsx-1-client} time: 00:25:50 info: http://pulpito.ceph.com/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6567053/ [6567054] krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/legacy msgr-failures/many tasks/rbd_fio} time: 00:27:52 info: http://pulpito.ceph.com/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6567054/ [6567055] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy msgr-failures/many tasks/krbd_data_pool} time: 00:23:23 info: http://pulpito.ceph.com/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6567055/ [6567056] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/few tasks/krbd_exclusive_option} time: 00:11:33 info: http://pulpito.ceph.com/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6567056/ [6567057] krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/secure msgr-failures/few tasks/rbd_workunit_kernel_untar_build} time: 00:18:37 info: http://pulpito.ceph.com/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6567057/ [6567058] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc msgr-failures/many tasks/krbd_fallocate} time: 00:12:09 info: http://pulpito.ceph.com/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6567058/ [6567059] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy msgr-failures/few tasks/krbd_huge_osdmap} time: 00:12:14 info: http://pulpito.ceph.com/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6567059/ [6567062] krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/crc tasks/krbd_parent_overlap} time: 00:09:56 info: http://pulpito.ceph.com/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6567062/ [6567063] krbd/fsx/{ceph/ceph clusters/3-node conf features/no-object-map ms_mode$/{prefer-crc} objectstore/bluestore-bitmap striping/default/{msgr-failures/few randomized-striping-off} tasks/fsx-3-client} time: 00:16:07 info: http://pulpito.ceph.com/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6567063/ [6567064] krbd/thrash/{bluestore-bitmap ceph/ceph clusters/fixed-3 conf ms_mode$/{legacy} thrashers/upmap thrashosds-health workloads/rbd_fio} time: 00:43:24 info: http://pulpito.ceph.com/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6567064/ [6567065] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc msgr-failures/few tasks/krbd_namespaces} time: 00:12:03 info: http://pulpito.ceph.com/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6567065/ [6567066] krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/legacy msgr-failures/few tasks/rbd_workunit_suites_ffsb} time: 01:20:21 info: http://pulpito.ceph.com/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6567066/ [6567067] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy msgr-failures/many tasks/krbd_udev_enumerate} time: 00:41:04 info: http://pulpito.ceph.com/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6567067/ [6567068] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/few tasks/krbd_udev_netlink_enobufs} time: 00:15:22 info: http://pulpito.ceph.com/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6567068/ [6567069] krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/secure msgr-failures/many tasks/rbd_workunit_suites_fsstress} time: 00:13:07 info: http://pulpito.ceph.com/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6567069/ [6567070] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc msgr-failures/many tasks/krbd_udev_netns} time: 00:11:47 info: http://pulpito.ceph.com/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6567070/ [6567071] krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/legacy tasks/krbd_read_only} time: 00:10:13 info: http://pulpito.ceph.com/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6567071/ [6567073] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy msgr-failures/few tasks/krbd_udev_symlinks} time: 00:12:21 info: http://pulpito.ceph.com/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6567073/ [6567074] krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/crc msgr-failures/few tasks/rbd_workunit_suites_fsstress_ext4} time: 00:11:54 info: http://pulpito.ceph.com/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6567074/ [6567076] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc msgr-failures/few tasks/rbd_huge_tickets} time: 00:10:39 info: http://pulpito.ceph.com/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6567076/ [6567079] krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/secure tasks/krbd_whole_object_zeroout} time: 00:10:12 info: http://pulpito.ceph.com/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6567079/ [6567080] krbd/fsx/{ceph/ceph clusters/3-node conf features/no-object-map ms_mode$/{prefer-crc} objectstore/bluestore-bitmap striping/default/{msgr-failures/many randomized-striping-off} tasks/fsx-3-client} time: 00:17:39 info: http://pulpito.ceph.com/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6567080/ [6567082] krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/secure msgr-failures/few tasks/rbd_workunit_suites_iozone} time: 00:16:41 info: http://pulpito.ceph.com/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6567082/ [6567083] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc msgr-failures/many tasks/rbd_map_snapshot_io} time: 00:10:54 info: http://pulpito.ceph.com/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6567083/ [6567084] krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy msgr-failures/few tasks/rbd_map_unmap} time: 00:15:46 info: http://pulpito.ceph.com/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6567084/ [6567085] krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/crc msgr-failures/many tasks/rbd_workunit_trivial_sync} time: 00:12:19 info: http://pulpito.ceph.com/teuthology-2021-12-17_03:20:02-krbd-master-testing-default-smithi/6567085/