2022-12-23T11:37:33.000 INFO:root:teuthology version: 0.0.1.dev18+g19d18db 2022-12-23T11:37:33.000 INFO:teuthology.results:subset: '37/99' 2022-12-23T11:37:33.000 INFO:teuthology.results:seed: '5482' 2022-12-23T11:37:43.970 INFO:root:teuthology version: 0.0.1.dev18+g19d18db 2022-12-23T11:37:43.970 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2022-12-23T23:38:29.505 WARNING:teuthology.results:test(s) did not finish before timeout of 43200 seconds 2022-12-23T23:38:33.624 INFO:scrape:Found 9 jobs 2022-12-23T23:38:39.562 INFO:scrape:Found 3 distinct failure reasons 2022-12-23T23:38:39.563 INFO:scrape:Failure: Cannot connect to remote host smithi098 2022-12-23T23:38:39.563 INFO:scrape:6 jobs: ['7126307', '7126311', '7126305', '7126308', '7126310', '7126306'] 2022-12-23T23:38:39.563 INFO:scrape:suites intersection: ['clusters/{fixed-3', 'min-compat-client/octopus', 'msgr-failures/few', 'openstack}', 'pool/small-cache-pool', 'rbd/librbd/{cache/writeback', 'workloads/rbd_fio}'] 2022-12-23T23:38:39.563 INFO:scrape:suites union: ['clusters/{fixed-3', 'config/copy-on-read', 'config/none', 'config/permit-partial-discard', 'min-compat-client/octopus', 'msgr-failures/few', 'objectstore/bluestore-comp-lz4', 'objectstore/bluestore-comp-snappy', 'objectstore/bluestore-comp-zlib', 'objectstore/bluestore-comp-zstd', 'objectstore/bluestore-hybrid', 'objectstore/bluestore-low-osd-mem-target', 'openstack}', 'pool/small-cache-pool', 'rbd/librbd/{cache/writeback', 'supported-random-distro$/{centos_8}', 'supported-random-distro$/{rhel_8}', 'supported-random-distro$/{ubuntu_latest}', 'workloads/rbd_fio}'] 2022-12-23T23:38:39.563 INFO:scrape: 2022-12-23T23:38:39.563 INFO:scrape:Dead: b'2022-12-23T14:41:01.026 INFO:tasks.daemonwatchdog.daemon_watchdog:BARK! unmounting mounts and killing all daemons' 2022-12-23T23:38:39.564 INFO:scrape:1 jobs: ['7126304'] 2022-12-23T23:38:39.564 INFO:scrape:suites: ['clusters/{fixed-3', 'config/permit-partial-discard', 'min-compat-client/octopus', 'msgr-failures/few', 'objectstore/bluestore-stupid', 'openstack}', 'pool/small-cache-pool', 'rbd/librbd/{cache/writeback', 'supported-random-distro$/{ubuntu_latest}', 'workloads/rbd_fio}'] 2022-12-23T23:38:39.564 INFO:scrape: 2022-12-23T23:38:39.564 INFO:scrape:Timeout 120 running ceph --cluster cluster2 pg dump --format=json 2022-12-23T23:38:39.564 INFO:scrape:1 jobs: ['7126309'] 2022-12-23T23:38:39.564 INFO:scrape:suites: ['clients/mirror', 'cluster/{2-node', 'msgr-failures/few', 'objectstore/bluestore-stupid', 'openstack}', 'policy/simple', 'rbd-mirror/four-per-cluster', 'rbd/mirror-thrash/{base/install', 'supported-random-distro$/{ubuntu_latest}', 'workloads/rbd-mirror-snapshot-stress-workunit-exclusive-lock}'] 2022-12-23T23:38:39.564 INFO:scrape: 2022-12-23T23:38:39.565 INFO:teuthology.results:Sending results to idryomov@gmail.com: Test Run: dis-2022-12-23_11:37:27-rbd-wip-dis-testing-distro-default-smithi ================================================================= info: http://pulpito.ceph.com/dis-2022-12-23_11:37:27-rbd-wip-dis-testing-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/dis-2022-12-23_11:37:27-rbd-wip-dis-testing-distro-default-smithi/ failed: 7 dead: 0 running: 1 waiting: 0 queued: 0 passed: 1 Fail ================================================================= [7126305] rbd/librbd/{cache/writeback clusters/{fixed-3 openstack} config/none min-compat-client/octopus msgr-failures/few objectstore/bluestore-low-osd-mem-target pool/small-cache-pool supported-random-distro$/{rhel_8} workloads/rbd_fio} ----------------------------------------------------------------- time: 02:51:23 info: http://pulpito.ceph.com/dis-2022-12-23_11:37:27-rbd-wip-dis-testing-distro-default-smithi/7126305/ log: http://qa-proxy.ceph.com/teuthology/dis-2022-12-23_11:37:27-rbd-wip-dis-testing-distro-default-smithi/7126305/ sentry: https://sentry.ceph.com/organizations/ceph/?query=ee1f8586281e4abbb8ffa7c260d49521 Cannot connect to remote host smithi191 [7126306] rbd/librbd/{cache/writeback clusters/{fixed-3 openstack} config/copy-on-read min-compat-client/octopus msgr-failures/few objectstore/bluestore-hybrid pool/small-cache-pool supported-random-distro$/{centos_8} workloads/rbd_fio} ----------------------------------------------------------------- time: 02:50:13 info: http://pulpito.ceph.com/dis-2022-12-23_11:37:27-rbd-wip-dis-testing-distro-default-smithi/7126306/ log: http://qa-proxy.ceph.com/teuthology/dis-2022-12-23_11:37:27-rbd-wip-dis-testing-distro-default-smithi/7126306/ sentry: https://sentry.ceph.com/organizations/ceph/?query=1edf4b0044644683a56f5121895de41f Cannot connect to remote host smithi053 [7126307] rbd/librbd/{cache/writeback clusters/{fixed-3 openstack} config/permit-partial-discard min-compat-client/octopus msgr-failures/few objectstore/bluestore-comp-zstd pool/small-cache-pool supported-random-distro$/{centos_8} workloads/rbd_fio} ----------------------------------------------------------------- time: 02:50:17 info: http://pulpito.ceph.com/dis-2022-12-23_11:37:27-rbd-wip-dis-testing-distro-default-smithi/7126307/ log: http://qa-proxy.ceph.com/teuthology/dis-2022-12-23_11:37:27-rbd-wip-dis-testing-distro-default-smithi/7126307/ sentry: https://sentry.ceph.com/organizations/ceph/?query=34478546924e4f8686f91fd32eec664f Cannot connect to remote host smithi098 [7126308] rbd/librbd/{cache/writeback clusters/{fixed-3 openstack} config/none min-compat-client/octopus msgr-failures/few objectstore/bluestore-comp-zlib pool/small-cache-pool supported-random-distro$/{rhel_8} workloads/rbd_fio} ----------------------------------------------------------------- time: 02:50:09 info: http://pulpito.ceph.com/dis-2022-12-23_11:37:27-rbd-wip-dis-testing-distro-default-smithi/7126308/ log: http://qa-proxy.ceph.com/teuthology/dis-2022-12-23_11:37:27-rbd-wip-dis-testing-distro-default-smithi/7126308/ sentry: https://sentry.ceph.com/organizations/ceph/?query=f311ed8e99a948c6a3c21f9ace3817d5 Cannot connect to remote host smithi158 [7126309] rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-stupid policy/simple rbd-mirror/four-per-cluster supported-random-distro$/{ubuntu_latest} workloads/rbd-mirror-snapshot-stress-workunit-exclusive-lock} ----------------------------------------------------------------- time: 00:51:53 info: http://pulpito.ceph.com/dis-2022-12-23_11:37:27-rbd-wip-dis-testing-distro-default-smithi/7126309/ log: http://qa-proxy.ceph.com/teuthology/dis-2022-12-23_11:37:27-rbd-wip-dis-testing-distro-default-smithi/7126309/ Command failed on smithi201 with status 124: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json' [7126310] rbd/librbd/{cache/writeback clusters/{fixed-3 openstack} config/copy-on-read min-compat-client/octopus msgr-failures/few objectstore/bluestore-comp-snappy pool/small-cache-pool supported-random-distro$/{rhel_8} workloads/rbd_fio} ----------------------------------------------------------------- time: 02:51:48 info: http://pulpito.ceph.com/dis-2022-12-23_11:37:27-rbd-wip-dis-testing-distro-default-smithi/7126310/ log: http://qa-proxy.ceph.com/teuthology/dis-2022-12-23_11:37:27-rbd-wip-dis-testing-distro-default-smithi/7126310/ sentry: https://sentry.ceph.com/organizations/ceph/?query=a45175668f3344b59bb236245b47e0a9 Cannot connect to remote host smithi115 [7126311] rbd/librbd/{cache/writeback clusters/{fixed-3 openstack} config/permit-partial-discard min-compat-client/octopus msgr-failures/few objectstore/bluestore-comp-lz4 pool/small-cache-pool supported-random-distro$/{ubuntu_latest} workloads/rbd_fio} ----------------------------------------------------------------- time: 04:07:08 info: http://pulpito.ceph.com/dis-2022-12-23_11:37:27-rbd-wip-dis-testing-distro-default-smithi/7126311/ log: http://qa-proxy.ceph.com/teuthology/dis-2022-12-23_11:37:27-rbd-wip-dis-testing-distro-default-smithi/7126311/ sentry: https://sentry.ceph.com/organizations/ceph/?query=34eef4c8c7c942b89990fa5720947fc9 Cannot connect to remote host smithi109 Running ================================================================= [7126304] rbd/librbd/{cache/writeback clusters/{fixed-3 openstack} config/permit-partial-discard min-compat-client/octopus msgr-failures/few objectstore/bluestore-stupid pool/small-cache-pool supported-random-distro$/{ubuntu_latest} workloads/rbd_fio} info: http://pulpito.ceph.com/dis-2022-12-23_11:37:27-rbd-wip-dis-testing-distro-default-smithi/7126304/ Pass ================================================================= [7126303] rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} msgr-failures/few objectstore/filestore-xfs policy/simple rbd-mirror/four-per-cluster supported-random-distro$/{ubuntu_latest} workloads/rbd-mirror-journal-stress-workunit} time: 00:57:53 info: http://pulpito.ceph.com/dis-2022-12-23_11:37:27-rbd-wip-dis-testing-distro-default-smithi/7126303/