2023-05-18T17:09:06.204 INFO:root:teuthology version: 0.0.1.dev98+g69caab8
2023-05-18T17:09:06.205 INFO:teuthology.results:subset: '36/99'
2023-05-18T17:09:06.206 INFO:teuthology.results:seed: '1898'
2023-05-18T17:23:06.961 INFO:root:teuthology version: 0.0.1.dev98+g69caab8
2023-05-18T17:23:06.964 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2023-05-19T05:24:00.940 WARNING:teuthology.results:test(s) did not finish before timeout of 43200 seconds
2023-05-19T05:24:02.845 INFO:scrape:Found 19 jobs
2023-05-19T05:24:03.713 INFO:scrape:Found 4 distinct failure reasons
2023-05-19T05:24:03.714 INFO:scrape:Failure: "1684431662.839964 mon.a (mon.0) 249 : cluster [WRN] Health check failed: Degraded data redundancy: 1/120 objects degraded (0.833%), 1 pg degraded (PG_DEGRADED)" in cluster log
2023-05-19T05:24:03.715 INFO:scrape:1 jobs: ['7277724']
2023-05-19T05:24:03.715 INFO:scrape:suites: ['clusters/{fixed-3', 'config/none', 'min-compat-client/default', 'msgr-failures/few', 'objectstore/bluestore-comp-snappy', 'openstack}', 'pool/small-cache-pool', 'rbd/librbd/{cache/writeback', 'supported-random-distro$/{ubuntu_latest}', 'workloads/c_api_tests_with_journaling}']
2023-05-19T05:24:03.715 INFO:scrape:
2023-05-19T05:24:03.715 INFO:scrape:Failure: Command failed on smithi103 with status 1: 'test -f /home/ubuntu/cephtest/archive/qemu/client.0/success'
2023-05-19T05:24:03.716 INFO:scrape:1 jobs: ['7277727']
2023-05-19T05:24:03.716 INFO:scrape:suites: ['2-cluster/{fix-2', '3-supported-random-distro$/{ubuntu_20.04}', '4-cache-path', '5-cache-mode/ssd', '6-cache-size/5G', '7-workloads/qemu_xfstests}', 'openstack}', 'rbd/pwl-cache/tmpfs/{1-base/install']
2023-05-19T05:24:03.716 INFO:scrape:
2023-05-19T05:24:03.717 INFO:scrape:Dead: b"2023-05-19T04:42:05.872 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ rbd --cluster cluster1 -p mirror --namespace '' ls"
2023-05-19T05:24:03.717 INFO:scrape:ceph version 18.0.0-3137-gb7effb87 (b7effb87b9c2fcd75d268d929fe6c286fb3c1c08) reef (dev)
 1: /lib64/libpthread.so.0(+0x12ce0) [0x7efe42846ce0]
 2: gsignal()
 3: abort()
 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0x197) [0x55d30e7cfce1]
 5: (KernelDevice::_aio_thread()+0x1798) [0x55d30f3dce38]
 6: (KernelDevice::AioCompletionThread::entry()+0x11) [0x55d30f3df4d1]
 7: /lib64/libpthread.so.0(+0x81cf) [0x7efe4283c1cf]
 8: clone()
2023-05-19T05:24:03.717 INFO:scrape:1 jobs: ['7277726']
2023-05-19T05:24:03.717 INFO:scrape:suites: ['clients/mirror', 'cluster/{2-node', 'msgr-failures/few', 'objectstore/bluestore-stupid', 'openstack}', 'policy/none', 'rbd-mirror/four-per-cluster', 'rbd/mirror-thrash/{base/install', 'supported-random-distro$/{rhel_8}', 'workloads/rbd-mirror-snapshot-stress-workunit-exclusive-lock}']
2023-05-19T05:24:03.718 INFO:scrape:
2023-05-19T05:24:03.718 INFO:scrape:Timeout 120 running ceph --cluster ceph pg dump --format=json
2023-05-19T05:24:03.718 INFO:scrape:1 jobs: ['7277711']
2023-05-19T05:24:03.719 INFO:scrape:suites: ['clusters/{fixed-3', 'features/defaults', 'msgr-failures/few', 'objectstore/bluestore-stupid', 'openstack}', 'pool/ec-cache-pool', 'rbd/encryption/{cache/none', 'supported-random-distro$/{centos_8}', 'workloads/qemu_xfstests_luks1_luks1}']
2023-05-19T05:24:03.719 INFO:scrape:
2023-05-19T05:24:03.720 INFO:teuthology.results:Sending results to idryomov@gmail.com: Test Run: dis-2023-05-18_15:55:03-rbd-wip-rbd-reef-backports-2-distro-default-smithi
=================================================================
info:    http://pulpito.ceph.com/dis-2023-05-18_15:55:03-rbd-wip-rbd-reef-backports-2-distro-default-smithi/
logs:    http://qa-proxy.ceph.com/teuthology/dis-2023-05-18_15:55:03-rbd-wip-rbd-reef-backports-2-distro-default-smithi/
failed:  3
dead:    0
running: 1
waiting: 0
queued:  0
passed:  15


Fail
=================================================================
[7277711]  rbd/encryption/{cache/none clusters/{fixed-3 openstack} features/defaults msgr-failures/few objectstore/bluestore-stupid pool/ec-cache-pool supported-random-distro$/{centos_8} workloads/qemu_xfstests_luks1_luks1}
-----------------------------------------------------------------
time:   02:10:22
info:   http://pulpito.ceph.com/dis-2023-05-18_15:55:03-rbd-wip-rbd-reef-backports-2-distro-default-smithi/7277711/
log:    http://qa-proxy.ceph.com/teuthology/dis-2023-05-18_15:55:03-rbd-wip-rbd-reef-backports-2-distro-default-smithi/7277711/

    Command failed on smithi053 with status 124: 'sudo adjust-ulimits ceph-
    coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster
    ceph pg dump --format=json'

[7277724]  rbd/librbd/{cache/writeback clusters/{fixed-3 openstack} config/none min-compat-client/default msgr-failures/few objectstore/bluestore-comp-snappy pool/small-cache-pool supported-random-distro$/{ubuntu_latest} workloads/c_api_tests_with_journaling}
-----------------------------------------------------------------
time:   00:35:08
info:   http://pulpito.ceph.com/dis-2023-05-18_15:55:03-rbd-wip-rbd-reef-backports-2-distro-default-smithi/7277724/
log:    http://qa-proxy.ceph.com/teuthology/dis-2023-05-18_15:55:03-rbd-wip-rbd-reef-backports-2-distro-default-smithi/7277724/

    "1684431662.839964 mon.a (mon.0) 249 : cluster [WRN] Health check failed:
    Degraded data redundancy: 1/120 objects degraded (0.833%), 1 pg degraded
    (PG_DEGRADED)" in cluster log

[7277727]  rbd/pwl-cache/tmpfs/{1-base/install 2-cluster/{fix-2 openstack} 3-supported-random-distro$/{ubuntu_20.04} 4-cache-path 5-cache-mode/ssd 6-cache-size/5G 7-workloads/qemu_xfstests}
-----------------------------------------------------------------
time:   01:14:25
info:   http://pulpito.ceph.com/dis-2023-05-18_15:55:03-rbd-wip-rbd-reef-backports-2-distro-default-smithi/7277727/
log:    http://qa-proxy.ceph.com/teuthology/dis-2023-05-18_15:55:03-rbd-wip-rbd-reef-backports-2-distro-default-smithi/7277727/

    Command failed on smithi103 with status 1: 'test -f
    /home/ubuntu/cephtest/archive/qemu/client.0/success'



Running
=================================================================
[7277726] rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-stupid policy/none rbd-mirror/four-per-cluster supported-random-distro$/{rhel_8} workloads/rbd-mirror-snapshot-stress-workunit-exclusive-lock}
info:   http://pulpito.ceph.com/dis-2023-05-18_15:55:03-rbd-wip-rbd-reef-backports-2-distro-default-smithi/7277726/



Pass
=================================================================
[7277712] rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-hybrid policy/none rbd-mirror/four-per-cluster supported-random-distro$/{rhel_8} workloads/rbd-mirror-journal-stress-workunit}
time:   01:03:10
info:   http://pulpito.ceph.com/dis-2023-05-18_15:55:03-rbd-wip-rbd-reef-backports-2-distro-default-smithi/7277712/

[7277713] rbd/immutable-object-cache/{clusters/{fix-2 openstack} pool/ceph_and_immutable_object_cache supported-random-distro$/{ubuntu_latest} workloads/c_api_tests_with_defaults}
time:   00:50:34
info:   http://pulpito.ceph.com/dis-2023-05-18_15:55:03-rbd-wip-rbd-reef-backports-2-distro-default-smithi/7277713/

[7277714] rbd/singleton-bluestore/{all/issue-20295 objectstore/bluestore-bitmap openstack supported-random-distro$/{rhel_8}}
time:   00:22:05
info:   http://pulpito.ceph.com/dis-2023-05-18_15:55:03-rbd-wip-rbd-reef-backports-2-distro-default-smithi/7277714/

[7277715] rbd/cli/{base/install clusters/{fixed-1 openstack} features/defaults msgr-failures/few objectstore/bluestore-comp-lz4 pool/ec-data-pool supported-random-distro$/{centos_8} workloads/rbd_cli_luks_encryption}
time:   00:13:14
info:   http://pulpito.ceph.com/dis-2023-05-18_15:55:03-rbd-wip-rbd-reef-backports-2-distro-default-smithi/7277715/

[7277716] rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-comp-zlib supported-random-distro$/{ubuntu_20.04} workloads/rbd-mirror-workunit-config-key}
time:   00:23:57
info:   http://pulpito.ceph.com/dis-2023-05-18_15:55:03-rbd-wip-rbd-reef-backports-2-distro-default-smithi/7277716/

[7277717] rbd/librbd/{cache/writeback clusters/{fixed-3 openstack} config/copy-on-read min-compat-client/default msgr-failures/few objectstore/bluestore-stupid pool/ec-data-pool supported-random-distro$/{ubuntu_20.04} workloads/python_api_tests_with_journaling}
time:   00:21:52
info:   http://pulpito.ceph.com/dis-2023-05-18_15:55:03-rbd-wip-rbd-reef-backports-2-distro-default-smithi/7277717/

[7277718] rbd/cli_v1/{base/install clusters/{fixed-1 openstack} features/format-1 msgr-failures/few objectstore/bluestore-comp-lz4 pool/none supported-random-distro$/{centos_8} workloads/rbd_cli_import_export}
time:   00:11:19
info:   http://pulpito.ceph.com/dis-2023-05-18_15:55:03-rbd-wip-rbd-reef-backports-2-distro-default-smithi/7277718/

[7277719] rbd/librbd/{cache/writethrough clusters/{fixed-3 openstack} config/none min-compat-client/octopus msgr-failures/few objectstore/bluestore-bitmap pool/none supported-random-distro$/{ubuntu_latest} workloads/rbd_fio}
time:   00:31:25
info:   http://pulpito.ceph.com/dis-2023-05-18_15:55:03-rbd-wip-rbd-reef-backports-2-distro-default-smithi/7277719/

[7277720] rbd/nbd/{base/install cluster/{fixed-3 openstack} msgr-failures/few objectstore/bluestore-comp-lz4 supported-random-distro$/{ubuntu_20.04} thrashers/default thrashosds-health workloads/rbd_nbd}
time:   00:12:50
info:   http://pulpito.ceph.com/dis-2023-05-18_15:55:03-rbd-wip-rbd-reef-backports-2-distro-default-smithi/7277720/

[7277721] rbd/librbd/{cache/writearound clusters/{fixed-3 openstack} config/copy-on-read min-compat-client/octopus msgr-failures/few objectstore/bluestore-comp-lz4 pool/replicated-data-pool supported-random-distro$/{centos_8} workloads/c_api_tests_with_defaults}
time:   00:49:02
info:   http://pulpito.ceph.com/dis-2023-05-18_15:55:03-rbd-wip-rbd-reef-backports-2-distro-default-smithi/7277721/

[7277722] rbd/thrash/{base/install clusters/{fixed-2 openstack} msgr-failures/few objectstore/bluestore-stupid supported-random-distro$/{ubuntu_latest} thrashers/cache thrashosds-health workloads/rbd_fsx_deep_copy}
time:   00:18:35
info:   http://pulpito.ceph.com/dis-2023-05-18_15:55:03-rbd-wip-rbd-reef-backports-2-distro-default-smithi/7277722/

[7277723] rbd/pwl-cache/tmpfs/{1-base/install 2-cluster/{fix-2 openstack} 3-supported-random-distro$/{centos_8} 4-cache-path 5-cache-mode/rwl 6-cache-size/1G 7-workloads/qemu_xfstests}
time:   01:00:32
info:   http://pulpito.ceph.com/dis-2023-05-18_15:55:03-rbd-wip-rbd-reef-backports-2-distro-default-smithi/7277723/

[7277725] rbd/maintenance/{base/install clusters/{fixed-3 openstack} objectstore/bluestore-low-osd-mem-target qemu/xfstests supported-random-distro$/{ubuntu_latest} workloads/dynamic_features}
time:   02:10:38
info:   http://pulpito.ceph.com/dis-2023-05-18_15:55:03-rbd-wip-rbd-reef-backports-2-distro-default-smithi/7277725/

[7277728] rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-hybrid policy/none rbd-mirror/four-per-cluster supported-random-distro$/{rhel_8} workloads/rbd-mirror-snapshot-stress-workunit-exclusive-lock}
time:   01:00:29
info:   http://pulpito.ceph.com/dis-2023-05-18_15:55:03-rbd-wip-rbd-reef-backports-2-distro-default-smithi/7277728/

[7277729] rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-low-osd-mem-target policy/simple rbd-mirror/four-per-cluster supported-random-distro$/{centos_8} workloads/rbd-mirror-snapshot-stress-workunit-fast-diff}
time:   00:55:03
info:   http://pulpito.ceph.com/dis-2023-05-18_15:55:03-rbd-wip-rbd-reef-backports-2-distro-default-smithi/7277729/