2023-12-09T23:55:57.989 INFO:root:teuthology version: 0.0.1.dev214+gf2de108
2023-12-09T23:55:57.989 INFO:teuthology.results:subset: '17/99'
2023-12-09T23:55:57.989 INFO:teuthology.results:seed: '2777'
2023-12-09T23:56:51.889 INFO:root:teuthology version: 0.0.1.dev214+gf2de108
2023-12-09T23:56:51.893 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2023-12-10T11:57:43.096 WARNING:teuthology.results:test(s) did not finish before timeout of 43200 seconds
2023-12-10T11:57:44.970 INFO:scrape:Found 23 jobs
2023-12-10T11:57:45.716 INFO:scrape:Found 4 distinct failure reasons
2023-12-10T11:57:45.717 INFO:scrape:Failure: failed to install new kernel version within timeout
2023-12-10T11:57:45.717 INFO:scrape:1 jobs: ['7485459']
2023-12-10T11:57:45.717 INFO:scrape:suites: ['clusters/{fixed-3', 'conf/{disable-pool-app}', 'data-pool/none', 'extra-conf/permit-partial-discard', 'min-compat-client/default', 'msgr-failures/few', 'objectstore/bluestore-stupid', 'openstack}', 'rbd/librbd/{cache/writeback', 'supported-random-distro$/{centos_latest}', 'workloads/c_api_tests_with_journaling}']
2023-12-10T11:57:45.717 INFO:scrape:
2023-12-10T11:57:45.717 INFO:scrape:Failure: Command failed on smithi040 with status 1: 'sudo yum install -y kernel'
2023-12-10T11:57:45.717 INFO:scrape:3 jobs: ['7485460', '7485461', '7485467']
2023-12-10T11:57:45.717 INFO:scrape:suites intersection: ['openstack}']
2023-12-10T11:57:45.718 INFO:scrape:suites union: ['2-cluster/{fix-2', '3-supported-random-distro$/{centos_latest}', '4-cache-path', '5-cache-mode/ssd', '6-cache-size/5G', '7-workloads/qemu_xfstests', 'clusters/{fixed-1', 'conf/{disable-pool-app}', 'conf/{disable-pool-app}}', 'features/format-1', 'msgr-failures/few', 'objectstore/bluestore-comp-lz4', 'objectstore/bluestore-low-osd-mem-target', 'openstack}', 'rbd/basic/{base/install', 'rbd/cli_v1/{base/install', 'rbd/pwl-cache/tmpfs/{1-base/install', 'supported-random-distro$/{centos_latest}', 'tasks/rbd_api_tests_old_format}', 'workloads/rbd_cli_import_export}']
2023-12-10T11:57:45.718 INFO:scrape:
2023-12-10T11:57:45.718 INFO:scrape:Failure: Command failed (workunit test rbd/test_librbd.sh) on smithi097 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=86ca2652261a3a50668690509bf073959d0391bd TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 RBD_FEATURES=125 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/test_librbd.sh'
2023-12-10T11:57:45.718 INFO:scrape:1 jobs: ['7485465']
2023-12-10T11:57:45.718 INFO:scrape:suites: ['clusters/{fixed-3', 'conf/{disable-pool-app}', 'data-pool/ec', 'extra-conf/none', 'min-compat-client/default', 'msgr-failures/few', 'objectstore/bluestore-low-osd-mem-target', 'openstack}', 'rbd/librbd/{cache/writeback', 'supported-random-distro$/{centos_latest}', 'workloads/c_api_tests_with_journaling}']
2023-12-10T11:57:45.718 INFO:scrape:
2023-12-10T11:57:45.718 INFO:scrape:Dead: b'2023-12-10T11:56:54.150 INFO:tasks.workunit.cluster1.client.mirror.smithi137.stderr:+ rbd --cluster cluster2 -p mirror_parent --namespace ns1 ls -l'
2023-12-10T11:57:45.718 INFO:scrape:ceph version 19.0.0-114-g86ca2652 (86ca2652261a3a50668690509bf073959d0391bd) reef (dev)
 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7fac37ab8520]
 2: pthread_kill()
 3: raise()
 4: abort()
 5: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0x17b) [0x559c40b90805]
 6: (KernelDevice::_aio_thread()+0x9bc) [0x559c414787fc]
 7: ceph-osd(+0x15b40e1) [0x559c4145b0e1]
 8: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7fac37b0ab43]
 9: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7fac37b9ca00]
2023-12-10T11:57:45.718 INFO:scrape:1 jobs: ['7485463']
2023-12-10T11:57:45.719 INFO:scrape:suites: ['clients/mirror', 'cluster/{2-node', 'conf/{disable-pool-app}', 'msgr-failures/few', 'objectstore/bluestore-stupid', 'openstack}', 'policy/none', 'rbd-mirror/four-per-cluster', 'rbd/mirror-thrash/{base/install', 'supported-random-distro$/{ubuntu_latest}', 'workloads/rbd-mirror-snapshot-stress-workunit-minimum}']
2023-12-10T11:57:45.719 INFO:scrape:
2023-12-10T11:57:45.720 INFO:teuthology.results:Sending results to idryomov@gmail.com: Test Run: dis-2023-12-09_23:55:51-rbd-wip-dis-testing-distro-default-smithi
=================================================================
info:    https://pulpito.ceph.com/dis-2023-12-09_23:55:51-rbd-wip-dis-testing-distro-default-smithi/
logs:    http://qa-proxy.ceph.com/teuthology/dis-2023-12-09_23:55:51-rbd-wip-dis-testing-distro-default-smithi/
failed:  5
dead:    0
running: 1
waiting: 0
queued:  0
passed:  17


Fail
=================================================================
[7485459]  rbd/librbd/{cache/writeback clusters/{fixed-3 openstack} conf/{disable-pool-app} data-pool/none extra-conf/permit-partial-discard min-compat-client/default msgr-failures/few objectstore/bluestore-stupid supported-random-distro$/{centos_latest} workloads/c_api_tests_with_journaling}
-----------------------------------------------------------------
time:   00:00:00
info:   https://pulpito.ceph.com/dis-2023-12-09_23:55:51-rbd-wip-dis-testing-distro-default-smithi/7485459/
log:    http://qa-proxy.ceph.com/teuthology/dis-2023-12-09_23:55:51-rbd-wip-dis-testing-distro-default-smithi/7485459/
sentry: https://sentry.ceph.com/organizations/ceph/?query=8469848a49a144faa6ea90de4255c2e4

    failed to install new kernel version within timeout

[7485460]  rbd/basic/{base/install clusters/{fixed-1 openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-low-osd-mem-target supported-random-distro$/{centos_latest} tasks/rbd_api_tests_old_format}
-----------------------------------------------------------------
time:   00:00:00
info:   https://pulpito.ceph.com/dis-2023-12-09_23:55:51-rbd-wip-dis-testing-distro-default-smithi/7485460/
log:    http://qa-proxy.ceph.com/teuthology/dis-2023-12-09_23:55:51-rbd-wip-dis-testing-distro-default-smithi/7485460/
sentry: https://sentry.ceph.com/organizations/ceph/?query=dbe82a5d1d3647b9a5f1a3c6c79ee266

    Command failed on smithi040 with status 1: 'sudo yum install -y kernel'

[7485461]  rbd/cli_v1/{base/install clusters/{fixed-1 openstack} conf/{disable-pool-app} features/format-1 msgr-failures/few objectstore/bluestore-comp-lz4 supported-random-distro$/{centos_latest} workloads/rbd_cli_import_export}
-----------------------------------------------------------------
time:   00:00:00
info:   https://pulpito.ceph.com/dis-2023-12-09_23:55:51-rbd-wip-dis-testing-distro-default-smithi/7485461/
log:    http://qa-proxy.ceph.com/teuthology/dis-2023-12-09_23:55:51-rbd-wip-dis-testing-distro-default-smithi/7485461/
sentry: https://sentry.ceph.com/organizations/ceph/?query=90b302ab180e486fa37d7fdbf01dcc44

    Command failed on smithi022 with status 1: 'sudo yum install -y kernel'

[7485465]  rbd/librbd/{cache/writeback clusters/{fixed-3 openstack} conf/{disable-pool-app} data-pool/ec extra-conf/none min-compat-client/default msgr-failures/few objectstore/bluestore-low-osd-mem-target supported-random-distro$/{centos_latest} workloads/c_api_tests_with_journaling}
-----------------------------------------------------------------
time:   00:33:37
info:   https://pulpito.ceph.com/dis-2023-12-09_23:55:51-rbd-wip-dis-testing-distro-default-smithi/7485465/
log:    http://qa-proxy.ceph.com/teuthology/dis-2023-12-09_23:55:51-rbd-wip-dis-testing-distro-default-smithi/7485465/
sentry: https://sentry.ceph.com/organizations/ceph/?query=e5b24a0fa0974ca9a480c23b40cc21b9

    Command failed (workunit test rbd/test_librbd.sh) on smithi097 with status
    1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=86ca2652261a3a50668690509bf073959d0391bd
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 RBD_FEATURES=125 adjust-ulimits ceph-
    coverage /home/ubuntu/cephtest/archive/coverage timeout 3h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/test_librbd.sh'

[7485467]  rbd/pwl-cache/tmpfs/{1-base/install 2-cluster/{fix-2 openstack} 3-supported-random-distro$/{centos_latest} 4-cache-path 5-cache-mode/ssd 6-cache-size/5G 7-workloads/qemu_xfstests conf/{disable-pool-app}}
-----------------------------------------------------------------
time:   00:59:53
info:   https://pulpito.ceph.com/dis-2023-12-09_23:55:51-rbd-wip-dis-testing-distro-default-smithi/7485467/
log:    http://qa-proxy.ceph.com/teuthology/dis-2023-12-09_23:55:51-rbd-wip-dis-testing-distro-default-smithi/7485467/

    Command failed on smithi081 with status 1: 'test -f
    /home/ubuntu/cephtest/archive/qemu/client.0/success'



Running
=================================================================
[7485463] rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-stupid policy/none rbd-mirror/four-per-cluster supported-random-distro$/{ubuntu_latest} workloads/rbd-mirror-snapshot-stress-workunit-minimum}
info:   https://pulpito.ceph.com/dis-2023-12-09_23:55:51-rbd-wip-dis-testing-distro-default-smithi/7485463/



Pass
=================================================================
[7485453] rbd/maintenance/{base/install clusters/{fixed-3 openstack} conf/{disable-pool-app} objectstore/bluestore-low-osd-mem-target qemu/xfstests supported-random-distro$/{ubuntu_latest} workloads/rebuild_object_map}
time:   01:57:03
info:   https://pulpito.ceph.com/dis-2023-12-09_23:55:51-rbd-wip-dis-testing-distro-default-smithi/7485453/

[7485454] rbd/nbd/{base/install cluster/{fixed-3 openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-low-osd-mem-target supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/rbd_nbd_diff_continuous}
time:   00:51:22
info:   https://pulpito.ceph.com/dis-2023-12-09_23:55:51-rbd-wip-dis-testing-distro-default-smithi/7485454/

[7485455] rbd/cli_v1/{base/install clusters/{fixed-1 openstack} conf/{disable-pool-app} features/format-1 msgr-failures/few objectstore/bluestore-bitmap supported-random-distro$/{ubuntu_latest} workloads/rbd_cli_generic}
time:   00:31:57
info:   https://pulpito.ceph.com/dis-2023-12-09_23:55:51-rbd-wip-dis-testing-distro-default-smithi/7485455/

[7485456] rbd/migration/{1-base/install 2-clusters/{fixed-3 openstack} 3-objectstore/bluestore-comp-lz4 4-supported-random-distro$/{centos_latest} 5-data-pool/replicated 6-prepare/raw-file 7-io-workloads/qemu_xfstests 8-migrate-workloads/execute 9-cleanup/cleanup conf/{disable-pool-app}}
time:   01:02:37
info:   https://pulpito.ceph.com/dis-2023-12-09_23:55:51-rbd-wip-dis-testing-distro-default-smithi/7485456/

[7485457] rbd/cli/{base/install clusters/{fixed-1 openstack} conf/{disable-pool-app} data-pool/ec features/defaults msgr-failures/few objectstore/bluestore-stupid supported-random-distro$/{ubuntu_latest} workloads/rbd_cli_generic}
time:   00:43:20
info:   https://pulpito.ceph.com/dis-2023-12-09_23:55:51-rbd-wip-dis-testing-distro-default-smithi/7485457/

[7485458] rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-low-osd-mem-target supported-random-distro$/{ubuntu_latest} workloads/rbd-mirror-workunit-config-key}
time:   00:49:13
info:   https://pulpito.ceph.com/dis-2023-12-09_23:55:51-rbd-wip-dis-testing-distro-default-smithi/7485458/

[7485462] rbd/pwl-cache/tmpfs/{1-base/install 2-cluster/{fix-2 openstack} 3-supported-random-distro$/{ubuntu_latest} 4-cache-path 5-cache-mode/rwl 6-cache-size/1G 7-workloads/qemu_xfstests conf/{disable-pool-app}}
time:   01:46:54
info:   https://pulpito.ceph.com/dis-2023-12-09_23:55:51-rbd-wip-dis-testing-distro-default-smithi/7485462/

[7485464] rbd/valgrind/{base/install centos_latest clusters/{fixed-1 openstack} conf/{disable-pool-app} objectstore/bluestore-comp-zlib validator/memcheck workloads/rbd_mirror}
time:   00:52:27
info:   https://pulpito.ceph.com/dis-2023-12-09_23:55:51-rbd-wip-dis-testing-distro-default-smithi/7485464/

[7485466] rbd/valgrind/{base/install centos_latest clusters/{fixed-1 openstack} conf/{disable-pool-app} objectstore/bluestore-comp-zlib validator/memcheck workloads/c_api_tests}
time:   01:22:26
info:   https://pulpito.ceph.com/dis-2023-12-09_23:55:51-rbd-wip-dis-testing-distro-default-smithi/7485466/

[7485468] rbd/librbd/{cache/writeback clusters/{fixed-3 openstack} conf/{disable-pool-app} data-pool/replicated extra-conf/copy-on-read min-compat-client/default msgr-failures/few objectstore/bluestore-hybrid supported-random-distro$/{centos_latest} workloads/c_api_tests_with_journaling}
time:   00:43:01
info:   https://pulpito.ceph.com/dis-2023-12-09_23:55:51-rbd-wip-dis-testing-distro-default-smithi/7485468/

[7485469] rbd/valgrind/{base/install centos_latest clusters/{fixed-1 openstack} conf/{disable-pool-app} objectstore/bluestore-comp-zstd validator/memcheck workloads/c_api_tests_with_defaults}
time:   01:26:05
info:   https://pulpito.ceph.com/dis-2023-12-09_23:55:51-rbd-wip-dis-testing-distro-default-smithi/7485469/

[7485470] rbd/singleton-bluestore/{all/issue-20295 conf/{disable-pool-app} objectstore/bluestore-comp-snappy openstack supported-random-distro$/{centos_latest}}
time:   00:26:38
info:   https://pulpito.ceph.com/dis-2023-12-09_23:55:51-rbd-wip-dis-testing-distro-default-smithi/7485470/

[7485471] rbd/valgrind/{base/install centos_latest clusters/{fixed-1 openstack} conf/{disable-pool-app} objectstore/bluestore-hybrid validator/memcheck workloads/c_api_tests_with_journaling}
time:   00:55:22
info:   https://pulpito.ceph.com/dis-2023-12-09_23:55:51-rbd-wip-dis-testing-distro-default-smithi/7485471/

[7485472] rbd/librbd/{cache/writeback clusters/{fixed-3 openstack} conf/{disable-pool-app} data-pool/none extra-conf/permit-partial-discard min-compat-client/default msgr-failures/few objectstore/bluestore-comp-zstd supported-random-distro$/{centos_latest} workloads/c_api_tests_with_journaling}
time:   00:34:32
info:   https://pulpito.ceph.com/dis-2023-12-09_23:55:51-rbd-wip-dis-testing-distro-default-smithi/7485472/

[7485473] rbd/librbd/{cache/none clusters/{fixed-3 openstack} conf/{disable-pool-app} data-pool/none extra-conf/permit-partial-discard min-compat-client/default msgr-failures/few objectstore/bluestore-comp-lz4 supported-random-distro$/{ubuntu_latest} workloads/c_api_tests}
time:   01:04:15
info:   https://pulpito.ceph.com/dis-2023-12-09_23:55:51-rbd-wip-dis-testing-distro-default-smithi/7485473/

[7485474] rbd/nbd/{base/install cluster/{fixed-3 openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-stupid supported-random-distro$/{centos_latest} thrashers/default thrashosds-health workloads/rbd_nbd_diff_continuous}
time:   00:45:10
info:   https://pulpito.ceph.com/dis-2023-12-09_23:55:51-rbd-wip-dis-testing-distro-default-smithi/7485474/

[7485475] rbd/librbd/{cache/writeback clusters/{fixed-3 openstack} conf/{disable-pool-app} data-pool/ec extra-conf/none min-compat-client/default msgr-failures/few objectstore/bluestore-comp-zlib supported-random-distro$/{ubuntu_latest} workloads/c_api_tests_with_journaling}
time:   00:42:45
info:   https://pulpito.ceph.com/dis-2023-12-09_23:55:51-rbd-wip-dis-testing-distro-default-smithi/7485475/