2023-10-26T05:25:07.601 INFO:root:teuthology version: 0.0.1.dev201+g202b180 2023-10-26T05:25:07.602 INFO:teuthology.results:subset: '66/256' 2023-10-26T05:25:07.602 INFO:teuthology.results:seed: '7595' 2023-10-26T06:10:19.422 INFO:root:teuthology version: 0.0.1.dev201+g202b180 2023-10-26T06:10:19.457 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2023-10-26T15:34:03.311 INFO:teuthology.results:Tests finished! gathering results... 2023-10-26T15:34:06.021 INFO:scrape:Found 30 jobs 2023-10-26T15:34:14.865 WARNING:scrape:Misunderstood line: 2023-10-26T15:34:18.194 WARNING:scrape:Misunderstood line: 2023-10-26T15:34:18.293 WARNING:scrape:Misunderstood line: 2023-10-26T15:34:23.755 WARNING:scrape:Saw end of BT but not start 2023-10-26T15:34:23.756 WARNING:scrape:Saw end of BT but not start 2023-10-26T15:34:23.757 WARNING:scrape:Saw end of BT but not start 2023-10-26T15:34:23.757 WARNING:scrape:Saw end of BT but not start 2023-10-26T15:34:23.758 WARNING:scrape:Saw end of BT but not start 2023-10-26T15:34:23.758 WARNING:scrape:Saw end of BT but not start 2023-10-26T15:34:23.759 WARNING:scrape:Saw end of BT but not start 2023-10-26T15:34:23.759 WARNING:scrape:Saw end of BT but not start 2023-10-26T15:34:23.825 WARNING:scrape:Saw end of BT but not start 2023-10-26T15:34:23.826 WARNING:scrape:Saw end of BT but not start 2023-10-26T15:34:23.826 WARNING:scrape:Saw end of BT but not start 2023-10-26T15:34:23.827 WARNING:scrape:Saw end of BT but not start 2023-10-26T15:34:23.827 WARNING:scrape:Saw end of BT but not start 2023-10-26T15:34:23.827 WARNING:scrape:Saw end of BT but not start 2023-10-26T15:34:23.828 WARNING:scrape:Saw end of BT but not start 2023-10-26T15:34:23.828 WARNING:scrape:Saw end of BT but not start 2023-10-26T15:34:23.892 WARNING:scrape:Saw end of BT but not start 2023-10-26T15:34:23.893 WARNING:scrape:Saw end of BT but not start 2023-10-26T15:34:23.894 WARNING:scrape:Saw end of BT but not start 2023-10-26T15:34:23.894 WARNING:scrape:Saw end of BT but not start 2023-10-26T15:34:23.895 WARNING:scrape:Saw end of BT but not start 2023-10-26T15:34:23.895 WARNING:scrape:Saw end of BT but not start 2023-10-26T15:34:23.896 WARNING:scrape:Saw end of BT but not start 2023-10-26T15:34:23.896 WARNING:scrape:Saw end of BT but not start 2023-10-26T15:34:23.961 WARNING:scrape:Saw end of BT but not start 2023-10-26T15:34:23.962 WARNING:scrape:Saw end of BT but not start 2023-10-26T15:34:23.962 WARNING:scrape:Saw end of BT but not start 2023-10-26T15:34:23.963 WARNING:scrape:Saw end of BT but not start 2023-10-26T15:34:23.963 WARNING:scrape:Saw end of BT but not start 2023-10-26T15:34:23.963 WARNING:scrape:Saw end of BT but not start 2023-10-26T15:34:23.964 WARNING:scrape:Saw end of BT but not start 2023-10-26T15:34:23.964 WARNING:scrape:Saw end of BT but not start 2023-10-26T15:34:24.027 WARNING:scrape:Saw end of BT but not start 2023-10-26T15:34:24.028 WARNING:scrape:Saw end of BT but not start 2023-10-26T15:34:24.029 WARNING:scrape:Saw end of BT but not start 2023-10-26T15:34:24.029 WARNING:scrape:Saw end of BT but not start 2023-10-26T15:34:24.029 WARNING:scrape:Saw end of BT but not start 2023-10-26T15:34:24.030 WARNING:scrape:Saw end of BT but not start 2023-10-26T15:34:24.030 WARNING:scrape:Saw end of BT but not start 2023-10-26T15:34:24.031 WARNING:scrape:Saw end of BT but not start 2023-10-26T15:34:24.094 WARNING:scrape:Saw end of BT but not start 2023-10-26T15:34:24.095 WARNING:scrape:Saw end of BT but not start 2023-10-26T15:34:24.095 WARNING:scrape:Saw end of BT but not start 2023-10-26T15:34:24.095 WARNING:scrape:Saw end of BT but not start 2023-10-26T15:34:24.096 WARNING:scrape:Saw end of BT but not start 2023-10-26T15:34:24.096 WARNING:scrape:Saw end of BT but not start 2023-10-26T15:34:24.097 WARNING:scrape:Saw end of BT but not start 2023-10-26T15:34:24.097 WARNING:scrape:Saw end of BT but not start 2023-10-26T15:34:24.160 WARNING:scrape:Saw end of BT but not start 2023-10-26T15:34:24.161 WARNING:scrape:Saw end of BT but not start 2023-10-26T15:34:24.161 WARNING:scrape:Saw end of BT but not start 2023-10-26T15:34:24.162 WARNING:scrape:Saw end of BT but not start 2023-10-26T15:34:24.162 WARNING:scrape:Saw end of BT but not start 2023-10-26T15:34:24.162 WARNING:scrape:Saw end of BT but not start 2023-10-26T15:34:24.163 WARNING:scrape:Saw end of BT but not start 2023-10-26T15:34:24.163 WARNING:scrape:Saw end of BT but not start 2023-10-26T15:34:24.247 WARNING:scrape:Saw end of BT but not start 2023-10-26T15:34:24.247 WARNING:scrape:Saw end of BT but not start 2023-10-26T15:34:24.248 WARNING:scrape:Saw end of BT but not start 2023-10-26T15:34:24.248 WARNING:scrape:Saw end of BT but not start 2023-10-26T15:34:24.249 WARNING:scrape:Saw end of BT but not start 2023-10-26T15:34:24.249 WARNING:scrape:Saw end of BT but not start 2023-10-26T15:34:24.250 WARNING:scrape:Saw end of BT but not start 2023-10-26T15:34:24.250 WARNING:scrape:Saw end of BT but not start 2023-10-26T15:34:24.316 WARNING:scrape:Saw end of BT but not start 2023-10-26T15:34:24.317 WARNING:scrape:Saw end of BT but not start 2023-10-26T15:34:24.317 WARNING:scrape:Saw end of BT but not start 2023-10-26T15:34:24.318 WARNING:scrape:Saw end of BT but not start 2023-10-26T15:34:24.318 WARNING:scrape:Saw end of BT but not start 2023-10-26T15:34:24.319 WARNING:scrape:Saw end of BT but not start 2023-10-26T15:34:24.319 WARNING:scrape:Saw end of BT but not start 2023-10-26T15:34:24.320 WARNING:scrape:Saw end of BT but not start 2023-10-26T15:34:24.384 WARNING:scrape:Saw end of BT but not start 2023-10-26T15:34:24.384 WARNING:scrape:Saw end of BT but not start 2023-10-26T15:34:24.385 WARNING:scrape:Saw end of BT but not start 2023-10-26T15:34:24.385 WARNING:scrape:Saw end of BT but not start 2023-10-26T15:34:24.386 WARNING:scrape:Saw end of BT but not start 2023-10-26T15:34:24.386 WARNING:scrape:Saw end of BT but not start 2023-10-26T15:34:24.387 WARNING:scrape:Saw end of BT but not start 2023-10-26T15:34:24.387 WARNING:scrape:Saw end of BT but not start 2023-10-26T15:34:24.451 WARNING:scrape:Saw end of BT but not start 2023-10-26T15:34:24.452 WARNING:scrape:Saw end of BT but not start 2023-10-26T15:34:24.453 WARNING:scrape:Saw end of BT but not start 2023-10-26T15:34:24.453 WARNING:scrape:Saw end of BT but not start 2023-10-26T15:34:24.454 WARNING:scrape:Saw end of BT but not start 2023-10-26T15:34:24.454 WARNING:scrape:Saw end of BT but not start 2023-10-26T15:34:24.455 WARNING:scrape:Saw end of BT but not start 2023-10-26T15:34:24.455 WARNING:scrape:Saw end of BT but not start 2023-10-26T15:34:24.520 WARNING:scrape:Saw end of BT but not start 2023-10-26T15:34:24.521 WARNING:scrape:Saw end of BT but not start 2023-10-26T15:34:24.522 WARNING:scrape:Saw end of BT but not start 2023-10-26T15:34:24.522 WARNING:scrape:Saw end of BT but not start 2023-10-26T15:34:24.522 WARNING:scrape:Saw end of BT but not start 2023-10-26T15:34:24.523 WARNING:scrape:Saw end of BT but not start 2023-10-26T15:34:24.523 WARNING:scrape:Saw end of BT but not start 2023-10-26T15:34:24.524 WARNING:scrape:Saw end of BT but not start 2023-10-26T15:34:24.585 WARNING:scrape:Saw end of BT but not start 2023-10-26T15:34:24.586 WARNING:scrape:Saw end of BT but not start 2023-10-26T15:34:24.586 WARNING:scrape:Saw end of BT but not start 2023-10-26T15:34:24.587 WARNING:scrape:Saw end of BT but not start 2023-10-26T15:34:24.587 WARNING:scrape:Saw end of BT but not start 2023-10-26T15:34:24.588 WARNING:scrape:Saw end of BT but not start 2023-10-26T15:34:24.588 WARNING:scrape:Saw end of BT but not start 2023-10-26T15:34:24.589 WARNING:scrape:Saw end of BT but not start 2023-10-26T15:34:26.496 INFO:scrape:Found 13 distinct failure reasons 2023-10-26T15:34:26.497 INFO:scrape:Failure: "2023-10-26T06:30:33.038056+0000 osd.0 (osd.0) 3 : cluster [WRN] OSD bench result of 69740.643613 IOPS exceeded the threshold limit of 500.000000 IOPS for osd.0. IOPS capacity is unchanged at 315.000000 IOPS. The recommendation is to establish the osd's IOPS capacity using other benchmark tools (e.g. Fio) and then override osd_mclock_max_capacity_iops_[hdd|ssd]." in cluster log 2023-10-26T15:34:26.498 INFO:scrape:2 jobs: ['7438470', '7438445'] 2023-10-26T15:34:26.498 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'no_client_pidfile}', 'osd}', 'overrides/{ignorelist_health', 'tasks/mds-full}'] 2023-10-26T15:34:26.499 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_latest}', 'distro/{ubuntu_latest}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'tasks/mds-full}'] 2023-10-26T15:34:26.499 INFO:scrape: 2023-10-26T15:34:26.499 INFO:scrape:Failure: error during scrub thrashing: rank damage found: {'backtrace'} 2023-10-26T15:34:26.500 INFO:scrape:6 jobs: ['7438448', '7438454', '7438456', '7438461', '7438446', '7438455'] 2023-10-26T15:34:26.500 INFO:scrape:suites intersection: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/yes', '3-snaps/no', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'ranks/multi/{export-check', 'session_timeout}', 'standby-replay'] 2023-10-26T15:34:26.501 INFO:scrape:suites union: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/yes', '3-snaps/no', '4-flush/no', '4-flush/yes', '5-workunit/kernel_untar_build}}', '5-workunit/postgres}}', '5-workunit/suites/blogbench}}', '5-workunit/suites/dbench}}', '5-workunit/suites/fsx}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{base/{mount-syntax/{v1}', 'mount/kclient/{base/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/crc', 'ms_mode/secure', 'n/3', 'n/5', 'objectstore-ec/bluestore-comp', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'omap_limit/10', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/stock/{k-stock', 'ranks/multi/{export-check', 'replication/always}', 'replication/default}', 'rhel_8}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{no-subvolume}', 'tasks/{0-subvolume/{with-namespace-isolated}', 'tasks/{0-subvolume/{with-quota}', 'wsync/yes}'] 2023-10-26T15:34:26.501 INFO:scrape: 2023-10-26T15:34:26.502 INFO:scrape:Failure: "2023-10-26T06:10:20.661255+0000 mon.a (mon.0) 2517 : cluster [WRN] Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY)" in cluster log 2023-10-26T15:34:26.502 INFO:scrape:6 jobs: ['7438449', '7438465', '7438444', '7438443', '7438442', '7438463'] 2023-10-26T15:34:26.502 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}'] 2023-10-26T15:34:26.503 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', 'centos_latest', 'cephfs-mirror/one-per-cluster', 'clients/mirror', 'clients/{mirror}', 'cluster/1-node', 'cluster/{1-node}', 'clusters/1a3s-mds-2c-client', 'clusters/1a3s-mds-4c-client', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'debug', 'distro/{centos_latest}', 'distro/{ubuntu/{overrides', 'distro/{ubuntu_latest}', 'fs/functional/{begin/{0-install', 'fs/mirror/{begin/{0-install', 'fs/multifs/{begin/{0-install', 'fs/valgrind/{begin/{0-install', 'fs/verify/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mirror/{cephfs-mirror/one-per-cluster', 'mon', 'mon-debug', 'mon-debug}', 'mount', 'mount/fuse', 'mount/kclient/{k-testing', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp', 'objectstore-ec/bluestore-ec-root', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/ignorelist_health', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'overrides/{ignorelist_health}', 'ranks/5', 'session_timeout}', 'supported-random-distros$/{centos_latest}', 'tasks/dbench', 'tasks/failover}', 'tasks/metrics}', 'tasks/mirror}', 'tasks/mirror}}', 'tasks/multifs-auth}', 'ubuntu_latest}}', 'validater/valgrind}'] 2023-10-26T15:34:26.503 INFO:scrape: 2023-10-26T15:34:26.504 INFO:scrape:Failure: Test failure: test_idem_unaffected_root_squash (tasks.cephfs.test_admin.TestFsAuthorizeUpdate) 2023-10-26T15:34:26.504 INFO:scrape:1 jobs: ['7438462'] 2023-10-26T15:34:26.504 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_latest}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{ignorelist_health', 'rhel_8}', 'tasks/admin}'] 2023-10-26T15:34:26.505 INFO:scrape: 2023-10-26T15:34:26.505 INFO:scrape:Failure: "2023-10-26T06:34:32.593318+0000 mds.b (mds.0) 1 : cluster [WRN] evicting unresponsive client smithi007:0 (7074), after 303.428 seconds" in cluster log 2023-10-26T15:34:26.505 INFO:scrape:1 jobs: ['7438468'] 2023-10-26T15:34:26.506 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_latest}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'tasks/journal-repair}'] 2023-10-26T15:34:26.506 INFO:scrape: 2023-10-26T15:34:26.507 INFO:scrape:Failure: Test failure: test_generic (tasks.cephfs.tests_from_xfstests_dev.TestXFSTestsDev) 2023-10-26T15:34:26.507 INFO:scrape:1 jobs: ['7438460'] 2023-10-26T15:34:26.507 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'tasks/xfstests-dev}'] 2023-10-26T15:34:26.508 INFO:scrape: 2023-10-26T15:34:26.508 INFO:scrape:Failure: Command failed (workunit test kernel_untar_build.sh) on smithi155 with status 2: 'mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=384993681abceb8cb8affea10538d607052db4b4 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.1/qa/workunits/kernel_untar_build.sh' 2023-10-26T15:34:26.509 INFO:scrape:2 jobs: ['7438447', '7438469'] 2023-10-26T15:34:26.509 INFO:scrape:suites intersection: ['2-logrotate}', 'conf/{client', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}'] 2023-10-26T15:34:26.509 INFO:scrape:suites union: ['1-ceph', '1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/yes', '3-snaps/yes', '4-flush/no', '5-workunit/kernel_untar_build}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'clusters/1a3s-mds-2c-client', 'conf/{client', 'distro/{centos_latest}', 'frag', 'fs/mixed-clients/{begin/{0-install', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'kclient-overrides/{distro/testing/k-testing', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v1}', 'ms-die-on-skipped}', 'ms-die-on-skipped}}', 'ms_mode/legacy', 'n/3', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-ec-root', 'omap_limit/10', 'osd-asserts', 'osd-asserts}', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'ranks/multi/{export-check', 'replication/always}', 'session_timeout}', 'standby-replay', 'tasks/kernel_cfuse_workunits_untarbuild_blogbench}', 'tasks/{0-subvolume/{with-namespace-isolated}', 'wsync/no}'] 2023-10-26T15:34:26.510 INFO:scrape: 2023-10-26T15:34:26.510 INFO:scrape:Valgrind: mon (Leak_StillReachable) 2023-10-26T15:34:26.511 INFO:scrape:1 jobs: ['7438458'] 2023-10-26T15:34:26.511 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{ubuntu/{overrides', 'fs/verify/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mon-debug', 'mount', 'mount/kclient/{k-testing', 'ms-die-on-skipped}', 'objectstore-ec/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'ranks/1', 'session_timeout}', 'tasks/dbench', 'ubuntu_latest}}', 'validater/valgrind}'] 2023-10-26T15:34:26.511 INFO:scrape: 2023-10-26T15:34:26.512 INFO:scrape:Valgrind: client (Leak_StillReachable), mon (Leak_StillReachable) 2023-10-26T15:34:26.512 INFO:scrape:1 jobs: ['7438453'] 2023-10-26T15:34:26.513 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{centos_latest}', 'fs/verify/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mon-debug', 'mount/fuse', 'objectstore-ec/bluestore-ec-root', 'osd}', 'overrides/{ignorelist_health', 'ranks/5', 'session_timeout}', 'tasks/fsstress', 'validater/valgrind}'] 2023-10-26T15:34:26.513 INFO:scrape: 2023-10-26T15:34:26.513 INFO:scrape:Timeout 120 running ceph --cluster ceph tell mds.1:0 scrub status 2023-10-26T15:34:26.514 INFO:scrape:1 jobs: ['7438457'] 2023-10-26T15:34:26.514 INFO:scrape:suites: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/yes', '3-snaps/no', '4-flush/yes', '5-workunit/suites/ffsb}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'ms_mode/crc', 'objectstore-ec/bluestore-ec-root', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/stock/{k-stock', 'ranks/1', 'rhel_8}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-namespace-isolated-and-quota}', 'wsync/yes}'] 2023-10-26T15:34:26.515 INFO:scrape: 2023-10-26T15:34:26.515 INFO:scrape:Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x188) [0x7fcec476d151] 2023-10-26T15:34:26.515 INFO:scrape:ceph version 18.0.0-6863-g38499368 (384993681abceb8cb8affea10538d607052db4b4) reef (dev) 1: /lib64/libc.so.6(+0x54db0) [0x7fcec3e54db0] 2: /lib64/libc.so.6(+0xa154c) [0x7fcec3ea154c] 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x188) [0x7fcec476d151] 6: /usr/lib64/ceph/libceph-common.so.2(+0x16d2b5) [0x7fcec476d2b5] 7: (MDSTableServer::handle_prepare(boost::intrusive_ptr const&)+0x354) [0x560a42dce674] 8: (MDSRank::handle_message(boost::intrusive_ptr const&)+0xe9) [0x560a42b5f8d9] 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x4eb) [0x560a42b6235b] 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x560a42b6284c] 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x195) [0x560a42b4c595] 12: (DispatchQueue::entry()+0x53a) [0x7fcec496721a] 13: /usr/lib64/ceph/libceph-common.so.2(+0x3f7be1) [0x7fcec49f7be1] 14: /lib64/libc.so.6(+0x9f802) [0x7fcec3e9f802] 15: /lib64/libc.so.6(+0x3f450) [0x7fcec3e3f450] 2023-10-26T15:34:26.516 INFO:scrape:1 jobs: ['7438459'] 2023-10-26T15:34:26.516 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_latest}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{ignorelist_health', 'tasks/snapshots}'] 2023-10-26T15:34:26.517 INFO:scrape: 2023-10-26T15:34:26.517 INFO:scrape:Assertion: 2023-10-26T06:28:03.767 INFO:journalctl@ceph.osd.4.smithi169.stdout:Oct 26 06:28:03 smithi169 ceph-6fa9c606-73c5-11ee-8db9-212e2dc638e7-osd-4[146402]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x559677fc505d] 2023-10-26T15:34:26.517 INFO:scrape:1 jobs: ['7438451'] 2023-10-26T15:34:26.518 INFO:scrape:suites: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/no', '3-snaps/yes', '4-flush/no', '5-workunit/suites/ffsb}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'ms_mode/legacy', 'objectstore-ec/bluestore-bitmap', 'omap_limit/10', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/testing/k-testing', 'ranks/1', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-namespace-isolated}', 'wsync/no}'] 2023-10-26T15:34:26.518 INFO:scrape: 2023-10-26T15:34:26.518 INFO:scrape:Failure: Command failed (workunit test suites/ffsb.sh) on smithi089 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=384993681abceb8cb8affea10538d607052db4b4 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/ffsb.sh' 2023-10-26T15:34:26.519 INFO:scrape:1 jobs: ['7438471'] 2023-10-26T15:34:26.519 INFO:scrape:suites: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/yes', '3-snaps/yes', '4-flush/no', '5-workunit/suites/ffsb}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/legacy', 'objectstore-ec/bluestore-bitmap', 'omap_limit/10', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/testing/k-testing', 'ranks/1', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-quota}', 'wsync/no}'] 2023-10-26T15:34:26.520 INFO:scrape: 2023-10-26T15:34:26.522 INFO:teuthology.results:Sending results to pdonnell@redhat.com: Test Run: pdonnell-2023-10-26_05:21:22-fs-wip-batrick-testing-20231024.144545-distro-default-smithi ================================================================= info: http://pulpito.ceph.com/pdonnell-2023-10-26_05:21:22-fs-wip-batrick-testing-20231024.144545-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-10-26_05:21:22-fs-wip-batrick-testing-20231024.144545-distro-default-smithi/ failed: 25 dead: 0 running: 0 waiting: 0 queued: 0 passed: 5 Fail ================================================================= [7438442] fs/mirror/{begin/{0-install 1-ceph 2-logrotate} cephfs-mirror/one-per-cluster clients/{mirror} cluster/{1-node} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health} supported-random-distros$/{centos_latest} tasks/mirror} ----------------------------------------------------------------- time: 01:58:50 info: http://pulpito.ceph.com/pdonnell-2023-10-26_05:21:22-fs-wip-batrick-testing-20231024.144545-distro-default-smithi/7438442/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-10-26_05:21:22-fs-wip-batrick-testing-20231024.144545-distro-default-smithi/7438442/ "2023-10-26T06:00:30.954633+0000 mon.a (mon.0) 2166 : cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log [7438443] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/multifs-auth} ----------------------------------------------------------------- time: 00:33:19 info: http://pulpito.ceph.com/pdonnell-2023-10-26_05:21:22-fs-wip-batrick-testing-20231024.144545-distro-default-smithi/7438443/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-10-26_05:21:22-fs-wip-batrick-testing-20231024.144545-distro-default-smithi/7438443/ "2023-10-26T05:54:25.824349+0000 mon.a (mon.0) 1345 : cluster [WRN] Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY)" in cluster log [7438444] fs/valgrind/{begin/{0-install 1-ceph 2-logrotate} centos_latest debug mirror/{cephfs-mirror/one-per-cluster clients/mirror cluster/1-node mount/fuse overrides/ignorelist_health tasks/mirror}} ----------------------------------------------------------------- time: 02:05:30 info: http://pulpito.ceph.com/pdonnell-2023-10-26_05:21:22-fs-wip-batrick-testing-20231024.144545-distro-default-smithi/7438444/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-10-26_05:21:22-fs-wip-batrick-testing-20231024.144545-distro-default-smithi/7438444/ "2023-10-26T06:05:47.282692+0000 mon.a (mon.0) 2164 : cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log [7438445] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/mds-full} ----------------------------------------------------------------- time: 00:23:21 info: http://pulpito.ceph.com/pdonnell-2023-10-26_05:21:22-fs-wip-batrick-testing-20231024.144545-distro-default-smithi/7438445/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-10-26_05:21:22-fs-wip-batrick-testing-20231024.144545-distro-default-smithi/7438445/ "2023-10-26T05:47:08.297699+0000 osd.5 (osd.5) 3 : cluster [WRN] OSD bench result of 164877.815646 IOPS exceeded the threshold limit of 500.000000 IOPS for osd.5. IOPS capacity is unchanged at 315.000000 IOPS. The recommendation is to establish the osd's IOPS capacity using other benchmark tools (e.g. Fio) and then override osd_mclock_max_capacity_iops_[hdd|ssd]." in cluster log [7438446] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/postgres}} ----------------------------------------------------------------- time: 00:48:16 info: http://pulpito.ceph.com/pdonnell-2023-10-26_05:21:22-fs-wip-batrick-testing-20231024.144545-distro-default-smithi/7438446/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-10-26_05:21:22-fs-wip-batrick-testing-20231024.144545-distro-default-smithi/7438446/ error during scrub thrashing: rank damage found: {'backtrace'} [7438447] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_latest} kclient-overrides/{distro/testing/k-testing ms-die-on-skipped} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_untarbuild_blogbench} ----------------------------------------------------------------- time: 00:32:10 info: http://pulpito.ceph.com/pdonnell-2023-10-26_05:21:22-fs-wip-batrick-testing-20231024.144545-distro-default-smithi/7438447/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-10-26_05:21:22-fs-wip-batrick-testing-20231024.144545-distro-default-smithi/7438447/ sentry: https://sentry.ceph.com/organizations/ceph/?query=c1a81070c50f4cca8931684a6734022f Command failed (workunit test kernel_untar_build.sh) on smithi155 with status 2: 'mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=384993681abceb8cb8affea10538d607052db4b4 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.1/qa/workunits/kernel_untar_build.sh' [7438448] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/dbench}} ----------------------------------------------------------------- time: 01:03:55 info: http://pulpito.ceph.com/pdonnell-2023-10-26_05:21:22-fs-wip-batrick-testing-20231024.144545-distro-default-smithi/7438448/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-10-26_05:21:22-fs-wip-batrick-testing-20231024.144545-distro-default-smithi/7438448/ error during scrub thrashing: rank damage found: {'backtrace'} [7438449] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/metrics} ----------------------------------------------------------------- time: 00:31:51 info: http://pulpito.ceph.com/pdonnell-2023-10-26_05:21:22-fs-wip-batrick-testing-20231024.144545-distro-default-smithi/7438449/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-10-26_05:21:22-fs-wip-batrick-testing-20231024.144545-distro-default-smithi/7438449/ "2023-10-26T06:10:20.661255+0000 mon.a (mon.0) 2517 : cluster [WRN] Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY)" in cluster log [7438451] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/suites/ffsb}} ----------------------------------------------------------------- time: 00:43:22 info: http://pulpito.ceph.com/pdonnell-2023-10-26_05:21:22-fs-wip-batrick-testing-20231024.144545-distro-default-smithi/7438451/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-10-26_05:21:22-fs-wip-batrick-testing-20231024.144545-distro-default-smithi/7438451/ sentry: https://sentry.ceph.com/organizations/ceph/?query=8657e0e435014215923e09eaef82894c Command failed (workunit test suites/ffsb.sh) on smithi033 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=384993681abceb8cb8affea10538d607052db4b4 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/ffsb.sh' [7438453] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/5 tasks/fsstress validater/valgrind} ----------------------------------------------------------------- time: 01:21:16 info: http://pulpito.ceph.com/pdonnell-2023-10-26_05:21:22-fs-wip-batrick-testing-20231024.144545-distro-default-smithi/7438453/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-10-26_05:21:22-fs-wip-batrick-testing-20231024.144545-distro-default-smithi/7438453/ saw valgrind issues [7438454] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/fsx}} ----------------------------------------------------------------- time: 00:32:57 info: http://pulpito.ceph.com/pdonnell-2023-10-26_05:21:22-fs-wip-batrick-testing-20231024.144545-distro-default-smithi/7438454/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-10-26_05:21:22-fs-wip-batrick-testing-20231024.144545-distro-default-smithi/7438454/ error during scrub thrashing: rank damage found: {'backtrace'} [7438455] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/kernel_untar_build}} ----------------------------------------------------------------- time: 00:46:19 info: http://pulpito.ceph.com/pdonnell-2023-10-26_05:21:22-fs-wip-batrick-testing-20231024.144545-distro-default-smithi/7438455/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-10-26_05:21:22-fs-wip-batrick-testing-20231024.144545-distro-default-smithi/7438455/ error during scrub thrashing: rank damage found: {'backtrace'} [7438456] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/blogbench}} ----------------------------------------------------------------- time: 00:42:46 info: http://pulpito.ceph.com/pdonnell-2023-10-26_05:21:22-fs-wip-batrick-testing-20231024.144545-distro-default-smithi/7438456/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-10-26_05:21:22-fs-wip-batrick-testing-20231024.144545-distro-default-smithi/7438456/ error during scrub thrashing: rank damage found: {'backtrace'} [7438457] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/ffsb}} ----------------------------------------------------------------- time: 00:45:05 info: http://pulpito.ceph.com/pdonnell-2023-10-26_05:21:22-fs-wip-batrick-testing-20231024.144545-distro-default-smithi/7438457/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-10-26_05:21:22-fs-wip-batrick-testing-20231024.144545-distro-default-smithi/7438457/ error during scrub thrashing: Command failed on smithi019 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status' [7438458] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu/{overrides ubuntu_latest}} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/1 tasks/dbench validater/valgrind} ----------------------------------------------------------------- time: 00:54:51 info: http://pulpito.ceph.com/pdonnell-2023-10-26_05:21:22-fs-wip-batrick-testing-20231024.144545-distro-default-smithi/7438458/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-10-26_05:21:22-fs-wip-batrick-testing-20231024.144545-distro-default-smithi/7438458/ saw valgrind issues [7438459] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snapshots} ----------------------------------------------------------------- time: 00:25:43 info: http://pulpito.ceph.com/pdonnell-2023-10-26_05:21:22-fs-wip-batrick-testing-20231024.144545-distro-default-smithi/7438459/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-10-26_05:21:22-fs-wip-batrick-testing-20231024.144545-distro-default-smithi/7438459/ sentry: https://sentry.ceph.com/organizations/ceph/?query=c1380e2150a94856ace7f37feeeb39a9 Test failure: test_kill_mdstable (tasks.cephfs.test_snapshots.TestSnapshots), test_kill_mdstable (tasks.cephfs.test_snapshots.TestSnapshots) [7438460] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/xfstests-dev} ----------------------------------------------------------------- time: 09:34:01 info: http://pulpito.ceph.com/pdonnell-2023-10-26_05:21:22-fs-wip-batrick-testing-20231024.144545-distro-default-smithi/7438460/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-10-26_05:21:22-fs-wip-batrick-testing-20231024.144545-distro-default-smithi/7438460/ sentry: https://sentry.ceph.com/organizations/ceph/?query=a42b10998e0f4413b4ea921894a3bd2d Test failure: test_generic (tasks.cephfs.tests_from_xfstests_dev.TestXFSTestsDev) [7438461] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/kernel_untar_build}} ----------------------------------------------------------------- time: 01:49:02 info: http://pulpito.ceph.com/pdonnell-2023-10-26_05:21:22-fs-wip-batrick-testing-20231024.144545-distro-default-smithi/7438461/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-10-26_05:21:22-fs-wip-batrick-testing-20231024.144545-distro-default-smithi/7438461/ error during scrub thrashing: rank damage found: {'backtrace'} [7438462] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/admin} ----------------------------------------------------------------- time: 00:45:00 info: http://pulpito.ceph.com/pdonnell-2023-10-26_05:21:22-fs-wip-batrick-testing-20231024.144545-distro-default-smithi/7438462/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-10-26_05:21:22-fs-wip-batrick-testing-20231024.144545-distro-default-smithi/7438462/ sentry: https://sentry.ceph.com/organizations/ceph/?query=40ef67eb4b8040b694dd0415303c0fb2 Test failure: test_idem_unaffected_root_squash (tasks.cephfs.test_admin.TestFsAuthorizeUpdate) [7438463] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/failover} ----------------------------------------------------------------- time: 01:11:10 info: http://pulpito.ceph.com/pdonnell-2023-10-26_05:21:22-fs-wip-batrick-testing-20231024.144545-distro-default-smithi/7438463/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-10-26_05:21:22-fs-wip-batrick-testing-20231024.144545-distro-default-smithi/7438463/ "2023-10-26T06:35:59.434146+0000 mon.a (mon.0) 3643 : cluster [WRN] Health check failed: Reduced data availability: 2 pgs inactive, 2 pgs peering (PG_AVAILABILITY)" in cluster log [7438465] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu/{overrides ubuntu_latest}} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/5 tasks/dbench validater/valgrind} ----------------------------------------------------------------- time: 00:48:17 info: http://pulpito.ceph.com/pdonnell-2023-10-26_05:21:22-fs-wip-batrick-testing-20231024.144545-distro-default-smithi/7438465/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-10-26_05:21:22-fs-wip-batrick-testing-20231024.144545-distro-default-smithi/7438465/ "2023-10-26T06:30:26.090571+0000 mon.a (mon.0) 376 : cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log [7438468] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/journal-repair} ----------------------------------------------------------------- time: 00:22:23 info: http://pulpito.ceph.com/pdonnell-2023-10-26_05:21:22-fs-wip-batrick-testing-20231024.144545-distro-default-smithi/7438468/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-10-26_05:21:22-fs-wip-batrick-testing-20231024.144545-distro-default-smithi/7438468/ "2023-10-26T06:34:32.593318+0000 mds.b (mds.0) 1 : cluster [WRN] evicting unresponsive client smithi007:0 (7074), after 303.428 seconds" in cluster log [7438469] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/kernel_untar_build}} ----------------------------------------------------------------- time: 00:42:16 info: http://pulpito.ceph.com/pdonnell-2023-10-26_05:21:22-fs-wip-batrick-testing-20231024.144545-distro-default-smithi/7438469/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-10-26_05:21:22-fs-wip-batrick-testing-20231024.144545-distro-default-smithi/7438469/ sentry: https://sentry.ceph.com/organizations/ceph/?query=4c9e8c1b74f8497791d203256b4e25df Command failed (workunit test kernel_untar_build.sh) on smithi050 with status 2: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=384993681abceb8cb8affea10538d607052db4b4 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/kernel_untar_build.sh' [7438470] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/mds-full} ----------------------------------------------------------------- time: 00:28:17 info: http://pulpito.ceph.com/pdonnell-2023-10-26_05:21:22-fs-wip-batrick-testing-20231024.144545-distro-default-smithi/7438470/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-10-26_05:21:22-fs-wip-batrick-testing-20231024.144545-distro-default-smithi/7438470/ "2023-10-26T06:30:33.038056+0000 osd.0 (osd.0) 3 : cluster [WRN] OSD bench result of 69740.643613 IOPS exceeded the threshold limit of 500.000000 IOPS for osd.0. IOPS capacity is unchanged at 315.000000 IOPS. The recommendation is to establish the osd's IOPS capacity using other benchmark tools (e.g. Fio) and then override osd_mclock_max_capacity_iops_[hdd|ssd]." in cluster log [7438471] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/suites/ffsb}} ----------------------------------------------------------------- time: 00:44:10 info: http://pulpito.ceph.com/pdonnell-2023-10-26_05:21:22-fs-wip-batrick-testing-20231024.144545-distro-default-smithi/7438471/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-10-26_05:21:22-fs-wip-batrick-testing-20231024.144545-distro-default-smithi/7438471/ sentry: https://sentry.ceph.com/organizations/ceph/?query=bc459db9ebcf452688ad94ee3f52b7d9 Command failed (workunit test suites/ffsb.sh) on smithi089 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=384993681abceb8cb8affea10538d607052db4b4 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/ffsb.sh' Pass ================================================================= [7438450] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/osd 2-workunit/fs/snaps}} time: 01:16:23 info: http://pulpito.ceph.com/pdonnell-2023-10-26_05:21:22-fs-wip-batrick-testing-20231024.144545-distro-default-smithi/7438450/ [7438452] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/fs/norstats}} time: 00:28:11 info: http://pulpito.ceph.com/pdonnell-2023-10-26_05:21:22-fs-wip-batrick-testing-20231024.144545-distro-default-smithi/7438452/ [7438464] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps} time: 00:38:27 info: http://pulpito.ceph.com/pdonnell-2023-10-26_05:21:22-fs-wip-batrick-testing-20231024.144545-distro-default-smithi/7438464/ [7438466] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/basic}} time: 01:51:48 info: http://pulpito.ceph.com/pdonnell-2023-10-26_05:21:22-fs-wip-batrick-testing-20231024.144545-distro-default-smithi/7438466/ [7438467] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/suites/fsstress}} time: 00:35:58 info: http://pulpito.ceph.com/pdonnell-2023-10-26_05:21:22-fs-wip-batrick-testing-20231024.144545-distro-default-smithi/7438467/