2024-04-12T02:01:58.185 INFO:root:teuthology version: 0.0.1.dev249+gcd45576
2024-04-12T02:01:58.185 INFO:teuthology.results:subset: '111/120000'
2024-04-12T02:01:58.186 INFO:teuthology.results:seed: '6550'
2024-04-12T02:17:31.089 INFO:root:teuthology version: 0.0.1.dev249+gcd45576
2024-04-12T02:17:31.092 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2024-04-12T14:18:27.515 WARNING:teuthology.results:test(s) did not finish before timeout of 43200 seconds
2024-04-12T14:18:28.959 INFO:scrape:Found 18 jobs
2024-04-12T14:18:30.221 WARNING:scrape:Saw end of BT but not start
2024-04-12T14:18:30.221 WARNING:scrape:Saw end of BT but not start
2024-04-12T14:18:30.221 WARNING:scrape:Saw end of BT but not start
2024-04-12T14:18:30.222 WARNING:scrape:Saw end of BT but not start
2024-04-12T14:18:30.363 WARNING:scrape:Saw end of BT but not start
2024-04-12T14:18:30.363 WARNING:scrape:Saw end of BT but not start
2024-04-12T14:18:30.364 WARNING:scrape:Saw end of BT but not start
2024-04-12T14:18:30.364 WARNING:scrape:Saw end of BT but not start
2024-04-12T14:18:30.479 WARNING:scrape:Saw end of BT but not start
2024-04-12T14:18:30.480 WARNING:scrape:Saw end of BT but not start
2024-04-12T14:18:30.480 WARNING:scrape:Saw end of BT but not start
2024-04-12T14:18:30.480 WARNING:scrape:Saw end of BT but not start
2024-04-12T14:18:30.576 WARNING:scrape:Saw end of BT but not start
2024-04-12T14:18:30.576 WARNING:scrape:Saw end of BT but not start
2024-04-12T14:18:30.576 WARNING:scrape:Saw end of BT but not start
2024-04-12T14:18:30.576 WARNING:scrape:Saw end of BT but not start
2024-04-12T14:18:30.682 WARNING:scrape:Saw end of BT but not start
2024-04-12T14:18:30.682 WARNING:scrape:Saw end of BT but not start
2024-04-12T14:18:30.682 WARNING:scrape:Saw end of BT but not start
2024-04-12T14:18:30.682 WARNING:scrape:Saw end of BT but not start
2024-04-12T14:18:30.779 WARNING:scrape:Saw end of BT but not start
2024-04-12T14:18:30.780 WARNING:scrape:Saw end of BT but not start
2024-04-12T14:18:30.780 WARNING:scrape:Saw end of BT but not start
2024-04-12T14:18:30.780 WARNING:scrape:Saw end of BT but not start
2024-04-12T14:18:44.560 INFO:scrape:Found 10 distinct failure reasons
2024-04-12T14:18:44.561 INFO:scrape:Dead: b'2024-04-12T14:09:21.674 DEBUG:teuthology.exit:Finished running handlers'
2024-04-12T14:18:44.561 INFO:scrape:3 jobs: ['7652569', '7652571', '7652565']
2024-04-12T14:18:44.561 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate', '3-modules}', 'conf/{client', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}', 'overrides/{ignorelist_health']
2024-04-12T14:18:44.561 INFO:scrape:suites union: ['1-ceph', '2-logrotate', '3-modules}', 'clusters/1a3s-mds-2c-client', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_latest}', 'distro/{ubuntu_20.04}', 'distro/{ubuntu_latest}', 'fs/functional/{begin/{0-install', 'fs/multifs/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mon-debug}', 'mount', 'mount/fuse', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{ignorelist_health', 'rhel_8}', 'subvol_versions/create_subvol_version_v1', 'tasks/admin}', 'tasks/multifs-auth}']
2024-04-12T14:18:44.561 INFO:scrape:
2024-04-12T14:18:44.562 INFO:scrape:Failure: error during scrub thrashing: rank damage found: {'backtrace'}
2024-04-12T14:18:44.562 INFO:scrape:1 jobs: ['7652575']
2024-04-12T14:18:44.562 INFO:scrape:suites: ['1-cephadm', '1-check-counter', '2-logrotate', '2-scrub/yes', '3-modules}', '3-snaps/yes', '4-flush/yes', '5-workunit/fs/misc}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'export-check', 'frag', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'ms_mode/crc', 'n/5', 'objectstore-ec/bluestore-bitmap', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/stock/{k-stock', 'ranks/multi/{balancer/random', 'replication/default}', 'rhel_8}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-namespace-isolated}', 'wsync/yes}']
2024-04-12T14:18:44.562 INFO:scrape:
2024-04-12T14:18:44.562 INFO:scrape:Failure: reached maximum tries (51) after waiting for 300 seconds
2024-04-12T14:18:44.562 INFO:scrape:4 jobs: ['7652579', '7652563', '7652574', '7652568']
2024-04-12T14:18:44.562 INFO:scrape:suites intersection: ['1-volume/{0-create', '2-client/fuse', '3-inline/no', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_8.stream_container_tools', 'conf/{client', 'fail_fs/yes', 'fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'ignorelist_upgrade', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'syntax}']
2024-04-12T14:18:44.563 INFO:scrape:suites union: ['1-ranks/1', '1-ranks/2', '1-volume/{0-create', '2-allow_standby_replay/no', '2-allow_standby_replay/yes', '2-client/fuse', '3-inline/no', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_8.stream_container_tools', 'conf/{client', 'fail_fs/yes', 'fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'ignorelist_upgrade', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'syntax}', 'tasks/{0-from/pacific', 'tasks/{0-from/quincy', 'tasks/{0-from/reef/{v18.2.1}']
2024-04-12T14:18:44.563 INFO:scrape:
2024-04-12T14:18:44.563 INFO:scrape:Failure: Command failed (workunit test suites/dbench.sh) on smithi037 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=d540ebaca6b131a1dd560e7f69e024b133bbaa42 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/dbench.sh'
2024-04-12T14:18:44.563 INFO:scrape:1 jobs: ['7652562']
2024-04-12T14:18:44.563 INFO:scrape:suites: ['1-cephadm', '1-check-counter', '2-logrotate', '2-scrub/yes', '3-modules}', '3-snaps/yes', '4-flush/no', '5-workunit/suites/dbench}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/legacy', 'objectstore-ec/bluestore-comp-ec-root', 'omap_limit/10', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/testing/k-testing', 'ranks/1', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-quota}', 'wsync/no}']
2024-04-12T14:18:44.563 INFO:scrape:
2024-04-12T14:18:44.563 INFO:scrape:Failure: Test failure: test_add_ancestor_and_child_directory (tasks.cephfs.test_mirroring.TestMirroring)
2024-04-12T14:18:44.563 INFO:scrape:1 jobs: ['7652572']
2024-04-12T14:18:44.563 INFO:scrape:suites: ['1-ceph', '2-logrotate', '3-modules}', 'centos_latest', 'clients/mirror', 'cluster/1-node', 'debug', 'fs/valgrind/{begin/{0-install', 'mirror/{cephfs-mirror/one-per-cluster', 'mount/fuse', 'overrides/ignorelist_health', 'tasks/mirror}}']
2024-04-12T14:18:44.563 INFO:scrape:
2024-04-12T14:18:44.564 INFO:scrape:Failure: Test failure: test_fscrypt_dummy_encryption_with_quick_group (tasks.cephfs.test_fscrypt.TestFscrypt)
2024-04-12T14:18:44.564 INFO:scrape:1 jobs: ['7652570']
2024-04-12T14:18:44.564 INFO:scrape:suites: ['1-ceph', '2-logrotate', '3-modules}', 'bluestore-bitmap', 'clusters/1-mds-1-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/fscrypt/{begin/{0-install', 'ignorelist_health_more', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/v1', 'osd}', 'overrides/{distro/testing/k-testing}}', 'overrides/{ignorelist_health', 'pg-warn}', 'tasks/fscrypt-common}']
2024-04-12T14:18:44.564 INFO:scrape:
2024-04-12T14:18:44.564 INFO:scrape:Failure: SELinux denials found on ubuntu@smithi023.front.sepia.ceph.com: ['type=AVC msg=audit(1712887967.337:194): avc:  denied  { checkpoint_restore } for  pid=1157 comm="agetty" capability=40  scontext=system_u:system_r:getty_t:s0-s0:c0.c1023 tcontext=system_u:system_r:getty_t:s0-s0:c0.c1023 tclass=capability2 permissive=1']
2024-04-12T14:18:44.564 INFO:scrape:1 jobs: ['7652567']
2024-04-12T14:18:44.564 INFO:scrape:suites: ['1-ceph', '2-logrotate', '2-workunit/cfuse_workunit_suites_pjd}}', '3-modules}', 'clusters/1a3s-mds-2c-client', 'conf/{client', 'distro/{centos_latest}', 'frag', 'fs/thrash/multifs/{begin/{0-install', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'msgr-failures/osd-mds-delay', 'multifs', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{client-shutdown', 'overrides/{distro/testing/k-testing', 'session_timeout', 'tasks/{1-thrash/mon', 'thrashosds-health}']
2024-04-12T14:18:44.564 INFO:scrape:
2024-04-12T14:18:44.564 INFO:scrape:Failure: "2024-04-12T02:36:19.413430+0000 mon.a (mon.0) 587 : cluster [WRN] Health check failed: Reduced data availability: 2 pgs peering (PG_AVAILABILITY)" in cluster log
2024-04-12T14:18:44.564 INFO:scrape:1 jobs: ['7652566']
2024-04-12T14:18:44.565 INFO:scrape:suites: ['1-ceph', '2-logrotate', '3-modules}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{ignorelist_health', 'subvol_versions/create_subvol_version_v2', 'tasks/snap-schedule}']
2024-04-12T14:18:44.565 INFO:scrape:
2024-04-12T14:18:44.565 INFO:scrape:Dead: b'2024-04-12T14:18:15.320 DEBUG:teuthology.orchestra.run.smithi153:> sudo logrotate /etc/logrotate.d/ceph-test.conf'
2024-04-12T14:18:44.565 INFO:scrape:1 jobs: ['7652577']
2024-04-12T14:18:44.565 INFO:scrape:suites: ['1-ceph', '2-logrotate', '3-modules}', 'clusters/1a3s-mds-2c-client', 'conf/{client', 'distro/{rhel_8}', 'fs/multifs/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mon-debug}', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'objectstore-ec/bluestore-bitmap', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'tasks/multifs-auth}']
2024-04-12T14:18:44.565 INFO:scrape:
2024-04-12T14:18:44.565 INFO:scrape:Failure: Command failed (workunit test fs/snaps/snaptest-git-ceph.sh) on smithi154 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=d540ebaca6b131a1dd560e7f69e024b133bbaa42 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-git-ceph.sh'
2024-04-12T14:18:44.565 INFO:scrape:1 jobs: ['7652573']
2024-04-12T14:18:44.565 INFO:scrape:suites: ['1-ceph', '2-logrotate', '2-workunit/cfuse_workunit_snaptests}}', '3-modules}', 'clusters/1a3s-mds-2c-client', 'conf/{client', 'distro/{rhel_8}', 'frag', 'fs/thrash/multifs/{begin/{0-install', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'msgr-failures/osd-mds-delay', 'multifs', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{client-shutdown', 'overrides/{distro/testing/k-testing', 'session_timeout', 'tasks/{1-thrash/mds', 'thrashosds-health}']
2024-04-12T14:18:44.565 INFO:scrape: