2024-04-11T23:31:29.232 INFO:root:teuthology version: 0.0.1.dev249+gcd45576
2024-04-11T23:31:29.232 INFO:teuthology.results:subset: '111/120000'
2024-04-11T23:31:29.232 INFO:teuthology.results:seed: '1968'
2024-04-12T00:32:56.216 INFO:root:teuthology version: 0.0.1.dev249+gcd45576
2024-04-12T00:32:56.241 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2024-04-12T12:06:51.706 INFO:teuthology.results:Tests finished! gathering results...
2024-04-12T12:06:56.346 INFO:scrape:Found 50 jobs
2024-04-12T12:07:38.544 INFO:scrape:Found 8 distinct failure reasons
2024-04-12T12:07:38.545 INFO:scrape:Failure: "1712880510.6210227 mon.a (mon.0) 503 : cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log
2024-04-12T12:07:38.545 INFO:scrape:19 jobs: ['7652431', '7652425', '7652437', '7652415', '7652423', '7652413', '7652441', '7652456', '7652473', '7652439', '7652426', '7652462', '7652418', '7652428', '7652447', '7652458', '7652420', '7652434', '7652446']
2024-04-12T12:07:38.545 INFO:scrape:suites intersection: []
2024-04-12T12:07:38.545 INFO:scrape:suites union: ['-', '0-roles', '0-start', '1-ceph-install/quincy', '1-start', '1-tasks', '1-tasks}', '2', '2-first-half-tasks/radosbench', '2-first-half-tasks/rbd-cls', '2-first-half-tasks/rbd-import-export', '2-first-half-tasks/rbd_api', '2-first-half-tasks/readwrite', '2-first-half-tasks/snaps-few-objects', '3-stress-tasks/{radosbench', '4-second-half-tasks/radosbench', '4-second-half-tasks/rbd-import-export', 'mon_election/classic', 'mon_election/classic}', 'mon_election/connectivity', 'mon_election/connectivity}', 'objectstore/filestore-xfs', 'rados_api', 'rados_loadgenbig', 'rbd-cls', 'rbd-import-export', 'rbd_api', 'rbd_import_export', 'readwrite', 'snaps-few-objects}', 'start}', 'test_rbd_api', 'test_rbd_python}}', 'ubuntu_latest}', 'upgrade', 'upgrade-sequence', 'upgrade/quincy-x/filestore-remove-check/{0-cluster/{openstack', 'upgrade/quincy-x/parallel/{0-random-distro$/{centos_9.stream_runc}', 'upgrade/quincy-x/parallel/{0-random-distro$/{centos_9.stream}', 'upgrade/quincy-x/parallel/{0-random-distro$/{ubuntu_22.04}', 'upgrade/quincy-x/stress-split/{0-distro/centos_9.stream_runc', 'upgrade/quincy-x/stress-split/{0-distro/ubuntu_22.04', 'upgrade/reef-x/parallel/{0-random-distro$/{centos_9.stream}', 'upgrade/reef-x/parallel/{0-random-distro$/{ubuntu_22.04}', 'upgrade/reef-x/stress-split/{0-distro/ubuntu_22.04', 'upgrade/telemetry-upgrade/quincy-x/{0-random-distro$/{centos_9.stream}', 'workload/{ec-rados-default']
2024-04-12T12:07:38.546 INFO:scrape:
2024-04-12T12:07:38.546 INFO:scrape:Failure: "2024-04-12T00:08:21.904596+0000 mon.a (mon.0) 180 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,b" in cluster log
2024-04-12T12:07:38.546 INFO:scrape:6 jobs: ['7652432', '7652430', '7652417', '7652471', '7652414', '7652443']
2024-04-12T12:07:38.546 INFO:scrape:suites intersection: ['0-roles', '1-start', '3-stress-tasks/{radosbench', 'rbd-cls', 'rbd-import-export', 'rbd_api', 'readwrite', 'snaps-few-objects}']
2024-04-12T12:07:38.546 INFO:scrape:suites union: ['0-roles', '1-start', '2-first-half-tasks/radosbench', '2-first-half-tasks/rbd-import-export', '2-first-half-tasks/rbd_api', '2-first-half-tasks/readwrite', '2-first-half-tasks/snaps-few-objects', '3-stress-tasks/{radosbench', '4-second-half-tasks/radosbench', '4-second-half-tasks/rbd-import-export', 'mon_election/classic}', 'mon_election/connectivity}', 'rbd-cls', 'rbd-import-export', 'rbd_api', 'readwrite', 'snaps-few-objects}', 'upgrade/reef-x/stress-split/{0-distro/centos_9.stream_runc', 'upgrade/reef-x/stress-split/{0-distro/ubuntu_22.04']
2024-04-12T12:07:38.546 INFO:scrape:
2024-04-12T12:07:38.546 INFO:scrape:Failure: "2024-04-12T01:40:02.899091+0000 osd.7 (osd.7) 34 : cluster [ERR] 50.25 soid 50:a4cc24c2:::smithi117515493-23:head : object info inconsistent , snapset inconsistent , attr value mismatch '__header'" in cluster log
2024-04-12T12:07:38.547 INFO:scrape:11 jobs: ['7652450', '7652435', '7652421', '7652444', '7652466', '7652422', '7652436', '7652440', '7652449', '7652419', '7652470']
2024-04-12T12:07:38.547 INFO:scrape:suites intersection: ['0-roles', '1-start', '3-stress-tasks/{radosbench', 'rbd-cls', 'rbd-import-export', 'rbd_api', 'readwrite', 'snaps-few-objects}']
2024-04-12T12:07:38.547 INFO:scrape:suites union: ['0-roles', '1-start', '2-first-half-tasks/radosbench', '2-first-half-tasks/rbd-cls', '2-first-half-tasks/rbd-import-export', '2-first-half-tasks/rbd_api', '2-first-half-tasks/readwrite', '2-first-half-tasks/snaps-few-objects', '3-stress-tasks/{radosbench', '4-second-half-tasks/radosbench', '4-second-half-tasks/rbd-import-export', 'mon_election/classic}', 'mon_election/connectivity}', 'rbd-cls', 'rbd-import-export', 'rbd_api', 'readwrite', 'snaps-few-objects}', 'upgrade/quincy-x/stress-split/{0-distro/centos_9.stream', 'upgrade/quincy-x/stress-split/{0-distro/centos_9.stream_runc', 'upgrade/reef-x/stress-split/{0-distro/centos_9.stream', 'upgrade/reef-x/stress-split/{0-distro/centos_9.stream_runc', 'upgrade/reef-x/stress-split/{0-distro/ubuntu_22.04']
2024-04-12T12:07:38.547 INFO:scrape:
2024-04-12T12:07:38.547 INFO:scrape:Failure: reached maximum tries (51) after waiting for 300 seconds
2024-04-12T12:07:38.547 INFO:scrape:4 jobs: ['7652460', '7652429', '7652438', '7652448']
2024-04-12T12:07:38.548 INFO:scrape:suites intersection: ['1-volume/{0-create', '2-client/fuse', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_9.stream', 'conf/{client', 'fail_fs/yes', 'ignorelist_upgrade', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'pg_health', 'roles', 'syntax}', 'tasks/{0-from/quincy', 'upgrade/cephfs/mds_upgrade_sequence/{bluestore-bitmap']
2024-04-12T12:07:38.548 INFO:scrape:suites union: ['1-ranks/1', '1-ranks/2', '1-volume/{0-create', '2-allow_standby_replay/no', '2-allow_standby_replay/yes', '2-client/fuse', '3-inline/no', '3-inline/yes', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_9.stream', 'conf/{client', 'fail_fs/yes', 'ignorelist_upgrade', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'pg_health', 'roles', 'syntax}', 'tasks/{0-from/quincy', 'upgrade/cephfs/mds_upgrade_sequence/{bluestore-bitmap']
2024-04-12T12:07:38.548 INFO:scrape:
2024-04-12T12:07:38.548 INFO:scrape:Dead: b'2024-04-12T12:02:59.351 DEBUG:teuthology.exit:Finished running handlers'
2024-04-12T12:07:38.548 INFO:scrape:1 jobs: ['7652452']
2024-04-12T12:07:38.548 INFO:scrape:suites: ['0-start', '1-tasks}', 'upgrade/telemetry-upgrade/pacific-x/{0-random-distro$/{centos_8.stream_container_tools}']
2024-04-12T12:07:38.548 INFO:scrape:
2024-04-12T12:07:38.548 INFO:scrape:Failure: Command failed on smithi142 with status 1: 'sudo yum -y upgrade ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd'
2024-04-12T12:07:38.548 INFO:scrape:1 jobs: ['7652445']
2024-04-12T12:07:38.548 INFO:scrape:suites: ['1-mount/mount/fuse', '2-clients/fuse-upgrade', '3-workload/stress_tests/iozone}}', 'branch/pacific', 'centos_8.stream', 'clusters/{1-mds-1-client-micro}', 'conf/{client', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'pg_health}', 'tasks/{0-install', 'upgrade/cephfs/upgraded_client/{bluestore-bitmap']
2024-04-12T12:07:38.549 INFO:scrape:
2024-04-12T12:07:38.549 INFO:scrape:Failure: ceph version 18.2.2-1189.gc96c8d47 was not installed, found 16.2.15-0.el8.
2024-04-12T12:07:38.549 INFO:scrape:2 jobs: ['7652464', '7652427']
2024-04-12T12:07:38.549 INFO:scrape:suites intersection: ['1-upgrade}}', 'centos_8.stream', 'conf/{client', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'no-mds-cluster', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'pg_health}', 'tasks/{0-pacific', 'upgrade/cephfs/nofs/{bluestore-bitmap']
2024-04-12T12:07:38.549 INFO:scrape:suites union: ['1-upgrade}}', 'centos_8.stream', 'conf/{client', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'no-mds-cluster', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'pg_health}', 'tasks/{0-pacific', 'upgrade/cephfs/nofs/{bluestore-bitmap']
2024-04-12T12:07:38.549 INFO:scrape:
2024-04-12T12:07:38.549 INFO:scrape:Failure: Command failed on smithi008 with status 22: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell 4.d deep-scrub'
2024-04-12T12:07:38.550 INFO:scrape:1 jobs: ['7652416']
2024-04-12T12:07:38.550 INFO:scrape:suites: ['1-client', '2-upgrade', '3-compat_client/quincy}}', 'centos_8.stream', 'clusters/1-mds-2-client-micro', 'conf/{client', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'multimds/no', 'multimds/yes', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'pg_health}', 'tasks/{0-octopus', 'upgrade/cephfs/featureful_client/old_client/{bluestore-bitmap']
2024-04-12T12:07:38.550 INFO:scrape: