2023-04-21T13:40:48.771 INFO:root:teuthology version: 0.0.1.dev96+g8d156ae 2023-04-21T13:40:48.772 INFO:teuthology.results:subset: '8/128' 2023-04-21T13:40:48.773 INFO:teuthology.results:seed: '9123' 2023-04-22T03:40:45.429 INFO:root:teuthology version: 0.0.1.dev96+g8d156ae 2023-04-22T03:40:45.482 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2023-04-22T12:00:28.746 INFO:teuthology.results:Tests finished! gathering results... 2023-04-22T12:00:44.762 INFO:scrape:Found 182 jobs 2023-04-22T12:00:54.589 INFO:scrape:Found 11 distinct failure reasons 2023-04-22T12:00:54.590 INFO:scrape:Failure: Failed to fetch package version from https://shaman.ceph.com/api/search/?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=58e06d348d8a2da339540be5425a40ec7683e512 2023-04-22T12:00:54.590 INFO:scrape:2 jobs: ['7247120', '7247186'] 2023-04-22T12:00:54.591 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-2c-client', 'conf/{client', 'fs/mixed-clients/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'kclient-overrides/{distro/testing/k-testing', 'mds', 'mon', 'ms-die-on-skipped}', 'osd-asserts}', 'osd}', 'overrides/{ignorelist_health', 'tasks/kernel_cfuse_workunits_untarbuild_blogbench}'] 2023-04-22T12:00:54.591 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-2c-client', 'conf/{client', 'fs/mixed-clients/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'kclient-overrides/{distro/testing/k-testing', 'mds', 'mon', 'ms-die-on-skipped}', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp', 'osd-asserts}', 'osd}', 'overrides/{ignorelist_health', 'tasks/kernel_cfuse_workunits_untarbuild_blogbench}'] 2023-04-22T12:00:54.591 INFO:scrape: 2023-04-22T12:00:54.592 INFO:scrape:Failure: SSH connection to smithi135 was lost: 'sudo grub2-mkconfig -o /boot/grub2/grub.cfg' 2023-04-22T12:00:54.592 INFO:scrape:2 jobs: ['7247112', '7247106'] 2023-04-22T12:00:54.592 INFO:scrape:suites intersection: ['centos_latest', 'conf/{client', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn}'] 2023-04-22T12:00:54.592 INFO:scrape:suites union: ['1-client', '1-client-upgrade', '2-client-sanity}}', '2-upgrade', '3-client-upgrade', '4-compat_client', '5-client-sanity}}', 'centos_latest', 'clusters/1-mds-2-client-micro', 'clusters/{1-mds-1-client-micro}', 'conf/{client', 'fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap', 'fs/upgrade/upgraded_client/from_nautilus/{bluestore-bitmap', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'multimds/no', 'osd}', 'overrides/{ignorelist_health', 'pg-warn}', 'tasks/{0-nautilus', 'tasks/{0-octopus'] 2023-04-22T12:00:54.593 INFO:scrape: 2023-04-22T12:00:54.593 INFO:scrape:Failure: Command failed on smithi112 with status 1: 'sudo yum install -y kernel' 2023-04-22T12:00:54.593 INFO:scrape:6 jobs: ['7247090', '7247165', '7247182', '7247107', '7247109', '7247169'] 2023-04-22T12:00:54.594 INFO:scrape:suites intersection: ['conf/{client', 'mds', 'mon', 'osd}'] 2023-04-22T12:00:54.594 INFO:scrape:suites union: ['1-ceph', '1-cephadm', '1-client', '1-ranks/2', '1-volume/{0-create', '2-allow_standby_replay/no', '2-client', '2-logrotate}', '2-upgrade', '2-workunit/suites/fsstress}}', '3-compat_client/no}}', '3-inline/yes', '3-upgrade-with-workload', '4-verify}', '4-verify}}', 'begin/{0-install', 'centos_8.stream_container_tools', 'centos_latest', 'clusters/1-mds-2-client-micro', 'clusters/1a11s-mds-1c-client-3node', 'clusters/1a5s-mds-1c-client', 'clusters/fixed-2-ucephfs', 'conf/{client', 'distro/{centos_8}', 'distro/{ubuntu_latest}', 'fs/permission/{begin/{0-install', 'fs/thrash/workloads/{begin/{0-install', 'fs/upgrade/featureful_client/old_client/{bluestore-bitmap', 'fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'ignorelist_wrongly_marked_down}', 'mds', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{mount-syntax/{v1}', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/{crc}', 'msgr-failures/none', 'multimds/no', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp-ec-root', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{distro/testing/k-testing', 'overrides/{frag', 'overrides/{ignorelist_health', 'pg-warn', 'pg-warn}', 'prefetch_dirfrags/no', 'races', 'ranks/1', 'ranks/3', 'rhel_8}', 'roles', 'scrub/yes', 'session_timeout', 'session_timeout}', 'standby-replay', 'syntax}', 'tasks/cfuse_workunit_suites_pjd}', 'tasks/{0-check-counter', 'tasks/{0-from/pacific', 'tasks/{0-octopus', 'tasks/{1-thrash/osd', 'thrashosds-health}', 'workunit/suites/fsstress}', 'wsync/{no}}'] 2023-04-22T12:00:54.594 INFO:scrape: 2023-04-22T12:00:54.595 INFO:scrape:Failure: "2023-04-21T22:30:13.435907+0000 mon.a (mon.0) 624 : cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log 2023-04-22T12:00:54.595 INFO:scrape:2 jobs: ['7247096', '7247237'] 2023-04-22T12:00:54.595 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'no_client_pidfile}', 'osd}', 'overrides/{ignorelist_health', 'tasks/snap-schedule}'] 2023-04-22T12:00:54.596 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_8}', 'distro/{rhel_8}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'tasks/snap-schedule}'] 2023-04-22T12:00:54.596 INFO:scrape: 2023-04-22T12:00:54.596 INFO:scrape:Failure: reached maximum tries (50) after waiting for 300 seconds 2023-04-22T12:00:54.596 INFO:scrape:1 jobs: ['7247081'] 2023-04-22T12:00:54.597 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'cephfs-mirror/three-per-cluster', 'clients/{mirror}', 'cluster/{1-node}', 'fs/mirror-ha/{begin/{0-install', 'objectstore/bluestore-bitmap', 'overrides/{whitelist_health}', 'supported-random-distro$/{ubuntu_latest}', 'workloads/cephfs-mirror-ha-workunit}'] 2023-04-22T12:00:54.597 INFO:scrape: 2023-04-22T12:00:54.597 INFO:scrape:Dead: b'2023-04-22T11:03:30.684 DEBUG:teuthology.exit:Finished running handlers' 2023-04-22T12:00:54.598 INFO:scrape:2 jobs: ['7247122', '7247148'] 2023-04-22T12:00:54.598 INFO:scrape:suites intersection: ['1-volume/{0-create', '2-client', '3-inline/yes', '3-upgrade-with-workload', '4-verify}', '4-verify}}', 'centos_8.stream_container_tools', 'conf/{client', 'fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'syntax}'] 2023-04-22T12:00:54.598 INFO:scrape:suites union: ['1-ranks/1', '1-ranks/2', '1-volume/{0-create', '2-allow_standby_replay/no', '2-allow_standby_replay/yes', '2-client', '3-inline/yes', '3-upgrade-with-workload', '4-verify}', '4-verify}}', 'centos_8.stream_container_tools', 'conf/{client', 'fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'syntax}', 'tasks/{0-from/pacific', 'tasks/{0-from/v16.2.4'] 2023-04-22T12:00:54.599 INFO:scrape: 2023-04-22T12:00:54.599 INFO:scrape:Failure: Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on smithi151 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=58e06d348d8a2da339540be5425a40ec7683e512 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-multiple-capsnaps.sh' 2023-04-22T12:00:54.599 INFO:scrape:1 jobs: ['7247177'] 2023-04-22T12:00:54.599 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-1c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/snaps/{begin/{0-install', 'ignorelist_wrongly_marked_down}', 'mds', 'mon', 'mount/fuse', 'objectstore-ec/bluestore-ec-root', 'osd}', 'overrides/{ignorelist_health', 'tasks/workunit/snaps}'] 2023-04-22T12:00:54.600 INFO:scrape: 2023-04-22T12:00:54.600 INFO:scrape:Timeout 6h running clone.client.0/qa/workunits/suites/fsstress.sh 2023-04-22T12:00:54.600 INFO:scrape:2 jobs: ['7247157', '7247223'] 2023-04-22T12:00:54.601 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'fs/verify/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mon-debug', 'mount/fuse', 'osd}', 'overrides/{ignorelist_health', 'session_timeout}', 'tasks/fsstress', 'validater/valgrind}'] 2023-04-22T12:00:54.601 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{centos_8}', 'distro/{rhel_8}', 'fs/verify/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mon-debug', 'mount/fuse', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp', 'osd}', 'overrides/{ignorelist_health', 'ranks/3', 'ranks/5', 'session_timeout}', 'tasks/fsstress', 'validater/valgrind}'] 2023-04-22T12:00:54.601 INFO:scrape: 2023-04-22T12:00:54.601 INFO:scrape:Failure: {'smithi012.front.sepia.ceph.com': {'_ansible_no_log': False, 'changed': False, 'invocation': {'module_args': {'allow_downgrade': False, 'allowerasing': False, 'autoremove': False, 'bugfix': False, 'conf_file': None, 'disable_excludes': None, 'disable_gpg_check': False, 'disable_plugin': [], 'disablerepo': [], 'download_dir': None, 'download_only': False, 'enable_plugin': [], 'enablerepo': [], 'exclude': [], 'install_repoquery': True, 'install_weak_deps': True, 'installroot': '/', 'list': None, 'lock_timeout': 30, 'name': ['krb5-workstation'], 'releasever': None, 'security': False, 'skip_broken': False, 'state': 'present', 'update_cache': False, 'update_only': False, 'validate_certs': True}}, 'msg': "Failed to download metadata for repo 'CentOS-PowerTools': Yum repo downloading error: Downloading error(s): repodata/e830a7a4e881ef24680d161802ae07874dd447031dd12d47f2d3d4a911245522-primary.xml.gz - Cannot download, all mirrors were already tried without success", 'rc': 1, 'results': []}} 2023-04-22T12:00:54.602 INFO:scrape:2 jobs: ['7247246', '7247085'] 2023-04-22T12:00:54.602 INFO:scrape:suites intersection: ['conf/{client', 'mds', 'mon', 'osd}', 'overrides/{ignorelist_health'] 2023-04-22T12:00:54.602 INFO:scrape:suites union: ['1-ceph', '1-ranks/1', '1-volume/{0-create', '2-allow_standby_replay/no', '2-client', '2-logrotate}', '3-inline/yes', '3-upgrade-with-workload', '4-verify}', '4-verify}}', 'centos_8.stream_container_tools', 'clusters/fixed-2-ucephfs', 'conf/{client', 'distro/{centos_8}', 'fs/permission/{begin/{0-install', 'fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'ignorelist_wrongly_marked_down', 'ignorelist_wrongly_marked_down}', 'mds', 'mon', 'mount/fuse', 'objectstore-ec/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'syntax}', 'tasks/cfuse_workunit_misc}', 'tasks/{0-from/pacific'] 2023-04-22T12:00:54.603 INFO:scrape: 2023-04-22T12:00:54.603 INFO:scrape:Failure: Command failed on smithi088 with status 22: 'sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 60e00d0a-e05a-11ed-9b00-001a4aab830c -- ceph orch daemon add osd smithi088:vg_nvme/lv_3' 2023-04-22T12:00:54.603 INFO:scrape:1 jobs: ['7247067'] 2023-04-22T12:00:54.603 INFO:scrape:suites: ['1-ranks/1', '1-volume/{0-create', '2-allow_standby_replay/yes', '2-client', '3-inline/yes', '3-upgrade-with-workload', '4-verify}', '4-verify}}', 'centos_8.stream_container_tools', 'conf/{client', 'fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'syntax}', 'tasks/{0-from/v16.2.4'] 2023-04-22T12:00:54.604 INFO:scrape: 2023-04-22T12:00:54.604 INFO:scrape:Failure: "2023-04-22T02:06:36.021815+0000 osd.6 (osd.6) 3 : cluster [WRN] OSD bench result of 236279.412061 IOPS exceeded the threshold limit of 500.000000 IOPS for osd.6. IOPS capacity is unchanged at 315.000000 IOPS. The recommendation is to establish the osd's IOPS capacity using other benchmark tools (e.g. Fio) and then override osd_mclock_max_capacity_iops_[hdd|ssd]." in cluster log 2023-04-22T12:00:54.604 INFO:scrape:1 jobs: ['7247194'] 2023-04-22T12:00:54.605 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{ignorelist_health', 'rhel_8}', 'tasks/mds-full}'] 2023-04-22T12:00:54.605 INFO:scrape: 2023-04-22T12:00:54.615 INFO:teuthology.results:Sending results to pdonnell@redhat.com: Test Run: pdonnell-2023-04-21_02:33:35-fs-wip-pdonnell-testing-20230420.183701-quincy-distro-default-smithi ================================================================= info: http://pulpito.ceph.com/pdonnell-2023-04-21_02:33:35-fs-wip-pdonnell-testing-20230420.183701-quincy-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-04-21_02:33:35-fs-wip-pdonnell-testing-20230420.183701-quincy-distro-default-smithi/ failed: 16 dead: 6 running: 0 waiting: 0 queued: 0 passed: 160 Fail ================================================================= [7247067] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} ----------------------------------------------------------------- time: 01:29:41 info: http://pulpito.ceph.com/pdonnell-2023-04-21_02:33:35-fs-wip-pdonnell-testing-20230420.183701-quincy-distro-default-smithi/7247067/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-04-21_02:33:35-fs-wip-pdonnell-testing-20230420.183701-quincy-distro-default-smithi/7247067/ sentry: https://sentry.ceph.com/organizations/ceph/?query=b873f920456844ebb99627661d9cae35 Command failed on smithi088 with status 22: 'sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 60e00d0a-e05a-11ed-9b00-001a4aab830c -- ceph orch daemon add osd smithi088:vg_nvme/lv_3' [7247081] fs/mirror-ha/{begin/{0-install 1-ceph 2-logrotate} cephfs-mirror/three-per-cluster clients/{mirror} cluster/{1-node} objectstore/bluestore-bitmap overrides/{whitelist_health} supported-random-distro$/{ubuntu_latest} workloads/cephfs-mirror-ha-workunit} ----------------------------------------------------------------- time: 00:40:03 info: http://pulpito.ceph.com/pdonnell-2023-04-21_02:33:35-fs-wip-pdonnell-testing-20230420.183701-quincy-distro-default-smithi/7247081/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-04-21_02:33:35-fs-wip-pdonnell-testing-20230420.183701-quincy-distro-default-smithi/7247081/ reached maximum tries (50) after waiting for 300 seconds [7247090] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/pdonnell-2023-04-21_02:33:35-fs-wip-pdonnell-testing-20230420.183701-quincy-distro-default-smithi/7247090/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-04-21_02:33:35-fs-wip-pdonnell-testing-20230420.183701-quincy-distro-default-smithi/7247090/ sentry: https://sentry.ceph.com/organizations/ceph/?query=42c17f9005e241f59eaf07c895bb31ed Command failed on smithi112 with status 1: 'sudo yum install -y kernel' [7247096] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snap-schedule} ----------------------------------------------------------------- time: 00:53:45 info: http://pulpito.ceph.com/pdonnell-2023-04-21_02:33:35-fs-wip-pdonnell-testing-20230420.183701-quincy-distro-default-smithi/7247096/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-04-21_02:33:35-fs-wip-pdonnell-testing-20230420.183701-quincy-distro-default-smithi/7247096/ "2023-04-21T22:30:13.435907+0000 mon.a (mon.0) 624 : cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log [7247107] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/osd 2-workunit/suites/fsstress}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/pdonnell-2023-04-21_02:33:35-fs-wip-pdonnell-testing-20230420.183701-quincy-distro-default-smithi/7247107/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-04-21_02:33:35-fs-wip-pdonnell-testing-20230420.183701-quincy-distro-default-smithi/7247107/ sentry: https://sentry.ceph.com/organizations/ceph/?query=3ddf7318328d4816bea8b8f45045b076 Command failed on smithi135 with status 2: 'sudo dpkg -i /tmp/linux- image.deb' [7247109] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/pdonnell-2023-04-21_02:33:35-fs-wip-pdonnell-testing-20230420.183701-quincy-distro-default-smithi/7247109/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-04-21_02:33:35-fs-wip-pdonnell-testing-20230420.183701-quincy-distro-default-smithi/7247109/ sentry: https://sentry.ceph.com/organizations/ceph/?query=37497dd4e84c4be7b3f465d30627dc3e Command failed on smithi135 with status 1: 'sudo yum install -y kernel' [7247120] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} kclient-overrides/{distro/testing/k-testing ms-die-on-skipped} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_untarbuild_blogbench} ----------------------------------------------------------------- time: 00:08:11 info: http://pulpito.ceph.com/pdonnell-2023-04-21_02:33:35-fs-wip-pdonnell-testing-20230420.183701-quincy-distro-default-smithi/7247120/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-04-21_02:33:35-fs-wip-pdonnell-testing-20230420.183701-quincy-distro-default-smithi/7247120/ sentry: https://sentry.ceph.com/organizations/ceph/?query=8b2c85c407194343b71cc06899aa8097 Failed to fetch package version from https://shaman.ceph.com/api/search/?st atus=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1 =58e06d348d8a2da339540be5425a40ec7683e512 [7247157] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/5 tasks/fsstress validater/valgrind} ----------------------------------------------------------------- time: 07:42:45 info: http://pulpito.ceph.com/pdonnell-2023-04-21_02:33:35-fs-wip-pdonnell-testing-20230420.183701-quincy-distro-default-smithi/7247157/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-04-21_02:33:35-fs-wip-pdonnell-testing-20230420.183701-quincy-distro-default-smithi/7247157/ sentry: https://sentry.ceph.com/organizations/ceph/?query=f7d2e939e8e44a80b2a6d12db968a99b Command failed (workunit test suites/fsstress.sh) on smithi066 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=58e06d348d8a2da339540be5425a40ec7683e512 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsstress.sh' [7247165] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/{crc} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/3 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/fsstress} wsync/{no}} ----------------------------------------------------------------- time: 00:21:40 info: http://pulpito.ceph.com/pdonnell-2023-04-21_02:33:35-fs-wip-pdonnell-testing-20230420.183701-quincy-distro-default-smithi/7247165/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-04-21_02:33:35-fs-wip-pdonnell-testing-20230420.183701-quincy-distro-default-smithi/7247165/ sentry: https://sentry.ceph.com/organizations/ceph/?query=d35a3610cb714143b8d5795741e8c63b Command failed on smithi151 with status 5: 'sudo systemctl stop ceph- be3e890a-e0a9-11ed-9b00-001a4aab830c@mon.b' [7247169] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/pdonnell-2023-04-21_02:33:35-fs-wip-pdonnell-testing-20230420.183701-quincy-distro-default-smithi/7247169/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-04-21_02:33:35-fs-wip-pdonnell-testing-20230420.183701-quincy-distro-default-smithi/7247169/ sentry: https://sentry.ceph.com/organizations/ceph/?query=595270a0e7b44d82ab553d4b58c06b22 Command failed on smithi118 with status 1: 'sudo yum install -y kernel' [7247177] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps} ----------------------------------------------------------------- time: 00:26:49 info: http://pulpito.ceph.com/pdonnell-2023-04-21_02:33:35-fs-wip-pdonnell-testing-20230420.183701-quincy-distro-default-smithi/7247177/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-04-21_02:33:35-fs-wip-pdonnell-testing-20230420.183701-quincy-distro-default-smithi/7247177/ sentry: https://sentry.ceph.com/organizations/ceph/?query=e23d1b0fb9e2426a92020da86dc87284 Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on smithi151 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=58e06d348d8a2da339540be5425a40ec7683e512 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest- multiple-capsnaps.sh' [7247182] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/pdonnell-2023-04-21_02:33:35-fs-wip-pdonnell-testing-20230420.183701-quincy-distro-default-smithi/7247182/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-04-21_02:33:35-fs-wip-pdonnell-testing-20230420.183701-quincy-distro-default-smithi/7247182/ sentry: https://sentry.ceph.com/organizations/ceph/?query=5e60336d680a440c9c4d533199ff7a52 Command failed on smithi112 with status 1: 'sudo yum install -y kernel' [7247186] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} kclient-overrides/{distro/testing/k-testing ms-die-on-skipped} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_untarbuild_blogbench} ----------------------------------------------------------------- time: 00:08:06 info: http://pulpito.ceph.com/pdonnell-2023-04-21_02:33:35-fs-wip-pdonnell-testing-20230420.183701-quincy-distro-default-smithi/7247186/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-04-21_02:33:35-fs-wip-pdonnell-testing-20230420.183701-quincy-distro-default-smithi/7247186/ sentry: https://sentry.ceph.com/organizations/ceph/?query=44a9160999134d8da331d2538981e754 Failed to fetch package version from https://shaman.ceph.com/api/search/?st atus=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1 =58e06d348d8a2da339540be5425a40ec7683e512 [7247194] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/mds-full} ----------------------------------------------------------------- time: 00:30:47 info: http://pulpito.ceph.com/pdonnell-2023-04-21_02:33:35-fs-wip-pdonnell-testing-20230420.183701-quincy-distro-default-smithi/7247194/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-04-21_02:33:35-fs-wip-pdonnell-testing-20230420.183701-quincy-distro-default-smithi/7247194/ "2023-04-22T02:06:36.021815+0000 osd.6 (osd.6) 3 : cluster [WRN] OSD bench result of 236279.412061 IOPS exceeded the threshold limit of 500.000000 IOPS for osd.6. IOPS capacity is unchanged at 315.000000 IOPS. The recommendation is to establish the osd's IOPS capacity using other benchmark tools (e.g. Fio) and then override osd_mclock_max_capacity_iops_[hdd|ssd]." in cluster log [7247223] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/3 tasks/fsstress validater/valgrind} ----------------------------------------------------------------- time: 06:48:23 info: http://pulpito.ceph.com/pdonnell-2023-04-21_02:33:35-fs-wip-pdonnell-testing-20230420.183701-quincy-distro-default-smithi/7247223/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-04-21_02:33:35-fs-wip-pdonnell-testing-20230420.183701-quincy-distro-default-smithi/7247223/ sentry: https://sentry.ceph.com/organizations/ceph/?query=5c1277be0f3740208b27464428c2d079 Command failed (workunit test suites/fsstress.sh) on smithi112 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=58e06d348d8a2da339540be5425a40ec7683e512 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsstress.sh' [7247237] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snap-schedule} ----------------------------------------------------------------- time: 01:49:41 info: http://pulpito.ceph.com/pdonnell-2023-04-21_02:33:35-fs-wip-pdonnell-testing-20230420.183701-quincy-distro-default-smithi/7247237/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-04-21_02:33:35-fs-wip-pdonnell-testing-20230420.183701-quincy-distro-default-smithi/7247237/ "2023-04-22T05:11:31.795136+0000 mon.a (mon.0) 1437 : cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log Dead ================================================================= [7247085] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_misc} ----------------------------------------------------------------- time: 00:36:13 info: http://pulpito.ceph.com/pdonnell-2023-04-21_02:33:35-fs-wip-pdonnell-testing-20230420.183701-quincy-distro-default-smithi/7247085/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-04-21_02:33:35-fs-wip-pdonnell-testing-20230420.183701-quincy-distro-default-smithi/7247085/ sentry: https://sentry.ceph.com/organizations/ceph/?query=7b0d31581c3e4be7b51b4f729d8ed34e {'smithi181.front.sepia.ceph.com': {'_ansible_no_log': False, 'changed': False, 'invocation': {'module_args': {'allow_downgrade': False, 'allowerasing': False, 'autoremove': False, 'bugfix': False, 'conf_file': None, 'disable_excludes': None, 'disable_gpg_check': False, 'disable_plugin': [], 'disablerepo': [], 'download_dir': None, 'download_only': False, 'enable_plugin': [], 'enablerepo': [], 'exclude': [], 'install_repoquery': True, 'install_weak_deps': True, 'installroot': '/', 'list': None, 'lock_timeout': 30, 'name': ['krb5-workstation'], 'releasever': None, 'security': False, 'skip_broken': False, 'state': 'present', 'update_cache': False, 'update_only': False, 'validate_certs': True}}, 'msg': "Failed to download metadata for repo 'CentOS-PowerTools': Yum repo downloading error: Downloading error(s): repodata/e830a7a4e881ef24 680d161802ae07874dd447031dd12d47f2d3d4a911245522-primary.xml.gz - Cannot download, all mirrors were already tried without success; repodata/144bb6d0 3f4cceafda5d5248f92ece7a8539e8858d490dfec5ebeb61d487bb20-filelists.xml.gz - Cannot download, all mirrors were already tried without success; repodata/38c2c78c3f89e4d347f07263805d33d8ae17df60ebab2e8a259218c57bcae2fb- comps-PowerTools.x86_64.xml - Cannot download, all mirrors were already tried without success", 'rc': 1, 'results': []}} [7247106] fs/upgrade/upgraded_client/from_nautilus/{bluestore-bitmap centos_latest clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-nautilus 1-client-upgrade 2-client-sanity}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/pdonnell-2023-04-21_02:33:35-fs-wip-pdonnell-testing-20230420.183701-quincy-distro-default-smithi/7247106/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-04-21_02:33:35-fs-wip-pdonnell-testing-20230420.183701-quincy-distro-default-smithi/7247106/ sentry: https://sentry.ceph.com/organizations/ceph/?query=17a0d91937774678b09b6911d7b920d1 SSH connection to smithi135 was lost: 'sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic' [7247112] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/pdonnell-2023-04-21_02:33:35-fs-wip-pdonnell-testing-20230420.183701-quincy-distro-default-smithi/7247112/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-04-21_02:33:35-fs-wip-pdonnell-testing-20230420.183701-quincy-distro-default-smithi/7247112/ sentry: https://sentry.ceph.com/organizations/ceph/?query=6348f06a17884621b28c52869e6d9243 SSH connection to smithi135 was lost: 'sudo grub2-mkconfig -o /boot/grub2/grub.cfg' [7247122] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/pdonnell-2023-04-21_02:33:35-fs-wip-pdonnell-testing-20230420.183701-quincy-distro-default-smithi/7247122/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-04-21_02:33:35-fs-wip-pdonnell-testing-20230420.183701-quincy-distro-default-smithi/7247122/ hit max job timeout [7247148] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/pdonnell-2023-04-21_02:33:35-fs-wip-pdonnell-testing-20230420.183701-quincy-distro-default-smithi/7247148/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-04-21_02:33:35-fs-wip-pdonnell-testing-20230420.183701-quincy-distro-default-smithi/7247148/ hit max job timeout [7247246] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} ----------------------------------------------------------------- time: 00:40:40 info: http://pulpito.ceph.com/pdonnell-2023-04-21_02:33:35-fs-wip-pdonnell-testing-20230420.183701-quincy-distro-default-smithi/7247246/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-04-21_02:33:35-fs-wip-pdonnell-testing-20230420.183701-quincy-distro-default-smithi/7247246/ sentry: https://sentry.ceph.com/organizations/ceph/?query=c22516165eb64c4eaef0912e27adf22e {'smithi012.front.sepia.ceph.com': {'_ansible_no_log': False, 'changed': False, 'invocation': {'module_args': {'allow_downgrade': False, 'allowerasing': False, 'autoremove': False, 'bugfix': False, 'conf_file': None, 'disable_excludes': None, 'disable_gpg_check': False, 'disable_plugin': [], 'disablerepo': [], 'download_dir': None, 'download_only': False, 'enable_plugin': [], 'enablerepo': [], 'exclude': [], 'install_repoquery': True, 'install_weak_deps': True, 'installroot': '/', 'list': None, 'lock_timeout': 30, 'name': ['krb5-workstation'], 'releasever': None, 'security': False, 'skip_broken': False, 'state': 'present', 'update_cache': False, 'update_only': False, 'validate_certs': True}}, 'msg': "Failed to download metadata for repo 'CentOS-PowerTools': Yum repo downloading error: Downloading error(s): repodata/e830a7a4e881ef24 680d161802ae07874dd447031dd12d47f2d3d4a911245522-primary.xml.gz - Cannot download, all mirrors were already tried without success", 'rc': 1, 'results': []}} Pass ================================================================= [7247068] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/{secure} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/5 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/pjd} wsync/{no}} time: 00:41:40 info: http://pulpito.ceph.com/pdonnell-2023-04-21_02:33:35-fs-wip-pdonnell-testing-20230420.183701-quincy-distro-default-smithi/7247068/ [7247069] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/recovery-fs} time: 00:11:33 info: http://pulpito.ceph.com/pdonnell-2023-04-21_02:33:35-fs-wip-pdonnell-testing-20230420.183701-quincy-distro-default-smithi/7247069/ [7247070] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_suites_fsstress}} time: 00:20:34 info: http://pulpito.ceph.com/pdonnell-2023-04-21_02:33:35-fs-wip-pdonnell-testing-20230420.183701-quincy-distro-default-smithi/7247070/ [7247071] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/pacific}} time: 01:45:23 info: http://pulpito.ceph.com/pdonnell-2023-04-21_02:33:35-fs-wip-pdonnell-testing-20230420.183701-quincy-distro-default-smithi/7247071/ [7247072] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/scrub} time: 00:25:15 info: http://pulpito.ceph.com/pdonnell-2023-04-21_02:33:35-fs-wip-pdonnell-testing-20230420.183701-quincy-distro-default-smithi/7247072/ [7247073] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mds 2-workunit/suites/pjd}} time: 00:24:35 info: http://pulpito.ceph.com/pdonnell-2023-04-21_02:33:35-fs-wip-pdonnell-testing-20230420.183701-quincy-distro-default-smithi/7247073/ [7247074] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/sessionmap} time: 00:18:30 info: http://pulpito.ceph.com/pdonnell-2023-04-21_02:33:35-fs-wip-pdonnell-testing-20230420.183701-quincy-distro-default-smithi/7247074/ [7247075] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress} time: 01:09:13 info: http://pulpito.ceph.com/pdonnell-2023-04-21_02:33:35-fs-wip-pdonnell-testing-20230420.183701-quincy-distro-default-smithi/7247075/ [7247076] fs/bugs/client_trim_caps/{begin/{0-install 1-ceph 2-logrotate} centos_latest clusters/small-cluster conf/{client mds mon osd} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/trim-i24137} time: 01:06:48 info: http://pulpito.ceph.com/pdonnell-2023-04-21_02:33:35-fs-wip-pdonnell-testing-20230420.183701-quincy-distro-default-smithi/7247076/ [7247077] fs/cephadm/multivolume/{0-start 1-mount 2-workload/dbench distro/single-container-host} time: 02:07:30 info: http://pulpito.ceph.com/pdonnell-2023-04-21_02:33:35-fs-wip-pdonnell-testing-20230420.183701-quincy-distro-default-smithi/7247077/ [7247078] fs/full/{begin/{0-install 1-ceph 2-logrotate} clusters/1-node-1-mds-1-osd conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-bitmap overrides overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/mgr-osd-full} time: 01:16:31 info: http://pulpito.ceph.com/pdonnell-2023-04-21_02:33:35-fs-wip-pdonnell-testing-20230420.183701-quincy-distro-default-smithi/7247078/ [7247079] fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{rhel_8} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/client} time: 00:13:02 info: http://pulpito.ceph.com/pdonnell-2023-04-21_02:33:35-fs-wip-pdonnell-testing-20230420.183701-quincy-distro-default-smithi/7247079/ [7247080] fs/mirror/{begin/{0-install 1-ceph 2-logrotate} cephfs-mirror/one-per-cluster clients/{mirror} cluster/{1-node} mount/fuse objectstore/bluestore-bitmap overrides/{whitelist_health} supported-random-distros$/{rhel_8} tasks/mirror} time: 02:02:27 info: http://pulpito.ceph.com/pdonnell-2023-04-21_02:33:35-fs-wip-pdonnell-testing-20230420.183701-quincy-distro-default-smithi/7247080/ [7247082] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} kclient-overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_untarbuild_blogbench} time: 00:51:16 info: http://pulpito.ceph.com/pdonnell-2023-04-21_02:33:35-fs-wip-pdonnell-testing-20230420.183701-quincy-distro-default-smithi/7247082/ [7247083] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cephfs_misc_tests} time: 00:47:58 info: http://pulpito.ceph.com/pdonnell-2023-04-21_02:33:35-fs-wip-pdonnell-testing-20230420.183701-quincy-distro-default-smithi/7247083/ [7247084] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/failover} time: 00:50:40 info: http://pulpito.ceph.com/pdonnell-2023-04-21_02:33:35-fs-wip-pdonnell-testing-20230420.183701-quincy-distro-default-smithi/7247084/ [7247086] fs/shell/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/cephfs-shell} time: 00:36:40 info: http://pulpito.ceph.com/pdonnell-2023-04-21_02:33:35-fs-wip-pdonnell-testing-20230420.183701-quincy-distro-default-smithi/7247086/ [7247087] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps} time: 00:53:20 info: http://pulpito.ceph.com/pdonnell-2023-04-21_02:33:35-fs-wip-pdonnell-testing-20230420.183701-quincy-distro-default-smithi/7247087/ [7247088] fs/top/{begin/{0-install 1-ceph 2-logrotate} cluster/{1-node} mount/fuse objectstore/bluestore-bitmap overrides/ignorelist_health supported-random-distros$/{rhel_8} tasks/fstop} time: 00:19:02 info: http://pulpito.ceph.com/pdonnell-2023-04-21_02:33:35-fs-wip-pdonnell-testing-20230420.183701-quincy-distro-default-smithi/7247088/ [7247089] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress traceless/50pc} time: 00:20:44 info: http://pulpito.ceph.com/pdonnell-2023-04-21_02:33:35-fs-wip-pdonnell-testing-20230420.183701-quincy-distro-default-smithi/7247089/ [7247091] fs/valgrind/{begin/{0-install 1-ceph 2-logrotate} centos_latest debug mirror/{cephfs-mirror/one-per-cluster clients/mirror cluster/1-node mount/fuse overrides/whitelist_health tasks/mirror}} time: 03:06:55 info: http://pulpito.ceph.com/pdonnell-2023-04-21_02:33:35-fs-wip-pdonnell-testing-20230420.183701-quincy-distro-default-smithi/7247091/ [7247092] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/1 tasks/fsstress validater/valgrind} time: 02:37:06 info: http://pulpito.ceph.com/pdonnell-2023-04-21_02:33:35-fs-wip-pdonnell-testing-20230420.183701-quincy-distro-default-smithi/7247092/ [7247093] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/basic}} time: 01:31:24 info: http://pulpito.ceph.com/pdonnell-2023-04-21_02:33:35-fs-wip-pdonnell-testing-20230420.183701-quincy-distro-default-smithi/7247093/ [7247094] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse ms_mode/{secure} objectstore-ec/bluestore-comp omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 scrub/no standby-replay tasks/{0-check-counter workunit/direct_io} wsync/{yes}} time: 00:32:21 info: http://pulpito.ceph.com/pdonnell-2023-04-21_02:33:35-fs-wip-pdonnell-testing-20230420.183701-quincy-distro-default-smithi/7247094/ [7247095] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mds 2-workunit/fs/snaps}} time: 00:53:47 info: http://pulpito.ceph.com/pdonnell-2023-04-21_02:33:35-fs-wip-pdonnell-testing-20230420.183701-quincy-distro-default-smithi/7247095/ [7247097] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}} time: 01:56:52 info: http://pulpito.ceph.com/pdonnell-2023-04-21_02:33:35-fs-wip-pdonnell-testing-20230420.183701-quincy-distro-default-smithi/7247097/ [7247098] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snap_schedule_snapdir} time: 00:12:08 info: http://pulpito.ceph.com/pdonnell-2023-04-21_02:33:35-fs-wip-pdonnell-testing-20230420.183701-quincy-distro-default-smithi/7247098/ [7247099] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/ffsb}} time: 00:44:20 info: http://pulpito.ceph.com/pdonnell-2023-04-21_02:33:35-fs-wip-pdonnell-testing-20230420.183701-quincy-distro-default-smithi/7247099/ [7247100] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/{secure} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/3 scrub/yes standby-replay tasks/{0-check-counter workunit/fs/misc} wsync/{yes}} time: 01:02:49 info: http://pulpito.ceph.com/pdonnell-2023-04-21_02:33:35-fs-wip-pdonnell-testing-20230420.183701-quincy-distro-default-smithi/7247100/ [7247101] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snapshots} time: 00:26:30 info: http://pulpito.ceph.com/pdonnell-2023-04-21_02:33:35-fs-wip-pdonnell-testing-20230420.183701-quincy-distro-default-smithi/7247101/ [7247102] fs/upgrade/nofs/{bluestore-bitmap centos_latest conf/{client mds mon osd} no-mds-cluster overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-octopus 1-upgrade}} time: 01:40:43 info: http://pulpito.ceph.com/pdonnell-2023-04-21_02:33:35-fs-wip-pdonnell-testing-20230420.183701-quincy-distro-default-smithi/7247102/ [7247103] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_suites_pjd}} time: 00:17:01 info: http://pulpito.ceph.com/pdonnell-2023-04-21_02:33:35-fs-wip-pdonnell-testing-20230420.183701-quincy-distro-default-smithi/7247103/ [7247104] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/strays} time: 01:23:57 info: http://pulpito.ceph.com/pdonnell-2023-04-21_02:33:35-fs-wip-pdonnell-testing-20230420.183701-quincy-distro-default-smithi/7247104/ [7247105] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/test_journal_migration} time: 00:18:14 info: http://pulpito.ceph.com/pdonnell-2023-04-21_02:33:35-fs-wip-pdonnell-testing-20230420.183701-quincy-distro-default-smithi/7247105/ [7247108] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} time: 00:15:16 info: http://pulpito.ceph.com/pdonnell-2023-04-21_02:33:35-fs-wip-pdonnell-testing-20230420.183701-quincy-distro-default-smithi/7247108/ [7247110] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/{secure} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/5 scrub/no standby-replay tasks/{0-check-counter workunit/kernel_untar_build} wsync/{yes}} time: 00:46:36 info: http://pulpito.ceph.com/pdonnell-2023-04-21_02:33:35-fs-wip-pdonnell-testing-20230420.183701-quincy-distro-default-smithi/7247110/ [7247111] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/truncate_delay} time: 00:15:48 info: http://pulpito.ceph.com/pdonnell-2023-04-21_02:33:35-fs-wip-pdonnell-testing-20230420.183701-quincy-distro-default-smithi/7247111/ [7247113] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mds 2-workunit/fs/trivial_sync}} time: 00:13:40 info: http://pulpito.ceph.com/pdonnell-2023-04-21_02:33:35-fs-wip-pdonnell-testing-20230420.183701-quincy-distro-default-smithi/7247113/ [7247114] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/workunit/dir-max-entries} time: 01:11:45 info: http://pulpito.ceph.com/pdonnell-2023-04-21_02:33:35-fs-wip-pdonnell-testing-20230420.183701-quincy-distro-default-smithi/7247114/ [7247115] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/ior-shared-file} time: 00:12:08 info: http://pulpito.ceph.com/pdonnell-2023-04-21_02:33:35-fs-wip-pdonnell-testing-20230420.183701-quincy-distro-default-smithi/7247115/ [7247116] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/multifs-auth} time: 00:22:15 info: http://pulpito.ceph.com/pdonnell-2023-04-21_02:33:35-fs-wip-pdonnell-testing-20230420.183701-quincy-distro-default-smithi/7247116/ [7247117] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps} time: 00:42:52 info: http://pulpito.ceph.com/pdonnell-2023-04-21_02:33:35-fs-wip-pdonnell-testing-20230420.183701-quincy-distro-default-smithi/7247117/ [7247118] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/clone}} time: 00:50:55 info: http://pulpito.ceph.com/pdonnell-2023-04-21_02:33:35-fs-wip-pdonnell-testing-20230420.183701-quincy-distro-default-smithi/7247118/ [7247119] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/workunit/quota} time: 00:20:34 info: http://pulpito.ceph.com/pdonnell-2023-04-21_02:33:35-fs-wip-pdonnell-testing-20230420.183701-quincy-distro-default-smithi/7247119/ [7247121] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_blogbench traceless/50pc} time: 00:21:42 info: http://pulpito.ceph.com/pdonnell-2023-04-21_02:33:35-fs-wip-pdonnell-testing-20230420.183701-quincy-distro-default-smithi/7247121/ [7247123] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu/{latest overrides}} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/3 tasks/dbench validater/lockdep} time: 01:18:04 info: http://pulpito.ceph.com/pdonnell-2023-04-21_02:33:35-fs-wip-pdonnell-testing-20230420.183701-quincy-distro-default-smithi/7247123/ [7247124] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse ms_mode/{secure} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/blogbench} wsync/{no}} time: 00:37:19 info: http://pulpito.ceph.com/pdonnell-2023-04-21_02:33:35-fs-wip-pdonnell-testing-20230420.183701-quincy-distro-default-smithi/7247124/ [7247125] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_trivial_sync}} time: 00:18:12 info: http://pulpito.ceph.com/pdonnell-2023-04-21_02:33:35-fs-wip-pdonnell-testing-20230420.183701-quincy-distro-default-smithi/7247125/ [7247126] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/acls} time: 01:19:32 info: http://pulpito.ceph.com/pdonnell-2023-04-21_02:33:35-fs-wip-pdonnell-testing-20230420.183701-quincy-distro-default-smithi/7247126/ [7247127] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/pacific}} time: 01:48:23 info: http://pulpito.ceph.com/pdonnell-2023-04-21_02:33:35-fs-wip-pdonnell-testing-20230420.183701-quincy-distro-default-smithi/7247127/ [7247128] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/iozone}} time: 00:18:15 info: http://pulpito.ceph.com/pdonnell-2023-04-21_02:33:35-fs-wip-pdonnell-testing-20230420.183701-quincy-distro-default-smithi/7247128/ [7247129] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/admin} time: 01:47:03 info: http://pulpito.ceph.com/pdonnell-2023-04-21_02:33:35-fs-wip-pdonnell-testing-20230420.183701-quincy-distro-default-smithi/7247129/ [7247130] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-comp overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress} time: 01:09:48 info: http://pulpito.ceph.com/pdonnell-2023-04-21_02:33:35-fs-wip-pdonnell-testing-20230420.183701-quincy-distro-default-smithi/7247130/ [7247131] fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{centos_8} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/ino_release_cb} time: 01:04:14 info: http://pulpito.ceph.com/pdonnell-2023-04-21_02:33:35-fs-wip-pdonnell-testing-20230420.183701-quincy-distro-default-smithi/7247131/ [7247132] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_misc} time: 00:14:12 info: http://pulpito.ceph.com/pdonnell-2023-04-21_02:33:35-fs-wip-pdonnell-testing-20230420.183701-quincy-distro-default-smithi/7247132/ [7247133] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/alternate-pool} time: 00:12:13 info: http://pulpito.ceph.com/pdonnell-2023-04-21_02:33:35-fs-wip-pdonnell-testing-20230420.183701-quincy-distro-default-smithi/7247133/ [7247134] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/{legacy} objectstore-ec/bluestore-comp omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/3 scrub/no standby-replay tasks/{0-check-counter workunit/suites/dbench} wsync/{yes}} time: 01:36:03 info: http://pulpito.ceph.com/pdonnell-2023-04-21_02:33:35-fs-wip-pdonnell-testing-20230420.183701-quincy-distro-default-smithi/7247134/ [7247135] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/osd 2-workunit/suites/pjd}} time: 00:18:47 info: http://pulpito.ceph.com/pdonnell-2023-04-21_02:33:35-fs-wip-pdonnell-testing-20230420.183701-quincy-distro-default-smithi/7247135/ [7247136] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}} time: 01:52:29 info: http://pulpito.ceph.com/pdonnell-2023-04-21_02:33:35-fs-wip-pdonnell-testing-20230420.183701-quincy-distro-default-smithi/7247136/ [7247137] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/asok_dump_tree} time: 01:20:55 info: http://pulpito.ceph.com/pdonnell-2023-04-21_02:33:35-fs-wip-pdonnell-testing-20230420.183701-quincy-distro-default-smithi/7247137/ [7247138] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/ffsb}} time: 00:28:03 info: http://pulpito.ceph.com/pdonnell-2023-04-21_02:33:35-fs-wip-pdonnell-testing-20230420.183701-quincy-distro-default-smithi/7247138/ [7247139] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/auto-repair} time: 01:06:47 info: http://pulpito.ceph.com/pdonnell-2023-04-21_02:33:35-fs-wip-pdonnell-testing-20230420.183701-quincy-distro-default-smithi/7247139/ [7247140] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}} time: 01:49:16 info: http://pulpito.ceph.com/pdonnell-2023-04-21_02:33:35-fs-wip-pdonnell-testing-20230420.183701-quincy-distro-default-smithi/7247140/ [7247141] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/{legacy} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/5 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/ffsb} wsync/{no}} time: 01:48:01 info: http://pulpito.ceph.com/pdonnell-2023-04-21_02:33:35-fs-wip-pdonnell-testing-20230420.183701-quincy-distro-default-smithi/7247141/ [7247142] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/backtrace} time: 00:14:57 info: http://pulpito.ceph.com/pdonnell-2023-04-21_02:33:35-fs-wip-pdonnell-testing-20230420.183701-quincy-distro-default-smithi/7247142/ [7247143] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/osd 2-workunit/fs/snaps}} time: 02:30:47 info: http://pulpito.ceph.com/pdonnell-2023-04-21_02:33:35-fs-wip-pdonnell-testing-20230420.183701-quincy-distro-default-smithi/7247143/ [7247144] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/mdtest} time: 00:14:06 info: http://pulpito.ceph.com/pdonnell-2023-04-21_02:33:35-fs-wip-pdonnell-testing-20230420.183701-quincy-distro-default-smithi/7247144/ [7247145] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/failover} time: 00:55:03 info: http://pulpito.ceph.com/pdonnell-2023-04-21_02:33:35-fs-wip-pdonnell-testing-20230420.183701-quincy-distro-default-smithi/7247145/ [7247146] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps} time: 00:44:34 info: http://pulpito.ceph.com/pdonnell-2023-04-21_02:33:35-fs-wip-pdonnell-testing-20230420.183701-quincy-distro-default-smithi/7247146/ [7247147] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/misc}} time: 00:22:59 info: http://pulpito.ceph.com/pdonnell-2023-04-21_02:33:35-fs-wip-pdonnell-testing-20230420.183701-quincy-distro-default-smithi/7247147/ [7247149] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/cap-flush} time: 01:09:18 info: http://pulpito.ceph.com/pdonnell-2023-04-21_02:33:35-fs-wip-pdonnell-testing-20230420.183701-quincy-distro-default-smithi/7247149/ [7247150] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mds 2-workunit/suites/ffsb}} time: 00:41:34 info: http://pulpito.ceph.com/pdonnell-2023-04-21_02:33:35-fs-wip-pdonnell-testing-20230420.183701-quincy-distro-default-smithi/7247150/ [7247151] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/client-limits} time: 00:20:58 info: http://pulpito.ceph.com/pdonnell-2023-04-21_02:33:35-fs-wip-pdonnell-testing-20230420.183701-quincy-distro-default-smithi/7247151/ [7247152] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} kclient-overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_dbench_iozone} time: 01:27:32 info: http://pulpito.ceph.com/pdonnell-2023-04-21_02:33:35-fs-wip-pdonnell-testing-20230420.183701-quincy-distro-default-smithi/7247152/ [7247153] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_dbench traceless/50pc} time: 00:50:13 info: http://pulpito.ceph.com/pdonnell-2023-04-21_02:33:35-fs-wip-pdonnell-testing-20230420.183701-quincy-distro-default-smithi/7247153/ [7247154] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} time: 02:02:55 info: http://pulpito.ceph.com/pdonnell-2023-04-21_02:33:35-fs-wip-pdonnell-testing-20230420.183701-quincy-distro-default-smithi/7247154/ [7247155] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-ec-root overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} time: 00:14:53 info: http://pulpito.ceph.com/pdonnell-2023-04-21_02:33:35-fs-wip-pdonnell-testing-20230420.183701-quincy-distro-default-smithi/7247155/ [7247156] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} time: 00:15:47 info: http://pulpito.ceph.com/pdonnell-2023-04-21_02:33:35-fs-wip-pdonnell-testing-20230420.183701-quincy-distro-default-smithi/7247156/ [7247158] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse ms_mode/{crc} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 scrub/no standby-replay tasks/{0-check-counter workunit/fs/norstats} wsync/{no}} time: 00:30:05 info: http://pulpito.ceph.com/pdonnell-2023-04-21_02:33:35-fs-wip-pdonnell-testing-20230420.183701-quincy-distro-default-smithi/7247158/ [7247159] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/client-readahead} time: 00:20:30 info: http://pulpito.ceph.com/pdonnell-2023-04-21_02:33:35-fs-wip-pdonnell-testing-20230420.183701-quincy-distro-default-smithi/7247159/ [7247160] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mon 2-workunit/suites/fsstress}} time: 01:14:00 info: http://pulpito.ceph.com/pdonnell-2023-04-21_02:33:35-fs-wip-pdonnell-testing-20230420.183701-quincy-distro-default-smithi/7247160/ [7247161] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}} time: 01:51:22 info: http://pulpito.ceph.com/pdonnell-2023-04-21_02:33:35-fs-wip-pdonnell-testing-20230420.183701-quincy-distro-default-smithi/7247161/ [7247162] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/client-recovery} time: 00:37:52 info: http://pulpito.ceph.com/pdonnell-2023-04-21_02:33:35-fs-wip-pdonnell-testing-20230420.183701-quincy-distro-default-smithi/7247162/ [7247163] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/iozone}} time: 01:11:24 info: http://pulpito.ceph.com/pdonnell-2023-04-21_02:33:35-fs-wip-pdonnell-testing-20230420.183701-quincy-distro-default-smithi/7247163/ [7247164] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/damage} time: 00:25:43 info: http://pulpito.ceph.com/pdonnell-2023-04-21_02:33:35-fs-wip-pdonnell-testing-20230420.183701-quincy-distro-default-smithi/7247164/ [7247166] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/pacific}} time: 01:45:10 info: http://pulpito.ceph.com/pdonnell-2023-04-21_02:33:35-fs-wip-pdonnell-testing-20230420.183701-quincy-distro-default-smithi/7247166/ [7247167] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/data-scan} time: 00:30:14 info: http://pulpito.ceph.com/pdonnell-2023-04-21_02:33:35-fs-wip-pdonnell-testing-20230420.183701-quincy-distro-default-smithi/7247167/ [7247168] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/osd 2-workunit/fs/trivial_sync}} time: 00:15:32 info: http://pulpito.ceph.com/pdonnell-2023-04-21_02:33:35-fs-wip-pdonnell-testing-20230420.183701-quincy-distro-default-smithi/7247168/ [7247170] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/exports} time: 00:47:25 info: http://pulpito.ceph.com/pdonnell-2023-04-21_02:33:35-fs-wip-pdonnell-testing-20230420.183701-quincy-distro-default-smithi/7247170/ [7247171] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-bitmap overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} time: 00:14:43 info: http://pulpito.ceph.com/pdonnell-2023-04-21_02:33:35-fs-wip-pdonnell-testing-20230420.183701-quincy-distro-default-smithi/7247171/ [7247172] fs/cephadm/renamevolume/{0-start 1-rename distro/single-container-host overrides/ignorelist_health} time: 01:42:42 info: http://pulpito.ceph.com/pdonnell-2023-04-21_02:33:35-fs-wip-pdonnell-testing-20230420.183701-quincy-distro-default-smithi/7247172/ [7247173] fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{rhel_8} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/libcephfs/{frag test}} time: 00:19:21 info: http://pulpito.ceph.com/pdonnell-2023-04-21_02:33:35-fs-wip-pdonnell-testing-20230420.183701-quincy-distro-default-smithi/7247173/ [7247174] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cephfs_misc_tests} time: 00:49:12 info: http://pulpito.ceph.com/pdonnell-2023-04-21_02:33:35-fs-wip-pdonnell-testing-20230420.183701-quincy-distro-default-smithi/7247174/ [7247175] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/multifs-auth} time: 00:29:51 info: http://pulpito.ceph.com/pdonnell-2023-04-21_02:33:35-fs-wip-pdonnell-testing-20230420.183701-quincy-distro-default-smithi/7247175/ [7247176] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} time: 00:14:44 info: http://pulpito.ceph.com/pdonnell-2023-04-21_02:33:35-fs-wip-pdonnell-testing-20230420.183701-quincy-distro-default-smithi/7247176/ [7247178] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/snapshot}} time: 00:36:10 info: http://pulpito.ceph.com/pdonnell-2023-04-21_02:33:35-fs-wip-pdonnell-testing-20230420.183701-quincy-distro-default-smithi/7247178/ [7247179] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mds 2-workunit/suites/iozone}} time: 00:19:29 info: http://pulpito.ceph.com/pdonnell-2023-04-21_02:33:35-fs-wip-pdonnell-testing-20230420.183701-quincy-distro-default-smithi/7247179/ [7247180] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/{crc} objectstore-ec/bluestore-comp omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/5 scrub/no standby-replay tasks/{0-check-counter workunit/suites/fsx} wsync/{yes}} time: 00:42:07 info: http://pulpito.ceph.com/pdonnell-2023-04-21_02:33:35-fs-wip-pdonnell-testing-20230420.183701-quincy-distro-default-smithi/7247180/ [7247181] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/forward-scrub} time: 00:20:57 info: http://pulpito.ceph.com/pdonnell-2023-04-21_02:33:35-fs-wip-pdonnell-testing-20230420.183701-quincy-distro-default-smithi/7247181/ [7247183] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/fragment} time: 00:19:22 info: http://pulpito.ceph.com/pdonnell-2023-04-21_02:33:35-fs-wip-pdonnell-testing-20230420.183701-quincy-distro-default-smithi/7247183/ [7247184] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_snaptests}} time: 01:55:55 info: http://pulpito.ceph.com/pdonnell-2023-04-21_02:33:35-fs-wip-pdonnell-testing-20230420.183701-quincy-distro-default-smithi/7247184/ [7247185] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/journal-repair} time: 00:24:52 info: http://pulpito.ceph.com/pdonnell-2023-04-21_02:33:35-fs-wip-pdonnell-testing-20230420.183701-quincy-distro-default-smithi/7247185/ [7247187] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_ffsb traceless/50pc} time: 01:01:11 info: http://pulpito.ceph.com/pdonnell-2023-04-21_02:33:35-fs-wip-pdonnell-testing-20230420.183701-quincy-distro-default-smithi/7247187/ [7247188] fs/upgrade/nofs/{bluestore-bitmap centos_latest conf/{client mds mon osd} no-mds-cluster overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-octopus 1-upgrade}} time: 01:34:42 info: http://pulpito.ceph.com/pdonnell-2023-04-21_02:33:35-fs-wip-pdonnell-testing-20230420.183701-quincy-distro-default-smithi/7247188/ [7247189] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/1 tasks/dbench validater/lockdep} time: 02:30:35 info: http://pulpito.ceph.com/pdonnell-2023-04-21_02:33:35-fs-wip-pdonnell-testing-20230420.183701-quincy-distro-default-smithi/7247189/ [7247190] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse ms_mode/{secure} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/fsync-tester} wsync/{yes}} time: 00:32:00 info: http://pulpito.ceph.com/pdonnell-2023-04-21_02:33:35-fs-wip-pdonnell-testing-20230420.183701-quincy-distro-default-smithi/7247190/ [7247191] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mon 2-workunit/suites/pjd}} time: 01:16:21 info: http://pulpito.ceph.com/pdonnell-2023-04-21_02:33:35-fs-wip-pdonnell-testing-20230420.183701-quincy-distro-default-smithi/7247191/ [7247192] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/mds-flush} time: 00:18:42 info: http://pulpito.ceph.com/pdonnell-2023-04-21_02:33:35-fs-wip-pdonnell-testing-20230420.183701-quincy-distro-default-smithi/7247192/ [7247193] fs/upgrade/upgraded_client/from_nautilus/{bluestore-bitmap centos_latest clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-nautilus 1-client-upgrade 2-client-sanity}} time: 01:50:40 info: http://pulpito.ceph.com/pdonnell-2023-04-21_02:33:35-fs-wip-pdonnell-testing-20230420.183701-quincy-distro-default-smithi/7247193/ [7247195] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mon 2-workunit/fs/snaps}} time: 00:50:45 info: http://pulpito.ceph.com/pdonnell-2023-04-21_02:33:35-fs-wip-pdonnell-testing-20230420.183701-quincy-distro-default-smithi/7247195/ [7247196] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress} time: 00:25:55 info: http://pulpito.ceph.com/pdonnell-2023-04-21_02:33:35-fs-wip-pdonnell-testing-20230420.183701-quincy-distro-default-smithi/7247196/ [7247197] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_misc} time: 00:14:27 info: http://pulpito.ceph.com/pdonnell-2023-04-21_02:33:35-fs-wip-pdonnell-testing-20230420.183701-quincy-distro-default-smithi/7247197/ [7247198] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/mds_creation_retry} time: 00:17:46 info: http://pulpito.ceph.com/pdonnell-2023-04-21_02:33:35-fs-wip-pdonnell-testing-20230420.183701-quincy-distro-default-smithi/7247198/ [7247199] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/{secure} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/3 scrub/no standby-replay tasks/{0-check-counter workunit/fs/test_o_trunc} wsync/{yes}} time: 00:37:52 info: http://pulpito.ceph.com/pdonnell-2023-04-21_02:33:35-fs-wip-pdonnell-testing-20230420.183701-quincy-distro-default-smithi/7247199/ [7247200] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}} time: 01:53:48 info: http://pulpito.ceph.com/pdonnell-2023-04-21_02:33:35-fs-wip-pdonnell-testing-20230420.183701-quincy-distro-default-smithi/7247200/ [7247201] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/ffsb}} time: 00:49:35 info: http://pulpito.ceph.com/pdonnell-2023-04-21_02:33:35-fs-wip-pdonnell-testing-20230420.183701-quincy-distro-default-smithi/7247201/ [7247202] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/metrics} time: 01:28:53 info: http://pulpito.ceph.com/pdonnell-2023-04-21_02:33:35-fs-wip-pdonnell-testing-20230420.183701-quincy-distro-default-smithi/7247202/ [7247203] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/ior-shared-file} time: 00:11:51 info: http://pulpito.ceph.com/pdonnell-2023-04-21_02:33:35-fs-wip-pdonnell-testing-20230420.183701-quincy-distro-default-smithi/7247203/ [7247204] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/failover} time: 00:50:32 info: http://pulpito.ceph.com/pdonnell-2023-04-21_02:33:35-fs-wip-pdonnell-testing-20230420.183701-quincy-distro-default-smithi/7247204/ [7247205] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps} time: 00:31:02 info: http://pulpito.ceph.com/pdonnell-2023-04-21_02:33:35-fs-wip-pdonnell-testing-20230420.183701-quincy-distro-default-smithi/7247205/ [7247206] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/basic}} time: 01:30:24 info: http://pulpito.ceph.com/pdonnell-2023-04-21_02:33:35-fs-wip-pdonnell-testing-20230420.183701-quincy-distro-default-smithi/7247206/ [7247207] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} time: 02:13:28 info: http://pulpito.ceph.com/pdonnell-2023-04-21_02:33:35-fs-wip-pdonnell-testing-20230420.183701-quincy-distro-default-smithi/7247207/ [7247208] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/multimds_misc} time: 00:25:00 info: http://pulpito.ceph.com/pdonnell-2023-04-21_02:33:35-fs-wip-pdonnell-testing-20230420.183701-quincy-distro-default-smithi/7247208/ [7247209] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_suites_fsstress}} time: 00:26:21 info: http://pulpito.ceph.com/pdonnell-2023-04-21_02:33:35-fs-wip-pdonnell-testing-20230420.183701-quincy-distro-default-smithi/7247209/ [7247210] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/openfiletable} time: 00:17:08 info: http://pulpito.ceph.com/pdonnell-2023-04-21_02:33:35-fs-wip-pdonnell-testing-20230420.183701-quincy-distro-default-smithi/7247210/ [7247211] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/{secure} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/5 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/iogen} wsync/{no}} time: 00:52:00 info: http://pulpito.ceph.com/pdonnell-2023-04-21_02:33:35-fs-wip-pdonnell-testing-20230420.183701-quincy-distro-default-smithi/7247211/ [7247212] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} time: 01:50:43 info: http://pulpito.ceph.com/pdonnell-2023-04-21_02:33:35-fs-wip-pdonnell-testing-20230420.183701-quincy-distro-default-smithi/7247212/ [7247213] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mds 2-workunit/suites/fsstress}} time: 00:22:14 info: http://pulpito.ceph.com/pdonnell-2023-04-21_02:33:35-fs-wip-pdonnell-testing-20230420.183701-quincy-distro-default-smithi/7247213/ [7247214] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/pool-perm} time: 01:08:46 info: http://pulpito.ceph.com/pdonnell-2023-04-21_02:33:35-fs-wip-pdonnell-testing-20230420.183701-quincy-distro-default-smithi/7247214/ [7247215] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} kclient-overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_dbench_iozone} time: 01:33:17 info: http://pulpito.ceph.com/pdonnell-2023-04-21_02:33:35-fs-wip-pdonnell-testing-20230420.183701-quincy-distro-default-smithi/7247215/ [7247216] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress traceless/50pc} time: 00:16:32 info: http://pulpito.ceph.com/pdonnell-2023-04-21_02:33:35-fs-wip-pdonnell-testing-20230420.183701-quincy-distro-default-smithi/7247216/ [7247217] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/quota} time: 00:13:10 info: http://pulpito.ceph.com/pdonnell-2023-04-21_02:33:35-fs-wip-pdonnell-testing-20230420.183701-quincy-distro-default-smithi/7247217/ [7247218] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/pacific}} time: 01:55:55 info: http://pulpito.ceph.com/pdonnell-2023-04-21_02:33:35-fs-wip-pdonnell-testing-20230420.183701-quincy-distro-default-smithi/7247218/ [7247219] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mon 2-workunit/fs/trivial_sync}} time: 00:17:00 info: http://pulpito.ceph.com/pdonnell-2023-04-21_02:33:35-fs-wip-pdonnell-testing-20230420.183701-quincy-distro-default-smithi/7247219/ [7247220] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} time: 00:14:25 info: http://pulpito.ceph.com/pdonnell-2023-04-21_02:33:35-fs-wip-pdonnell-testing-20230420.183701-quincy-distro-default-smithi/7247220/ [7247221] fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{ubuntu_latest} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/libcephfs_python} time: 00:11:56 info: http://pulpito.ceph.com/pdonnell-2023-04-21_02:33:35-fs-wip-pdonnell-testing-20230420.183701-quincy-distro-default-smithi/7247221/ [7247222] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} time: 01:17:31 info: http://pulpito.ceph.com/pdonnell-2023-04-21_02:33:35-fs-wip-pdonnell-testing-20230420.183701-quincy-distro-default-smithi/7247222/ [7247224] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse ms_mode/{legacy} objectstore-ec/bluestore-comp omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 scrub/no standby-replay tasks/{0-check-counter workunit/suites/iozone} wsync/{yes}} time: 00:40:50 info: http://pulpito.ceph.com/pdonnell-2023-04-21_02:33:35-fs-wip-pdonnell-testing-20230420.183701-quincy-distro-default-smithi/7247224/ [7247225] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/recovery-fs} time: 00:13:16 info: http://pulpito.ceph.com/pdonnell-2023-04-21_02:33:35-fs-wip-pdonnell-testing-20230420.183701-quincy-distro-default-smithi/7247225/ [7247226] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_suites_pjd}} time: 01:18:33 info: http://pulpito.ceph.com/pdonnell-2023-04-21_02:33:35-fs-wip-pdonnell-testing-20230420.183701-quincy-distro-default-smithi/7247226/ [7247227] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}} time: 01:55:39 info: http://pulpito.ceph.com/pdonnell-2023-04-21_02:33:35-fs-wip-pdonnell-testing-20230420.183701-quincy-distro-default-smithi/7247227/ [7247228] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/scrub} time: 00:27:46 info: http://pulpito.ceph.com/pdonnell-2023-04-21_02:33:35-fs-wip-pdonnell-testing-20230420.183701-quincy-distro-default-smithi/7247228/ [7247229] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/sessionmap} time: 00:14:44 info: http://pulpito.ceph.com/pdonnell-2023-04-21_02:33:35-fs-wip-pdonnell-testing-20230420.183701-quincy-distro-default-smithi/7247229/ [7247230] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/iozone}} time: 00:21:05 info: http://pulpito.ceph.com/pdonnell-2023-04-21_02:33:35-fs-wip-pdonnell-testing-20230420.183701-quincy-distro-default-smithi/7247230/ [7247231] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}} time: 01:55:29 info: http://pulpito.ceph.com/pdonnell-2023-04-21_02:33:35-fs-wip-pdonnell-testing-20230420.183701-quincy-distro-default-smithi/7247231/ [7247232] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/{crc} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/3 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/pjd} wsync/{no}} time: 00:39:12 info: http://pulpito.ceph.com/pdonnell-2023-04-21_02:33:35-fs-wip-pdonnell-testing-20230420.183701-quincy-distro-default-smithi/7247232/ [7247233] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/mdtest} time: 00:18:03 info: http://pulpito.ceph.com/pdonnell-2023-04-21_02:33:35-fs-wip-pdonnell-testing-20230420.183701-quincy-distro-default-smithi/7247233/ [7247234] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/multifs-auth} time: 00:31:13 info: http://pulpito.ceph.com/pdonnell-2023-04-21_02:33:35-fs-wip-pdonnell-testing-20230420.183701-quincy-distro-default-smithi/7247234/ [7247235] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps} time: 01:52:35 info: http://pulpito.ceph.com/pdonnell-2023-04-21_02:33:35-fs-wip-pdonnell-testing-20230420.183701-quincy-distro-default-smithi/7247235/ [7247236] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/clone}} time: 00:56:20 info: http://pulpito.ceph.com/pdonnell-2023-04-21_02:33:35-fs-wip-pdonnell-testing-20230420.183701-quincy-distro-default-smithi/7247236/ [7247238] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mds 2-workunit/suites/pjd}} time: 00:24:23 info: http://pulpito.ceph.com/pdonnell-2023-04-21_02:33:35-fs-wip-pdonnell-testing-20230420.183701-quincy-distro-default-smithi/7247238/ [7247239] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} time: 02:07:32 info: http://pulpito.ceph.com/pdonnell-2023-04-21_02:33:35-fs-wip-pdonnell-testing-20230420.183701-quincy-distro-default-smithi/7247239/ [7247240] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snap_schedule_snapdir} time: 01:08:34 info: http://pulpito.ceph.com/pdonnell-2023-04-21_02:33:35-fs-wip-pdonnell-testing-20230420.183701-quincy-distro-default-smithi/7247240/ [7247241] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-ec-root overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress} time: 01:19:35 info: http://pulpito.ceph.com/pdonnell-2023-04-21_02:33:35-fs-wip-pdonnell-testing-20230420.183701-quincy-distro-default-smithi/7247241/ [7247242] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_misc} time: 00:15:11 info: http://pulpito.ceph.com/pdonnell-2023-04-21_02:33:35-fs-wip-pdonnell-testing-20230420.183701-quincy-distro-default-smithi/7247242/ [7247243] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snapshots} time: 00:32:58 info: http://pulpito.ceph.com/pdonnell-2023-04-21_02:33:35-fs-wip-pdonnell-testing-20230420.183701-quincy-distro-default-smithi/7247243/ [7247244] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_trivial_sync}} time: 00:14:25 info: http://pulpito.ceph.com/pdonnell-2023-04-21_02:33:35-fs-wip-pdonnell-testing-20230420.183701-quincy-distro-default-smithi/7247244/ [7247245] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/{crc} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/5 scrub/no standby-replay tasks/{0-check-counter workunit/direct_io} wsync/{yes}} time: 00:35:40 info: http://pulpito.ceph.com/pdonnell-2023-04-21_02:33:35-fs-wip-pdonnell-testing-20230420.183701-quincy-distro-default-smithi/7247245/ [7247247] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/strays} time: 00:33:18 info: http://pulpito.ceph.com/pdonnell-2023-04-21_02:33:35-fs-wip-pdonnell-testing-20230420.183701-quincy-distro-default-smithi/7247247/ [7247248] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mds 2-workunit/fs/snaps}} time: 01:29:58 info: http://pulpito.ceph.com/pdonnell-2023-04-21_02:33:35-fs-wip-pdonnell-testing-20230420.183701-quincy-distro-default-smithi/7247248/