2022-12-20T12:15:42.363 INFO:root:teuthology version: 0.0.1.dev18+g19d18db 2022-12-20T12:15:42.364 INFO:teuthology.results:subset: '246/1024' 2022-12-20T12:15:42.364 INFO:teuthology.results:seed: '9877' 2022-12-20T18:20:42.991 INFO:root:teuthology version: 0.0.1.dev18+g19d18db 2022-12-20T18:20:43.017 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2022-12-21T00:30:23.672 INFO:teuthology.results:Tests finished! gathering results... 2022-12-21T00:30:38.399 INFO:scrape:Found 197 jobs 2022-12-21T00:30:38.400 WARNING:scrape:Missing teuthology log /home/teuthworker/archive/vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi/7122274/teuthology.log 2022-12-21T00:30:57.484 INFO:scrape:Found 14 distinct failure reasons 2022-12-21T00:30:57.484 INFO:scrape:Dead: None 2022-12-21T00:30:57.484 INFO:scrape:22 jobs: ['7122274', '7122303', '7122378', '7122204', '7122313', '7122253', '7122252', '7122316', '7122223', '7122376', '7122222', '7122241', '7122293', '7122288', '7122263', '7122245', '7122225', '7122231', '7122200', '7122306', '7122267', '7122205'] 2022-12-21T00:30:57.485 INFO:scrape: 2022-12-21T00:30:57.485 INFO:scrape:Failure: Command failed on smithi160 with status 22: "sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 14d8f512-807d-11ed-97a3-001a4aab830c -e sha1=47f77a750cb9b297b7b8ab2c8bc3d22102dbeba8 -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types mgr'" 2022-12-21T00:30:57.485 INFO:scrape:5 jobs: ['7122297', '7122321', '7122382', '7122268', '7122212'] 2022-12-21T00:30:57.485 INFO:scrape:suites intersection: ['1-volume/{0-create', '2-client', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_8.stream_container_tools', 'conf/{client', 'fail_fs/yes', 'fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'syntax}', 'tasks/{0-from/v16.2.4'] 2022-12-21T00:30:57.485 INFO:scrape:suites union: ['1-ranks/1', '1-ranks/2', '1-volume/{0-create', '2-allow_standby_replay/no', '2-allow_standby_replay/yes', '2-client', '3-inline/no', '3-inline/yes', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_8.stream_container_tools', 'conf/{client', 'fail_fs/yes', 'fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'syntax}', 'tasks/{0-from/v16.2.4'] 2022-12-21T00:30:57.485 INFO:scrape: 2022-12-21T00:30:57.485 INFO:scrape:Failure: error during scrub thrashing: rank damage found: {'backtrace'} 2022-12-21T00:30:57.486 INFO:scrape:4 jobs: ['7122269', '7122377', '7122310', '7122290'] 2022-12-21T00:30:57.486 INFO:scrape:suites intersection: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/yes', '3-snaps/no', '4-flush/yes', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'ranks/multi/{export-check', 'session_timeout}', 'standby-replay'] 2022-12-21T00:30:57.486 INFO:scrape:suites union: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/yes', '3-snaps/no', '4-flush/yes', '5-workunit/kernel_untar_build}}', '5-workunit/suites/blogbench}}', '5-workunit/suites/ffsb}}', '5-workunit/suites/iogen}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{base/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'ms_mode/legacy', 'n/3', 'n/5', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/stock/{k-stock', 'ranks/multi/{export-check', 'replication/always}', 'replication/default}', 'rhel_8}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-namespace-isolated-and-quota}', 'tasks/{0-subvolume/{with-namespace-isolated}', 'tasks/{0-subvolume/{with-quota}', 'wsync/no}'] 2022-12-21T00:30:57.486 INFO:scrape: 2022-12-21T00:30:57.486 INFO:scrape:Failure: Test failure: test_pool_perm (tasks.cephfs.test_pool_perm.TestPoolPerm) 2022-12-21T00:30:57.486 INFO:scrape:1 jobs: ['7122354'] 2022-12-21T00:30:57.486 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{rhel_8}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{ignorelist_health', 'tasks/pool-perm}'] 2022-12-21T00:30:57.487 INFO:scrape: 2022-12-21T00:30:57.487 INFO:scrape:Failure: Test failure: test_cephfs_mirror_restart_sync_on_blocklist (tasks.cephfs.test_mirroring.TestMirroring) 2022-12-21T00:30:57.487 INFO:scrape:1 jobs: ['7122337'] 2022-12-21T00:30:57.487 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'centos_latest', 'clients/mirror', 'cluster/1-node', 'debug', 'fs/valgrind/{begin/{0-install', 'mirror/{cephfs-mirror/one-per-cluster', 'mount/fuse', 'overrides/whitelist_health', 'tasks/mirror}}'] 2022-12-21T00:30:57.487 INFO:scrape: 2022-12-21T00:30:57.487 INFO:scrape:Failure: Command failed (workunit test suites/dbench.sh) on smithi031 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=47f77a750cb9b297b7b8ab2c8bc3d22102dbeba8 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/dbench.sh' 2022-12-21T00:30:57.487 INFO:scrape:1 jobs: ['7122300'] 2022-12-21T00:30:57.488 INFO:scrape:suites: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/no', '3-snaps/yes', '4-flush/no', '5-workunit/suites/dbench}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'ms_mode/crc', 'n/5', 'objectstore-ec/bluestore-ec-root', 'omap_limit/10', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/testing/k-testing', 'ranks/multi/{export-check', 'replication/default}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{no-subvolume}', 'wsync/yes}'] 2022-12-21T00:30:57.488 INFO:scrape: 2022-12-21T00:30:57.488 INFO:scrape:Failure: {'smithi187.front.sepia.ceph.com': {'_ansible_no_log': False, 'changed': True, 'cmd': ['cpan', 'Amazon::S3'], 'delta': '0:00:01.658378', 'end': '2022-12-20 13:36:00.254158', 'invocation': {'module_args': {'_raw_params': 'cpan Amazon::S3', '_uses_shell': False, 'argv': None, 'chdir': None, 'creates': None, 'executable': None, 'removes': None, 'stdin': None, 'stdin_add_newline': True, 'strip_empty_ends': True, 'warn': True}}, 'msg': 'non-zero return code', 'rc': 25, 'start': '2022-12-20 13:35:58.595780', 'stderr': '', 'stderr_lines': [], 'stdout': "Loading internal logger. Log::Log4perl recommended for better logging\nReading '/home/ubuntu/.cpan/Metadata'\n Database was generated on Fri, 12 Feb 2016 02:17:02 GMT\nRunning install for module 'Amazon::S3'\n\nWarning: checksum file '/home/ubuntu/.cpan/sources/authors/id/T/TI/TIMA/CHECKSUMS' not conforming.\n\nThe cksum does not contain the key 'cpan_path' for 'Amazon-S3-0.45.tar.gz'.\nProceed nonetheless? [no] no\nAborted.", 'stdout_lines': ['Loading internal logger. Log::Log4perl recommended for better logging', "Reading '/home/ubuntu/.cpan/Metadata'", ' Database was generated on Fri, 12 Feb 2016 02:17:02 GMT', "Running install for module 'Amazon::S3'", '', "Warning: checksum file '/home/ubuntu/.cpan/sources/authors/id/T/TI/TIMA/CHECKSUMS' not conforming.", '', "The cksum does not contain the key 'cpan_path' for 'Amazon-S3-0.45.tar.gz'.", 'Proceed nonetheless? [no] no', 'Aborted.']}, 'smithi131.front.sepia.ceph.com': {'_ansible_no_log': False, 'changed': True, 'cmd': ['cpan', 'Amazon::S3'], 'delta': '0:00:01.791174', 'end': '2022-12-20 13:36:01.275037', 'invocation': {'module_args': {'_raw_params': 'cpan Amazon::S3', '_uses_shell': False, 'argv': None, 'chdir': None, 'creates': None, 'executable': None, 'removes': None, 'stdin': None, 'stdin_add_newline': True, 'strip_empty_ends': True, 'warn': True}}, 'msg': 'non-zero return code', 'rc': 25, 'start': '2022-12-20 13:35:59.483863', 'stderr': '', 'stderr_lines': [], 'stdout': "Loading internal logger. Log::Log4perl recommended for better logging\nReading '/home/ubuntu/.cpan/Metadata'\n Database was generated on Fri, 12 Feb 2016 02:17:02 GMT\nRunning install for module 'Amazon::S3'\n\nWarning: checksum file '/home/ubuntu/.cpan/sources/authors/id/T/TI/TIMA/CHECKSUMS' not conforming.\n\nThe cksum does not contain the key 'cpan_path' for 'Amazon-S3-0.45.tar.gz'.\nProceed nonetheless? [no] no\nAborted.", 'stdout_lines': ['Loading internal logger. Log::Log4perl recommended for better logging', "Reading '/home/ubuntu/.cpan/Metadata'", ' Database was generated on Fri, 12 Feb 2016 02:17:02 GMT', "Running install for module 'Amazon::S3'", '', "Warning: checksum file '/home/ubuntu/.cpan/sources/authors/id/T/TI/TIMA/CHECKSUMS' not conforming.", '', "The cksum does not contain the key 'cpan_path' for 'Amazon-S3-0.45.tar.gz'.", 'Proceed nonetheless? [no] no', 'Aborted.']}} 2022-12-21T00:30:57.488 INFO:scrape:2 jobs: ['7122270', '7122349'] 2022-12-21T00:30:57.488 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-2c-client', 'conf/{client', 'fs/mixed-clients/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'kclient-overrides/{distro/testing/k-testing', 'mds', 'mon', 'ms-die-on-skipped}', 'osd-asserts}', 'osd}', 'overrides/{ignorelist_health', 'tasks/kernel_cfuse_workunits_untarbuild_blogbench}'] 2022-12-21T00:30:57.488 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-2c-client', 'conf/{client', 'fs/mixed-clients/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'kclient-overrides/{distro/testing/k-testing', 'mds', 'mon', 'ms-die-on-skipped}', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'osd-asserts}', 'osd}', 'overrides/{ignorelist_health', 'tasks/kernel_cfuse_workunits_untarbuild_blogbench}'] 2022-12-21T00:30:57.488 INFO:scrape: 2022-12-21T00:30:57.488 INFO:scrape:Failure: reached maximum tries (50) after waiting for 300 seconds 2022-12-21T00:30:57.489 INFO:scrape:1 jobs: ['7122330'] 2022-12-21T00:30:57.489 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'cephfs-mirror/three-per-cluster', 'clients/{mirror}', 'cluster/{1-node}', 'fs/mirror-ha/{begin/{0-install', 'objectstore/bluestore-bitmap', 'overrides/{whitelist_health}', 'supported-random-distro$/{centos_8}', 'workloads/cephfs-mirror-ha-workunit}'] 2022-12-21T00:30:57.489 INFO:scrape: 2022-12-21T00:30:57.489 INFO:scrape:Failure: Test failure: test_open_ino_errors (tasks.cephfs.test_damage.TestDamage) 2022-12-21T00:30:57.489 INFO:scrape:1 jobs: ['7122292'] 2022-12-21T00:30:57.489 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{ignorelist_health', 'tasks/damage}'] 2022-12-21T00:30:57.489 INFO:scrape: 2022-12-21T00:30:57.489 INFO:scrape:Failure: Command failed (workunit test suites/pjd.sh) on smithi052 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=47f77a750cb9b297b7b8ab2c8bc3d22102dbeba8 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/pjd.sh' 2022-12-21T00:30:57.490 INFO:scrape:1 jobs: ['7122243'] 2022-12-21T00:30:57.490 INFO:scrape:suites: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/no', '3-snaps/yes', '4-flush/no', '5-workunit/suites/pjd}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'ms_mode/secure', 'n/3', 'objectstore-ec/bluestore-comp-ec-root', 'omap_limit/10', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/testing/k-testing', 'ranks/multi/{export-check', 'replication/always}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-quota}', 'wsync/yes}'] 2022-12-21T00:30:57.490 INFO:scrape: 2022-12-21T00:30:57.490 INFO:scrape:Failure: Test failure: test_client_cache_size (tasks.cephfs.test_client_limits.TestClientLimits) 2022-12-21T00:30:57.490 INFO:scrape:1 jobs: ['7122281'] 2022-12-21T00:30:57.490 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'tasks/client-limits}'] 2022-12-21T00:30:57.490 INFO:scrape: 2022-12-21T00:30:57.490 INFO:scrape:Failure: Command failed (workunit test fs/snaps/snaptest-git-ceph.sh) on smithi192 with status 128: 'mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=47f77a750cb9b297b7b8ab2c8bc3d22102dbeba8 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.1/qa/workunits/fs/snaps/snaptest-git-ceph.sh' 2022-12-21T00:30:57.490 INFO:scrape:1 jobs: ['7122239'] 2022-12-21T00:30:57.491 INFO:scrape:suites: ['1-ceph', '2-logrotate}', '2-workunit/cfuse_workunit_snaptests}}', 'clusters/1a3s-mds-2c-client', 'conf/{client', 'distro/{centos_8}', 'fs/thrash/multifs/{begin/{0-install', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'msgr-failures/osd-mds-delay', 'multifs', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{frag', 'session_timeout', 'tasks/{1-thrash/mds', 'thrashosds-health}'] 2022-12-21T00:30:57.491 INFO:scrape: 2022-12-21T00:30:57.491 INFO:scrape:Timeout 6h running clone.client.0/qa/workunits/suites/fsstress.sh 2022-12-21T00:30:57.491 INFO:scrape:1 jobs: ['7122355'] 2022-12-21T00:30:57.491 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{centos_8}', 'fs/verify/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mon-debug', 'mount', 'mount/kclient/{k-testing', 'ms-die-on-skipped}', 'objectstore-ec/bluestore-comp', 'osd}', 'overrides/{ignorelist_health', 'ranks/3', 'session_timeout}', 'tasks/fsstress', 'validater/valgrind}'] 2022-12-21T00:30:57.491 INFO:scrape: 2022-12-21T00:30:57.491 INFO:scrape:Failure: Test failure: test_scrub_backtrace (tasks.cephfs.test_scrub.TestScrub) 2022-12-21T00:30:57.491 INFO:scrape:1 jobs: ['7122371'] 2022-12-21T00:30:57.492 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'tasks/scrub}'] 2022-12-21T00:30:57.492 INFO:scrape: 2022-12-21T00:30:57.499 INFO:teuthology.results:Sending results to vshankar@redhat.com: Test Run: vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi ================================================================= info: http://pulpito.ceph.com/vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi/ failed: 19 dead: 24 running: 0 waiting: 0 queued: 0 passed: 154 Fail ================================================================= [7122212] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:17:24 info: http://pulpito.ceph.com/vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi/7122212/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi/7122212/ sentry: https://sentry.ceph.com/organizations/ceph/?query=872073153a834799bd306c84358126a3 Command failed on smithi167 with status 22: "sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38f3e41c-8063-11ed-97a3-001a4aab830c -e sha1=47f77a750cb9b297b7b8ab2c8bc3d22102dbeba8 -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types mgr'" [7122239] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_snaptests}} ----------------------------------------------------------------- time: 00:36:11 info: http://pulpito.ceph.com/vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi/7122239/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi/7122239/ sentry: https://sentry.ceph.com/organizations/ceph/?query=ffe1f88b80ab40d0890de43f327a97f1 Command failed (workunit test fs/snaps/snaptest-git-ceph.sh) on smithi192 with status 128: 'mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=47f77a750cb9b297b7b8ab2c8bc3d22102dbeba8 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.1/qa/workunits/fs/snaps/snaptest-git- ceph.sh' [7122243] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/suites/pjd}} ----------------------------------------------------------------- time: 00:33:27 info: http://pulpito.ceph.com/vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi/7122243/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi/7122243/ sentry: https://sentry.ceph.com/organizations/ceph/?query=acb3e52db6a24c16a283d8c5a5e75eba Command failed (workunit test suites/pjd.sh) on smithi052 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=47f77a750cb9b297b7b8ab2c8bc3d22102dbeba8 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/pjd.sh' [7122268] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:17:59 info: http://pulpito.ceph.com/vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi/7122268/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi/7122268/ sentry: https://sentry.ceph.com/organizations/ceph/?query=9ff520b86b794fe7a8a035bd29b2c775 Command failed on smithi181 with status 22: "sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 336dd4cc-806c-11ed-97a3-001a4aab830c -e sha1=47f77a750cb9b297b7b8ab2c8bc3d22102dbeba8 -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types mgr'" [7122269] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/kernel_untar_build}} ----------------------------------------------------------------- time: 00:39:32 info: http://pulpito.ceph.com/vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi/7122269/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi/7122269/ error during scrub thrashing: rank damage found: {'backtrace'} [7122281] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/client-limits} ----------------------------------------------------------------- time: 00:13:45 info: http://pulpito.ceph.com/vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi/7122281/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi/7122281/ sentry: https://sentry.ceph.com/organizations/ceph/?query=f93fd8cb5cde43feabaa42405cfd751a Test failure: test_client_cache_size (tasks.cephfs.test_client_limits.TestClientLimits) [7122290] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/blogbench}} ----------------------------------------------------------------- time: 00:34:49 info: http://pulpito.ceph.com/vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi/7122290/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi/7122290/ error during scrub thrashing: rank damage found: {'backtrace'} [7122292] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/damage} ----------------------------------------------------------------- time: 00:24:37 info: http://pulpito.ceph.com/vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi/7122292/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi/7122292/ sentry: https://sentry.ceph.com/organizations/ceph/?query=b09cae2cfbb940fca2c9641663040d8a Test failure: test_open_ino_errors (tasks.cephfs.test_damage.TestDamage) [7122297] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:18:09 info: http://pulpito.ceph.com/vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi/7122297/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi/7122297/ sentry: https://sentry.ceph.com/organizations/ceph/?query=14a7f552e4e543dfb261ee261dab88a1 Command failed on smithi160 with status 22: "sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 14d8f512-807d-11ed-97a3-001a4aab830c -e sha1=47f77a750cb9b297b7b8ab2c8bc3d22102dbeba8 -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types mgr'" [7122300] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/suites/dbench}} ----------------------------------------------------------------- time: 00:55:32 info: http://pulpito.ceph.com/vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi/7122300/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi/7122300/ sentry: https://sentry.ceph.com/organizations/ceph/?query=d76d5c6dcca947579790ee6d80bd7f87 Command failed (workunit test suites/dbench.sh) on smithi031 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=47f77a750cb9b297b7b8ab2c8bc3d22102dbeba8 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/dbench.sh' [7122310] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/ffsb}} ----------------------------------------------------------------- time: 00:34:14 info: http://pulpito.ceph.com/vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi/7122310/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi/7122310/ error during scrub thrashing: rank damage found: {'backtrace'} [7122321] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:25:00 info: http://pulpito.ceph.com/vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi/7122321/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi/7122321/ sentry: https://sentry.ceph.com/organizations/ceph/?query=33ea3dcf018847d285fafb27d0011a82 Command failed on smithi144 with status 22: "sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 13509e54-808f-11ed-97a3-001a4aab830c -e sha1=47f77a750cb9b297b7b8ab2c8bc3d22102dbeba8 -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types mgr'" [7122330] fs/mirror-ha/{begin/{0-install 1-ceph 2-logrotate} cephfs-mirror/three-per-cluster clients/{mirror} cluster/{1-node} objectstore/bluestore-bitmap overrides/{whitelist_health} supported-random-distro$/{centos_8} workloads/cephfs-mirror-ha-workunit} ----------------------------------------------------------------- time: 00:40:51 info: http://pulpito.ceph.com/vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi/7122330/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi/7122330/ reached maximum tries (50) after waiting for 300 seconds [7122337] fs/valgrind/{begin/{0-install 1-ceph 2-logrotate} centos_latest debug mirror/{cephfs-mirror/one-per-cluster clients/mirror cluster/1-node mount/fuse overrides/whitelist_health tasks/mirror}} ----------------------------------------------------------------- time: 01:39:51 info: http://pulpito.ceph.com/vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi/7122337/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi/7122337/ sentry: https://sentry.ceph.com/organizations/ceph/?query=bb6720dc844d4c659b67e63ce5bc3975 Test failure: test_cephfs_mirror_restart_sync_on_blocklist (tasks.cephfs.test_mirroring.TestMirroring) [7122354] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/pool-perm} ----------------------------------------------------------------- time: 00:21:45 info: http://pulpito.ceph.com/vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi/7122354/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi/7122354/ sentry: https://sentry.ceph.com/organizations/ceph/?query=6437d73f7ae44f92bea1742a6583699b Test failure: test_pool_perm (tasks.cephfs.test_pool_perm.TestPoolPerm) [7122355] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/3 tasks/fsstress validater/valgrind} ----------------------------------------------------------------- time: 06:33:39 info: http://pulpito.ceph.com/vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi/7122355/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi/7122355/ sentry: https://sentry.ceph.com/organizations/ceph/?query=73b074e4cd824a969d3c11f92d85b189 Command failed (workunit test suites/fsstress.sh) on smithi040 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=47f77a750cb9b297b7b8ab2c8bc3d22102dbeba8 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsstress.sh' [7122371] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/scrub} ----------------------------------------------------------------- time: 00:26:41 info: http://pulpito.ceph.com/vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi/7122371/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi/7122371/ sentry: https://sentry.ceph.com/organizations/ceph/?query=de3fd54928864be18565324e98d9aef0 Test failure: test_scrub_backtrace (tasks.cephfs.test_scrub.TestScrub) [7122377] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/iogen}} ----------------------------------------------------------------- time: 00:50:31 info: http://pulpito.ceph.com/vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi/7122377/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi/7122377/ error during scrub thrashing: rank damage found: {'backtrace'} [7122382] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:23:38 info: http://pulpito.ceph.com/vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi/7122382/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi/7122382/ sentry: https://sentry.ceph.com/organizations/ceph/?query=afd6f8e9c98347b6a9b02355e804d731 Command failed on smithi115 with status 22: "sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c2c4edc6-8095-11ed-97a3-001a4aab830c -e sha1=47f77a750cb9b297b7b8ab2c8bc3d22102dbeba8 -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types mgr'" Dead ================================================================= [7122200] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress traceless/50pc} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi/7122200/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi/7122200/ Error reimaging machines: reached maximum tries (60) after waiting for 900 seconds [7122204] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/scrub} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi/7122204/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi/7122204/ Error reimaging machines: Failed to power on smithi168 [7122205] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi/7122205/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi/7122205/ Error reimaging machines: reached maximum tries (60) after waiting for 900 seconds [7122222] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/iozone}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi/7122222/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi/7122222/ Error reimaging machines: reached maximum tries (60) after waiting for 900 seconds [7122223] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/strays} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi/7122223/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi/7122223/ Error reimaging machines: Failed to power on smithi168 [7122225] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/test_journal_migration} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi/7122225/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi/7122225/ Error reimaging machines: reached maximum tries (60) after waiting for 900 seconds [7122231] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/multifs-auth} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi/7122231/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi/7122231/ Error reimaging machines: reached maximum tries (60) after waiting for 900 seconds [7122241] fs/upgrade/nofs/{bluestore-bitmap centos_latest conf/{client mds mon osd} no-mds-cluster overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-pacific 1-upgrade}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi/7122241/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi/7122241/ Error reimaging machines: Failed to power on smithi168 [7122245] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/workunit/quota} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi/7122245/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi/7122245/ Error reimaging machines: reached maximum tries (60) after waiting for 900 seconds [7122252] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/admin} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi/7122252/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi/7122252/ Error reimaging machines: Failed to power on smithi168 [7122253] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi/7122253/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi/7122253/ Error reimaging machines: reached maximum tries (60) after waiting for 900 seconds [7122263] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/failover} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi/7122263/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi/7122263/ Error reimaging machines: reached maximum tries (60) after waiting for 900 seconds [7122267] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mds 2-workunit/suites/ffsb}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi/7122267/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi/7122267/ Error reimaging machines: reached maximum tries (60) after waiting for 900 seconds [7122270] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} kclient-overrides/{distro/testing/k-testing ms-die-on-skipped} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_untarbuild_blogbench} ----------------------------------------------------------------- time: 00:02:06 info: http://pulpito.ceph.com/vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi/7122270/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi/7122270/ sentry: https://sentry.ceph.com/organizations/ceph/?query=af1ba2122e914f3ea4bb37a6a9158b91 {'smithi187.front.sepia.ceph.com': {'_ansible_no_log': False, 'changed': True, 'cmd': ['cpan', 'Amazon::S3'], 'delta': '0:00:01.658378', 'end': '2022-12-20 13:36:00.254158', 'invocation': {'module_args': {'_raw_params': 'cpan Amazon::S3', '_uses_shell': False, 'argv': None, 'chdir': None, 'creates': None, 'executable': None, 'removes': None, 'stdin': None, 'stdin_add_newline': True, 'strip_empty_ends': True, 'warn': True}}, 'msg': 'non-zero return code', 'rc': 25, 'start': '2022-12-20 13:35:58.595780', 'stderr': '', 'stderr_lines': [], 'stdout': "Loading internal logger. Log::Log4perl recommended for better logging\nReading '/home/ubuntu/.cpan/Metadata'\n Database was generated on Fri, 12 Feb 2016 02:17:02 GMT\nRunning install for module 'Amazon::S3'\n\nWarning: checksum file '/home/ubuntu/.cpan/sources/authors/id/T/TI/TIMA/CHECKSUMS' not conforming.\n\nThe cksum does not contain the key 'cpan_path' for 'Amazon-S3-0.45.tar.gz'.\nProceed nonetheless? [no] no\nAborted.", 'stdout_lines': ['Loading internal logger. Log::Log4perl recommended for better logging', "Reading '/home/ubuntu/.cpan/Metadata'", ' Database was generated on Fri, 12 Feb 2016 02:17:02 GMT', "Running install for module 'Amazon::S3'", '', "Warning: checksum file '/home/ubuntu/.cpan/sources/authors/id/T/TI/TIMA/CHECKSUMS' not conforming.", '', "The cksum does not contain the key 'cpan_path' for 'Amazon-S3-0.45.tar.gz'.", 'Proceed nonetheless? [no] no', 'Aborted.']}, 'smithi131.front.sepia.ceph.com': {'_ansible_no_log': False, 'changed': True, 'cmd': ['cpan', 'Amazon::S3'], 'delta': '0:00:01.791174', 'end': '2022-12-20 13:36:01.275037', 'invocation': {'module_args': {'_raw_params': 'cpan Amazon::S3', '_uses_shell': False, 'argv': None, 'chdir': None, 'creates': None, 'executable': None, 'removes': None, 'stdin': None, 'stdin_add_newline': True, 'strip_empty_ends': True, 'warn': True}}, 'msg': 'non-zero return code', 'rc': 25, 'start': '2022-12-20 13:35:59.483863', 'stderr': '', 'stderr_lines': [], 'stdout': "Loading internal logger. Log::Log4perl recommended for better logging\nReading '/home/ubuntu/.cpan/Metadata'\n Database was generated on Fri, 12 Feb 2016 02:17:02 GMT\nRunning install for module 'Amazon::S3'\n\nWarning: checksum file '/home/ubuntu/.cpan/sources/authors/id/T/TI/TIMA/CHECKSUMS' not conforming.\n\nThe cksum does not contain the key 'cpan_path' for 'Amazon-S3-0.45.tar.gz'.\nProceed nonetheless? [no] no\nAborted.", 'stdout_lines': ['Loading internal logger. Log::Log4perl recommended for better logging', "Reading '/home/ubuntu/.cpan/Metadata'", ' Database was generated on Fri, 12 Feb 2016 02:17:02 GMT', "Running install for module 'Amazon::S3'", '', "Warning: checksum file '/home/ubuntu/.cpan/sources/authors/id/T/TI/TIMA/CHECKSUMS' not conforming.", '', "The cksum does not contain the key 'cpan_path' for 'Amazon-S3-0.45.tar.gz'.", 'Proceed nonetheless? [no] no', 'Aborted.']}} [7122274] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/5 tasks/fsstress validater/valgrind} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi/7122274/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi/7122274/ Error reimaging machines: reached maximum tries (60) after waiting for 900 seconds [7122288] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/client-recovery} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi/7122288/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi/7122288/ Error reimaging machines: reached maximum tries (60) after waiting for 900 seconds [7122293] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/mdtest} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi/7122293/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi/7122293/ Error reimaging machines: reached maximum tries (60) after waiting for 900 seconds [7122303] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/exports} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi/7122303/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi/7122303/ Error reimaging machines: reached maximum tries (100) after waiting for 600 seconds [7122306] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} kclient-overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_dbench_iozone} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi/7122306/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi/7122306/ Error reimaging machines: reached maximum tries (180) after waiting for 2700 seconds [7122313] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/fragment} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi/7122313/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi/7122313/ Error reimaging machines: Failed to power on smithi190 [7122316] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/quincy}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi/7122316/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi/7122316/ Error reimaging machines: reached maximum tries (180) after waiting for 2700 seconds [7122349] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} kclient-overrides/{distro/testing/k-testing ms-die-on-skipped} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_untarbuild_blogbench} ----------------------------------------------------------------- time: 00:05:03 info: http://pulpito.ceph.com/vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi/7122349/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi/7122349/ sentry: https://sentry.ceph.com/organizations/ceph/?query=4cbb59efcc8545838c89bf9e4c604c5d {'smithi098.front.sepia.ceph.com': {'_ansible_no_log': False, 'changed': True, 'cmd': ['cpan', 'Amazon::S3'], 'delta': '0:00:01.783839', 'end': '2022-12-20 17:56:51.140418', 'invocation': {'module_args': {'_raw_params': 'cpan Amazon::S3', '_uses_shell': False, 'argv': None, 'chdir': None, 'creates': None, 'executable': None, 'removes': None, 'stdin': None, 'stdin_add_newline': True, 'strip_empty_ends': True, 'warn': True}}, 'msg': 'non-zero return code', 'rc': 25, 'start': '2022-12-20 17:56:49.356579', 'stderr': '', 'stderr_lines': [], 'stdout': "Loading internal logger. Log::Log4perl recommended for better logging\nReading '/home/ubuntu/.cpan/Metadata'\n Database was generated on Fri, 12 Feb 2016 02:17:02 GMT\nRunning install for module 'Amazon::S3'\n\nWarning: checksum file '/home/ubuntu/.cpan/sources/authors/id/T/TI/TIMA/CHECKSUMS' not conforming.\n\nThe cksum does not contain the key 'cpan_path' for 'Amazon-S3-0.45.tar.gz'.\nProceed nonetheless? [no] no\nAborted.", 'stdout_lines': ['Loading internal logger. Log::Log4perl recommended for better logging', "Reading '/home/ubuntu/.cpan/Metadata'", ' Database was generated on Fri, 12 Feb 2016 02:17:02 GMT', "Running install for module 'Amazon::S3'", '', "Warning: checksum file '/home/ubuntu/.cpan/sources/authors/id/T/TI/TIMA/CHECKSUMS' not conforming.", '', "The cksum does not contain the key 'cpan_path' for 'Amazon-S3-0.45.tar.gz'.", 'Proceed nonetheless? [no] no', 'Aborted.']}, 'smithi082.front.sepia.ceph.com': {'_ansible_no_log': False, 'changed': True, 'cmd': ['cpan', 'Amazon::S3'], 'delta': '0:00:01.685929', 'end': '2022-12-20 17:56:51.323113', 'invocation': {'module_args': {'_raw_params': 'cpan Amazon::S3', '_uses_shell': False, 'argv': None, 'chdir': None, 'creates': None, 'executable': None, 'removes': None, 'stdin': None, 'stdin_add_newline': True, 'strip_empty_ends': True, 'warn': True}}, 'msg': 'non-zero return code', 'rc': 25, 'start': '2022-12-20 17:56:49.637184', 'stderr': '', 'stderr_lines': [], 'stdout': "Loading internal logger. Log::Log4perl recommended for better logging\nReading '/home/ubuntu/.cpan/Metadata'\n Database was generated on Fri, 12 Feb 2016 02:17:02 GMT\nRunning install for module 'Amazon::S3'\n\nWarning: checksum file '/home/ubuntu/.cpan/sources/authors/id/T/TI/TIMA/CHECKSUMS' not conforming.\n\nThe cksum does not contain the key 'cpan_path' for 'Amazon-S3-0.45.tar.gz'.\nProceed nonetheless? [no] no\nAborted.", 'stdout_lines': ['Loading internal logger. Log::Log4perl recommended for better logging', "Reading '/home/ubuntu/.cpan/Metadata'", ' Database was generated on Fri, 12 Feb 2016 02:17:02 GMT', "Running install for module 'Amazon::S3'", '', "Warning: checksum file '/home/ubuntu/.cpan/sources/authors/id/T/TI/TIMA/CHECKSUMS' not conforming.", '', "The cksum does not contain the key 'cpan_path' for 'Amazon-S3-0.45.tar.gz'.", 'Proceed nonetheless? [no] no', 'Aborted.']}} [7122376] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snap-schedule} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi/7122376/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi/7122376/ Error reimaging machines: reached maximum tries (180) after waiting for 2700 seconds [7122378] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mds 2-workunit/fs/snaps}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi/7122378/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi/7122378/ Error reimaging machines: reached maximum tries (180) after waiting for 2700 seconds Pass ================================================================= [7122193] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}} time: 00:34:50 info: http://pulpito.ceph.com/vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi/7122193/ [7122194] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/recovery-fs} time: 00:12:36 info: http://pulpito.ceph.com/vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi/7122194/ [7122195] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} kclient-overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_dbench_iozone} time: 00:46:58 info: http://pulpito.ceph.com/vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi/7122195/ [7122196] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/mdtest} time: 00:15:09 info: http://pulpito.ceph.com/vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi/7122196/ [7122197] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/failover} time: 00:52:11 info: http://pulpito.ceph.com/vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi/7122197/ [7122198] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps} time: 00:45:59 info: http://pulpito.ceph.com/vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi/7122198/ [7122199] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_trivial_sync}} time: 00:14:32 info: http://pulpito.ceph.com/vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi/7122199/ [7122201] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/1 tasks/fsstress validater/valgrind} time: 01:23:53 info: http://pulpito.ceph.com/vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi/7122201/ [7122202] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/basic}} time: 02:18:06 info: http://pulpito.ceph.com/vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi/7122202/ [7122203] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/suites/fsync-tester}} time: 00:34:14 info: http://pulpito.ceph.com/vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi/7122203/ [7122206] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mds 2-workunit/fs/snaps}} time: 02:16:37 info: http://pulpito.ceph.com/vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi/7122206/ [7122207] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-ec-root overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} time: 00:15:55 info: http://pulpito.ceph.com/vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi/7122207/ [7122208] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} time: 00:16:16 info: http://pulpito.ceph.com/vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi/7122208/ [7122209] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/sessionmap} time: 00:17:35 info: http://pulpito.ceph.com/vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi/7122209/ [7122210] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/fs/test_o_trunc}} time: 00:35:10 info: http://pulpito.ceph.com/vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi/7122210/ [7122211] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/ffsb}} time: 00:23:46 info: http://pulpito.ceph.com/vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi/7122211/ [7122213] fs/fscrypt/{begin/{0-install 1-ceph 2-logrotate} bluestore-bitmap clusters/1-mds-1-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/v1 mount overrides/{distro/testing/k-testing}} overrides/{ignorelist_health ignorelist_health_more ignorelist_wrongly_marked_down pg-warn} tasks/fscrypt-ffsb} time: 00:31:59 info: http://pulpito.ceph.com/vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi/7122213/ [7122214] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snap-schedule} time: 00:50:50 info: http://pulpito.ceph.com/vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi/7122214/ [7122215] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/ffsb}} time: 00:21:43 info: http://pulpito.ceph.com/vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi/7122215/ [7122216] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snap_schedule_snapdir} time: 00:08:59 info: http://pulpito.ceph.com/vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi/7122216/ [7122217] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/quincy}} time: 00:25:50 info: http://pulpito.ceph.com/vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi/7122217/ [7122218] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/suites/iogen}} time: 00:46:59 info: http://pulpito.ceph.com/vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi/7122218/ [7122219] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/osd 2-workunit/suites/fsstress}} time: 00:28:09 info: http://pulpito.ceph.com/vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi/7122219/ [7122220] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snapshots} time: 00:25:44 info: http://pulpito.ceph.com/vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi/7122220/ [7122221] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} time: 00:35:30 info: http://pulpito.ceph.com/vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi/7122221/ [7122224] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/iozone}} time: 00:22:28 info: http://pulpito.ceph.com/vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi/7122224/ [7122226] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mds 2-workunit/fs/trivial_sync}} time: 00:09:47 info: http://pulpito.ceph.com/vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi/7122226/ [7122227] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} time: 00:15:57 info: http://pulpito.ceph.com/vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi/7122227/ [7122228] fs/cephadm/renamevolume/{0-start 1-rename distro/single-container-host overrides/ignorelist_health} time: 00:20:34 info: http://pulpito.ceph.com/vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi/7122228/ [7122229] fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{centos_8} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/libcephfs/{frag test}} time: 00:14:32 info: http://pulpito.ceph.com/vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi/7122229/ [7122230] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cephfs_misc_tests} time: 01:07:27 info: http://pulpito.ceph.com/vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi/7122230/ [7122232] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} time: 00:16:09 info: http://pulpito.ceph.com/vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi/7122232/ [7122233] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps} time: 01:29:54 info: http://pulpito.ceph.com/vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi/7122233/ [7122234] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/clone}} time: 01:16:32 info: http://pulpito.ceph.com/vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi/7122234/ [7122235] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}} time: 00:32:08 info: http://pulpito.ceph.com/vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi/7122235/ [7122236] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} kclient-overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_dbench_iozone} time: 00:40:07 info: http://pulpito.ceph.com/vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi/7122236/ [7122237] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_blogbench traceless/50pc} time: 00:22:15 info: http://pulpito.ceph.com/vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi/7122237/ [7122238] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/truncate_delay} time: 00:12:28 info: http://pulpito.ceph.com/vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi/7122238/ [7122240] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/3 tasks/dbench validater/lockdep} time: 00:50:59 info: http://pulpito.ceph.com/vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi/7122240/ [7122242] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/workunit/dir-max-entries} time: 00:17:55 info: http://pulpito.ceph.com/vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi/7122242/ [7122244] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/iozone}} time: 00:18:46 info: http://pulpito.ceph.com/vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi/7122244/ [7122246] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/osd 2-workunit/suites/pjd}} time: 00:18:02 info: http://pulpito.ceph.com/vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi/7122246/ [7122247] fs/upgrade/upgraded_client/from_nautilus/{bluestore-bitmap centos_latest clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-nautilus 1-client-upgrade 2-client-sanity}} time: 00:13:13 info: http://pulpito.ceph.com/vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi/7122247/ [7122248] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/acls} time: 00:16:50 info: http://pulpito.ceph.com/vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi/7122248/ [7122249] fs/fscrypt/{begin/{0-install 1-ceph 2-logrotate} bluestore-bitmap clusters/1-mds-1-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/v1 mount overrides/{distro/testing/k-testing}} overrides/{ignorelist_health ignorelist_health_more ignorelist_wrongly_marked_down pg-warn} tasks/fscrypt-iozone} time: 00:16:59 info: http://pulpito.ceph.com/vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi/7122249/ [7122250] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/direct_io}} time: 00:28:11 info: http://pulpito.ceph.com/vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi/7122250/ [7122251] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_suites_fsstress}} time: 00:22:44 info: http://pulpito.ceph.com/vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi/7122251/ [7122254] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress} time: 00:18:46 info: http://pulpito.ceph.com/vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi/7122254/ [7122255] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_misc} time: 00:15:49 info: http://pulpito.ceph.com/vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi/7122255/ [7122256] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/osd 2-workunit/fs/snaps}} time: 03:28:39 info: http://pulpito.ceph.com/vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi/7122256/ [7122257] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/alternate-pool} time: 00:11:32 info: http://pulpito.ceph.com/vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi/7122257/ [7122258] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/fs/misc}} time: 01:22:49 info: http://pulpito.ceph.com/vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi/7122258/ [7122259] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} time: 00:22:15 info: http://pulpito.ceph.com/vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi/7122259/ [7122260] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_suites_pjd}} time: 00:17:42 info: http://pulpito.ceph.com/vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi/7122260/ [7122261] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/asok_dump_tree} time: 00:27:06 info: http://pulpito.ceph.com/vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi/7122261/ [7122262] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/ior-shared-file} time: 00:14:37 info: http://pulpito.ceph.com/vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi/7122262/ [7122264] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps} time: 00:38:36 info: http://pulpito.ceph.com/vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi/7122264/ [7122265] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/misc}} time: 00:18:38 info: http://pulpito.ceph.com/vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi/7122265/ [7122266] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/auto-repair} time: 00:14:16 info: http://pulpito.ceph.com/vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi/7122266/ [7122271] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_dbench traceless/50pc} time: 00:51:14 info: http://pulpito.ceph.com/vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi/7122271/ [7122272] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/backtrace} time: 00:09:34 info: http://pulpito.ceph.com/vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi/7122272/ [7122273] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_trivial_sync}} time: 00:14:41 info: http://pulpito.ceph.com/vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi/7122273/ [7122275] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/quincy}} time: 00:25:04 info: http://pulpito.ceph.com/vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi/7122275/ [7122276] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/cap-flush} time: 00:13:17 info: http://pulpito.ceph.com/vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi/7122276/ [7122277] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mon 2-workunit/suites/fsstress}} time: 00:22:20 info: http://pulpito.ceph.com/vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi/7122277/ [7122278] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} time: 00:16:10 info: http://pulpito.ceph.com/vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi/7122278/ [7122279] fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{ubuntu_latest} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/libcephfs_python} time: 00:12:32 info: http://pulpito.ceph.com/vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi/7122279/ [7122280] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} time: 00:15:18 info: http://pulpito.ceph.com/vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi/7122280/ [7122282] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/postgres}} time: 01:18:29 info: http://pulpito.ceph.com/vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi/7122282/ [7122283] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} time: 00:36:18 info: http://pulpito.ceph.com/vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi/7122283/ [7122284] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/ffsb}} time: 00:50:22 info: http://pulpito.ceph.com/vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi/7122284/ [7122285] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/client-readahead} time: 00:09:29 info: http://pulpito.ceph.com/vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi/7122285/ [7122286] fs/fscrypt/{begin/{0-install 1-ceph 2-logrotate} bluestore-bitmap clusters/1-mds-1-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/v1 mount overrides/{distro/testing/k-testing}} overrides/{ignorelist_health ignorelist_health_more ignorelist_wrongly_marked_down pg-warn} tasks/fscrypt-pjd} time: 00:21:51 info: http://pulpito.ceph.com/vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi/7122286/ [7122287] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/osd 2-workunit/fs/trivial_sync}} time: 00:11:10 info: http://pulpito.ceph.com/vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi/7122287/ [7122289] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}} time: 00:31:04 info: http://pulpito.ceph.com/vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi/7122289/ [7122291] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mds 2-workunit/suites/iozone}} time: 00:16:56 info: http://pulpito.ceph.com/vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi/7122291/ [7122294] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/multifs-auth} time: 00:28:27 info: http://pulpito.ceph.com/vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi/7122294/ [7122295] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps} time: 01:07:58 info: http://pulpito.ceph.com/vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi/7122295/ [7122296] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/snapshot}} time: 00:37:59 info: http://pulpito.ceph.com/vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi/7122296/ [7122298] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/data-scan} time: 00:24:49 info: http://pulpito.ceph.com/vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi/7122298/ [7122299] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/iozone}} time: 00:13:52 info: http://pulpito.ceph.com/vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi/7122299/ [7122301] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-ec-root overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress} time: 00:19:30 info: http://pulpito.ceph.com/vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi/7122301/ [7122302] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_misc} time: 00:15:16 info: http://pulpito.ceph.com/vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi/7122302/ [7122304] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} time: 00:21:54 info: http://pulpito.ceph.com/vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi/7122304/ [7122305] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mon 2-workunit/suites/pjd}} time: 00:17:48 info: http://pulpito.ceph.com/vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi/7122305/ [7122307] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_ffsb traceless/50pc} time: 00:43:53 info: http://pulpito.ceph.com/vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi/7122307/ [7122308] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/forward-scrub} time: 00:25:23 info: http://pulpito.ceph.com/vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi/7122308/ [7122309] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_snaptests}} time: 01:44:23 info: http://pulpito.ceph.com/vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi/7122309/ [7122311] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} time: 00:41:14 info: http://pulpito.ceph.com/vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi/7122311/ [7122312] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu/{latest overrides}} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/1 tasks/dbench validater/lockdep} time: 00:52:16 info: http://pulpito.ceph.com/vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi/7122312/ [7122314] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mon 2-workunit/fs/snaps}} time: 01:53:49 info: http://pulpito.ceph.com/vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi/7122314/ [7122315] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/journal-repair} time: 00:26:38 info: http://pulpito.ceph.com/vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi/7122315/ [7122317] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_suites_fsstress}} time: 00:27:24 info: http://pulpito.ceph.com/vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi/7122317/ [7122318] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/fs/norstats}} time: 00:40:56 info: http://pulpito.ceph.com/vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi/7122318/ [7122319] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/mds-flush} time: 00:20:15 info: http://pulpito.ceph.com/vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi/7122319/ [7122320] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/ffsb}} time: 00:29:54 info: http://pulpito.ceph.com/vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi/7122320/ [7122322] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/mds-full} time: 00:24:16 info: http://pulpito.ceph.com/vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi/7122322/ [7122323] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress} time: 00:23:30 info: http://pulpito.ceph.com/vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi/7122323/ [7122324] fs/bugs/client_trim_caps/{begin/{0-install 1-ceph 2-logrotate} centos_latest clusters/small-cluster conf/{client mds mon osd} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/trim-i24137} time: 00:18:26 info: http://pulpito.ceph.com/vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi/7122324/ [7122325] fs/cephadm/multivolume/{0-start 1-mount 2-workload/dbench distro/single-container-host} time: 00:55:11 info: http://pulpito.ceph.com/vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi/7122325/ [7122326] fs/fscrypt/{begin/{0-install 1-ceph 2-logrotate} bluestore-bitmap clusters/1-mds-1-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/v1 mount overrides/{distro/testing/k-testing}} overrides/{ignorelist_health ignorelist_health_more ignorelist_wrongly_marked_down pg-warn} tasks/fscrypt-common} time: 00:37:57 info: http://pulpito.ceph.com/vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi/7122326/ [7122327] fs/full/{begin/{0-install 1-ceph 2-logrotate} clusters/1-node-1-mds-1-osd conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-bitmap overrides overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/mgr-osd-full} time: 00:28:02 info: http://pulpito.ceph.com/vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi/7122327/ [7122328] fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{centos_8} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/client} time: 00:19:08 info: http://pulpito.ceph.com/vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi/7122328/ [7122329] fs/mirror/{begin/{0-install 1-ceph 2-logrotate} cephfs-mirror/one-per-cluster clients/{mirror} cluster/{1-node} mount/fuse objectstore/bluestore-bitmap overrides/{whitelist_health} supported-random-distros$/{rhel_8} tasks/mirror} time: 02:21:37 info: http://pulpito.ceph.com/vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi/7122329/ [7122331] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cephfs_misc_tests} time: 01:06:54 info: http://pulpito.ceph.com/vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi/7122331/ [7122332] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/multifs-auth} time: 00:44:09 info: http://pulpito.ceph.com/vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi/7122332/ [7122333] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_misc} time: 00:22:00 info: http://pulpito.ceph.com/vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi/7122333/ [7122334] fs/shell/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/cephfs-shell} time: 00:54:54 info: http://pulpito.ceph.com/vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi/7122334/ [7122335] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps} time: 01:15:21 info: http://pulpito.ceph.com/vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi/7122335/ [7122336] fs/top/{begin/{0-install 1-ceph 2-logrotate} cluster/{1-node} mount/fuse objectstore/bluestore-bitmap overrides/ignorelist_health supported-random-distros$/{centos_8} tasks/fstop} time: 00:17:42 info: http://pulpito.ceph.com/vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi/7122336/ [7122338] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/basic}} time: 02:22:53 info: http://pulpito.ceph.com/vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi/7122338/ [7122339] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/mds_creation_retry} time: 00:19:30 info: http://pulpito.ceph.com/vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi/7122339/ [7122340] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_suites_pjd}} time: 00:18:30 info: http://pulpito.ceph.com/vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi/7122340/ [7122341] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/fsstress}} time: 00:33:56 info: http://pulpito.ceph.com/vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi/7122341/ [7122342] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}} time: 00:40:30 info: http://pulpito.ceph.com/vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi/7122342/ [7122343] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/metrics} time: 00:42:20 info: http://pulpito.ceph.com/vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi/7122343/ [7122344] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mds 2-workunit/suites/fsstress}} time: 00:35:01 info: http://pulpito.ceph.com/vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi/7122344/ [7122345] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/multimds_misc} time: 00:19:59 info: http://pulpito.ceph.com/vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi/7122345/ [7122346] fs/upgrade/nofs/{bluestore-bitmap centos_latest conf/{client mds mon osd} no-mds-cluster overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-pacific 1-upgrade}} time: 00:17:58 info: http://pulpito.ceph.com/vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi/7122346/ [7122347] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mon 2-workunit/fs/trivial_sync}} time: 00:14:29 info: http://pulpito.ceph.com/vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi/7122347/ [7122348] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/suites/fsx}} time: 00:42:29 info: http://pulpito.ceph.com/vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi/7122348/ [7122350] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress traceless/50pc} time: 00:24:53 info: http://pulpito.ceph.com/vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi/7122350/ [7122351] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/openfiletable} time: 00:20:41 info: http://pulpito.ceph.com/vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi/7122351/ [7122352] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_trivial_sync}} time: 00:17:38 info: http://pulpito.ceph.com/vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi/7122352/ [7122353] fs/upgrade/upgraded_client/from_nautilus/{bluestore-bitmap centos_latest clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-nautilus 1-client-upgrade 2-client-sanity}} time: 00:18:31 info: http://pulpito.ceph.com/vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi/7122353/ [7122356] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} time: 00:22:10 info: http://pulpito.ceph.com/vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi/7122356/ [7122357] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} time: 00:20:52 info: http://pulpito.ceph.com/vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi/7122357/ [7122358] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/iozone}} time: 00:24:42 info: http://pulpito.ceph.com/vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi/7122358/ [7122359] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/fsync-tester}} time: 00:26:00 info: http://pulpito.ceph.com/vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi/7122359/ [7122360] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/quota} time: 00:13:57 info: http://pulpito.ceph.com/vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi/7122360/ [7122361] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} time: 00:42:41 info: http://pulpito.ceph.com/vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi/7122361/ [7122362] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/ffsb}} time: 00:53:27 info: http://pulpito.ceph.com/vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi/7122362/ [7122363] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/recovery-fs} time: 00:19:36 info: http://pulpito.ceph.com/vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi/7122363/ [7122364] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/ior-shared-file} time: 00:17:31 info: http://pulpito.ceph.com/vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi/7122364/ [7122365] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/failover} time: 00:53:19 info: http://pulpito.ceph.com/vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi/7122365/ [7122366] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps} time: 00:49:44 info: http://pulpito.ceph.com/vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi/7122366/ [7122367] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/clone}} time: 00:58:13 info: http://pulpito.ceph.com/vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi/7122367/ [7122368] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} time: 00:28:17 info: http://pulpito.ceph.com/vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi/7122368/ [7122369] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/fs/test_o_trunc}} time: 00:41:21 info: http://pulpito.ceph.com/vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi/7122369/ [7122370] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mds 2-workunit/suites/pjd}} time: 00:24:41 info: http://pulpito.ceph.com/vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi/7122370/ [7122372] fs/fscrypt/{begin/{0-install 1-ceph 2-logrotate} bluestore-bitmap clusters/1-mds-1-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/v1 mount overrides/{distro/testing/k-testing}} overrides/{ignorelist_health ignorelist_health_more ignorelist_wrongly_marked_down pg-warn} tasks/fscrypt-dbench} time: 01:20:06 info: http://pulpito.ceph.com/vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi/7122372/ [7122373] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/sessionmap} time: 00:16:37 info: http://pulpito.ceph.com/vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi/7122373/ [7122374] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/iozone}} time: 00:17:15 info: http://pulpito.ceph.com/vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi/7122374/ [7122375] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/quincy}} time: 00:34:51 info: http://pulpito.ceph.com/vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi/7122375/ [7122379] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress} time: 00:20:59 info: http://pulpito.ceph.com/vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi/7122379/ [7122380] fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{rhel_8} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/ino_release_cb} time: 00:18:59 info: http://pulpito.ceph.com/vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi/7122380/ [7122381] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_misc} time: 00:21:28 info: http://pulpito.ceph.com/vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi/7122381/ [7122383] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snap_schedule_snapdir} time: 00:16:07 info: http://pulpito.ceph.com/vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi/7122383/ [7122384] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} kclient-overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_untarbuild_blogbench} time: 00:29:53 info: http://pulpito.ceph.com/vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi/7122384/ [7122385] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_blogbench traceless/50pc} time: 00:30:04 info: http://pulpito.ceph.com/vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi/7122385/ [7122386] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_snaptests}} time: 01:39:55 info: http://pulpito.ceph.com/vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi/7122386/ [7122387] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snapshots} time: 00:23:03 info: http://pulpito.ceph.com/vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi/7122387/ [7122388] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/suites/iozone}} time: 00:39:07 info: http://pulpito.ceph.com/vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi/7122388/ [7122389] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}} time: 00:38:09 info: http://pulpito.ceph.com/vshankar-2022-12-20_12:15:27-fs-wip-vshankar-testing-20221215.112736-testing-default-smithi/7122389/