2023-09-05T10:53:23.393 INFO:root:teuthology version: 0.0.1.dev192+gf9a8556 2023-09-05T10:53:23.394 INFO:teuthology.results:subset: '465/1024' 2023-09-05T10:53:23.394 INFO:teuthology.results:seed: '554' 2023-09-05T15:00:34.307 INFO:root:teuthology version: 0.0.1.dev192+gf9a8556 2023-09-05T15:00:34.338 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2023-09-06T03:01:33.667 WARNING:teuthology.results:test(s) did not finish before timeout of 43200 seconds 2023-09-06T03:01:51.152 INFO:scrape:Found 197 jobs 2023-09-06T03:01:58.637 WARNING:scrape:Missing teuthology log /home/teuthworker/archive/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388353/teuthology.log 2023-09-06T03:02:02.546 WARNING:scrape:Misunderstood line: 2023-09-06T03:02:35.959 INFO:scrape:Found 38 distinct failure reasons 2023-09-06T03:02:35.960 INFO:scrape:Failure: "2023-09-05T12:21:30.856960+0000 mon.a (mon.0) 137 : cluster [WRN] Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log 2023-09-06T03:02:35.960 INFO:scrape:21 jobs: ['7388280', '7388173', '7388168', '7388273', '7388218', '7388301', '7388187', '7388189', '7388317', '7388337', '7388344', '7388228', '7388242', '7388161', '7388295', '7388287', '7388278', '7388212', '7388341', '7388345', '7388179'] 2023-09-06T03:02:35.961 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}'] 2023-09-06T03:02:35.961 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', '2-workunit/ffsb}}', 'centos_latest', 'clients/mirror', 'cluster/1-node', 'clusters/1-mds-1-client-coloc', 'clusters/1a3s-mds-1c-client', 'clusters/1a3s-mds-2c-client', 'clusters/1a3s-mds-4c-client', 'clusters/fixed-2-ucephfs', 'conf/{client', 'debug', 'distro/{centos_latest}', 'distro/{ubuntu_latest}', 'fs/32bits/{begin/{0-install', 'fs/functional/{begin/{0-install', 'fs/libcephfs/{begin/{0-install', 'fs/permission/{begin/{0-install', 'fs/snaps/{begin/{0-install', 'fs/thrash/multifs/{begin/{0-install', 'fs/traceless/{begin/{0-install', 'fs/valgrind/{begin/{0-install', 'fs/volumes/{begin/{0-install', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'ignorelist_wrongly_marked_down}', 'mds', 'mirror/{cephfs-mirror/one-per-cluster', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{mount-syntax/{v1}', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'msgr-failures/none', 'multifs', 'no_client_pidfile}', 'objectstore-ec/bluestore-comp', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/whitelist_health', 'overrides/{distro/testing/k-testing', 'overrides/{faked-ino', 'overrides/{frag', 'overrides/{ignorelist_health', 'session_timeout', 'tasks/auto-repair}', 'tasks/cfuse_workunit_misc}', 'tasks/cfuse_workunit_suites_dbench', 'tasks/cfuse_workunit_suites_ffsb', 'tasks/cfuse_workunit_suites_fsstress}', 'tasks/cfuse_workunit_suites_pjd}', 'tasks/forward-scrub}', 'tasks/fragment}', 'tasks/libcephfs/{frag', 'tasks/mds-flush}', 'tasks/metrics}', 'tasks/mirror}}', 'tasks/snap-schedule}', 'tasks/snap_schedule_snapdir}', 'tasks/test_journal_migration}', 'tasks/volumes/{overrides', 'tasks/workunit/snaps}', 'tasks/{1-thrash/mds', 'test/clone}}', 'test}}', 'thrashosds-health}', 'traceless/50pc}'] 2023-09-06T03:02:35.962 INFO:scrape: 2023-09-06T03:02:35.962 INFO:scrape:Failure: [',', 'Bugzilla. In case of problems, contact the owner of this repository.', "Curl error (7): Couldn't connect to server for https://copr.fedorainfracloud.org/coprs/ceph/python3-asyncssh/repo/epel-8/dnf.repo?arch=x86_64 [Failed to connect to copr.fedorainfracloud.org port 443: Network is unreachable]", 'Enabling a Copr repository. Please note that this repository is not part', 'Please do not file bug reports about these packages in Fedora', 'The Fedora Project does not exercise any power over the contents of', 'and packages are not held to any quality or security level.', 'of the main distribution, and quality may vary.', 'this repository beyond the rules outlined in the Copr FAQ at'] 2023-09-06T03:02:35.963 INFO:scrape:2 jobs: ['7388313', '7388309'] 2023-09-06T03:02:35.963 INFO:scrape:suites intersection: ['conf/{client', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}'] 2023-09-06T03:02:35.964 INFO:scrape:suites union: ['1-cephadm', '1-check-counter', '1-client', '2-logrotate}', '2-scrub/yes', '2-upgrade', '3-compat_client/quincy}}', '3-snaps/no', '4-flush/no', '5-workunit/suites/blogbench}}', 'begin/{0-install', 'centos_8', 'clusters/1-mds-2-client-micro', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs/upgrade/featureful_client/old_client/{bluestore-bitmap', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/crc', 'multimds/yes', 'n/5', 'objectstore-ec/bluestore-comp', 'omap_limit/10', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'pg-warn}', 'ranks/multi/{export-check', 'replication/default}', 'session_timeout}', 'standby-replay', 'tasks/{0-octopus', 'tasks/{0-subvolume/{with-no-extra-options}', 'wsync/no}'] 2023-09-06T03:02:35.964 INFO:scrape: 2023-09-06T03:02:35.964 INFO:scrape:Failure: Host key for server 'smithi148.front.sepia.ceph.com' does not match: got 'AAAAC3NzaC1lZDI1NTE5AAAAIJc7byGD1pFofuIKgJk7qTBZPWs0c19aJbLO4zQeUYIn', expected 'AAAAB3NzaC1yc2EAAAADAQABAAABgQDH9teGDjQfIJg6eYYqkcRP/VRrJdqv7Ju2yWBhO1BVnrx5f0zaSy0dqqVZBrW0Gqf7GtKZoZFjdD4i3k5BlJV9DokGEn7j06ZAvDaMTQn6S9ml7jOiLhyDBeugzpeAgzFkoo9t/auF06c9ScRMzD9cbocH7F00fi0xcNPwCeJyB0l7Szmh4OKcC6UpNd/w4zk4t56FIO3lbHE4cusW4QBceHNWOrf+WBroXBHkUXixaUkToEfhxwVi5JD5eh5sOzQhqNaTDP6iwHrFxD0/N6zA8b6L26VONQDwymX1aeSlIuF8vAlB8NpD4830ri1SRxNsILYy0hmiA+fAMjpCXnP3L72qk2RS5og+BL94k1MZbm7KUfuLthehSFMIITVmt6qx2ZrxaaUgBK8V8s7yFbExMVqG851LlM4HJ1QxFkr6ArRJkBdRSeunfeDdMw/eUwklh065poCIGWoSR7CJ1sO8hOTuGC0p+WdJ72SNxVuVSN25h8N0maBfTImZxDzYsCk=' 2023-09-06T03:02:35.965 INFO:scrape:1 jobs: ['7388275'] 2023-09-06T03:02:35.965 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/volumes/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'tasks/volumes/{overrides', 'test/basic}}'] 2023-09-06T03:02:35.966 INFO:scrape: 2023-09-06T03:02:35.966 INFO:scrape:Failure: Host key for server 'smithi033.front.sepia.ceph.com' does not match: got 'AAAAC3NzaC1lZDI1NTE5AAAAII2b8+EHn0tR7UVQyFU4c9cfLZglh/CJaxT5iamCaOjc', expected 'AAAAB3NzaC1yc2EAAAADAQABAAABgQDgt/+m+Zx3ddB7u5WO5g/qF/s8evT93NNbdC+4MOjU2BoV/iWn6Gv9I7WhfVM+i557fgOj5eUMc7C2Bwp691ZtZARduEyt2U5FkF2jwLmI7bpM9DBDuedosDjAqtS6fk0PqVmlMtyJhZ5E6OBN472dkceH3+TGMHk3OVMA/lnwPpQonhb3oILDQQG8aANxFI7+JSZ0KJxhuNn9ZUdUK1sjp1FeH+sV4rGt2zzQpiub0atq/wBJ8eSy3BdoBYEuN3Pz/bBz4wyZec4ZkI1P2yMjp7dxPAFmwlOH2Q9e4qjtw90OOZdF8j1WwQx7umDJYoO3uAwDDJaWzW5VJa5aaYmgbP5K4ZUxAOkpKMcPNRtCEr5Ay+586lD8LX1M+9ynPgM3X6NBK17HgCIIp3IeRd3tD9imopTHPSIxPkSsQ35rp0HMUEiTM4zQfeV7fknUiRJ+puqeQjeig8s1V13peoSdiflC5ldnM5ybVpaJURqSBTxXXmA+Q2Y+crb7eKbyfzc=' 2023-09-06T03:02:35.966 INFO:scrape:1 jobs: ['7388177'] 2023-09-06T03:02:35.967 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{ignorelist_health', 'tasks/snap_schedule_snapdir}'] 2023-09-06T03:02:35.967 INFO:scrape: 2023-09-06T03:02:35.968 INFO:scrape:Failure: No module named 'tasks.kclient' 2023-09-06T03:02:35.968 INFO:scrape:6 jobs: ['7388251', '7388232', '7388255', '7388248', '7388233', '7388260'] 2023-09-06T03:02:35.968 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'conf/{client', 'mds', 'mon', 'osd}'] 2023-09-06T03:02:35.969 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', '2-workunit/cfuse_workunit_trivial_sync}}', '2-workunit/suites/pjd}}', 'centos_latest', 'clusters/1a3s-mds-1c-client', 'clusters/1a3s-mds-2c-client', 'clusters/1a5s-mds-1c-client', 'clusters/small-cluster', 'conf/{client', 'distro/{centos_8}', 'distro/{centos_latest}', 'distro/{ubuntu_20.04}', 'distro/{ubuntu_latest}', 'fs/bugs/client_trim_caps/{begin/{0-install', 'fs/mixed-clients/{begin/{0-install', 'fs/multifs/{begin/{0-install', 'fs/snaps/{begin/{0-install', 'fs/thrash/multifs/{begin/{0-install', 'fs/thrash/workloads/{begin/{0-install', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'ignorelist_wrongly_marked_down}', 'kclient-overrides/{distro/stock/{k-stock', 'mds', 'mon', 'mon-debug}', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}', 'ms-die-on-skipped}}', 'msgr-failures/none', 'msgr-failures/osd-mds-delay', 'multifs', 'no_client_pidfile}', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'objectstore/bluestore-bitmap', 'osd-asserts}', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{distro/testing/k-testing', 'overrides/{frag', 'overrides/{ignorelist_health', 'prefetch_dirfrags/no', 'prefetch_entire_dirfrags/yes', 'races', 'ranks/1', 'rhel_8}', 'session_timeout', 'tasks/failover}', 'tasks/kernel_cfuse_workunits_dbench_iozone}', 'tasks/trim-i24137}', 'tasks/workunit/snaps}', 'tasks/{1-thrash/mds', 'thrashosds-health}'] 2023-09-06T03:02:35.969 INFO:scrape: 2023-09-06T03:02:35.970 INFO:scrape:Failure: Host key for server 'smithi070.front.sepia.ceph.com' does not match: got 'AAAAC3NzaC1lZDI1NTE5AAAAIBtMSN1hD6JjOPqkidrP6QuhdjhpZuL0mUqQPWYcMSgE', expected 'AAAAB3NzaC1yc2EAAAADAQABAAABgQCQZj1In+CZD14/mPNuuKYl2Mom6qcGk+l7zco5s0D80pIsjm5RVslNTIBCmINeR8cSo5LETiSjr8UL2JkxhyU6KsW5oYw2Ltf3JUm9/Pu6Ie/hbDYtyLXMipwgckzPbE9dbszQh/qUTu+ZmG/HSuRW8pj5DNuti+D2QanoU6nxund9m3i7XnKxC3wx2ch1kVjukuvRbYnGU82yVvgL1XqJxgDr6Nw501XIxQ3lLAU5buAW9BqWshw5y2Z7pgHEKCF+rKSayY74Qweg7smolveH/a8MVOwH6BAwBfBwvo5Rbxl1+892Q85Vf8KVIm+j7XmAKQU9O1ABg1HcWnkK/DBHoQXjYDRhprAtiEMdMi+7vG4dGi4xPgtkT8/vPImX6svNmRCcmiJ0pwAAxbhti8JiaGcA03W9WFExf+qyHNFoy8WzrGIiRXgLhNgn2DGCeXOXYTdMWjGC7E63JJnIa2N6BZPk2hhNInFGdVibOdjmmw8UncFreo67MHWuSUvQMJM=' 2023-09-06T03:02:35.970 INFO:scrape:1 jobs: ['7388303'] 2023-09-06T03:02:35.971 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1-mds-3-client', 'conf/{client', 'distros/ubuntu_latest', 'fs/multiclient/{begin/{0-install', 'ignorelist_wrongly_marked_down}', 'mds', 'mon', 'mount/fuse', 'objectstore-ec/bluestore-comp-ec-root', 'osd}', 'overrides/{ignorelist_health', 'tasks/ior-shared-file}'] 2023-09-06T03:02:35.971 INFO:scrape: 2023-09-06T03:02:35.971 INFO:scrape:Failure: Command failed (workunit test fs/snaps/snaptest-git-ceph.sh) on smithi110 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=9d3f068cec6c5fb39569ca38ddb0f19fce7352a2 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-git-ceph.sh' 2023-09-06T03:02:35.972 INFO:scrape:2 jobs: ['7388209', '7388277'] 2023-09-06T03:02:35.972 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', '2-workunit/fs/snaps}}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'fs/thrash/workloads/{begin/{0-install', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'ms-die-on-skipped}}', 'msgr-failures/osd-mds-delay', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{frag', 'races', 'ranks/3', 'session_timeout', 'thrashosds-health}'] 2023-09-06T03:02:35.973 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', '2-workunit/fs/snaps}}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{centos_8}', 'distro/{rhel_8}', 'fs/thrash/workloads/{begin/{0-install', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'msgr-failures/osd-mds-delay', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{frag', 'prefetch_dirfrags/no', 'prefetch_dirfrags/yes', 'prefetch_entire_dirfrags/no', 'prefetch_entire_dirfrags/yes', 'races', 'ranks/3', 'session_timeout', 'tasks/{1-thrash/mds', 'tasks/{1-thrash/mon', 'thrashosds-health}'] 2023-09-06T03:02:35.973 INFO:scrape: 2023-09-06T03:02:35.973 INFO:scrape:Failure: Host key for server 'smithi032.front.sepia.ceph.com' does not match: got 'AAAAC3NzaC1lZDI1NTE5AAAAIDOtm5am3jRJYE/T1fxj9/g+WXT6PoaQiADP5H0gdFe0', expected 'AAAAB3NzaC1yc2EAAAADAQABAAABgQC2tMjUxLR1CzgQy34DbMB6w2ntYb+VKflQ0Lst2baI41oz7RySTQ8BZXmm4qbRQSjZODLpbvC5nCtb5zTftAuLNKuGV//exhPmq9UtunHTKN+7eLfm2fUQPE8kCD7w8ZqxvTZjl6W8Z/Y7IDHMDoF78dEEz98ePdwZu6wWTADfBzSE+mYQ5oJuWLZa1gToYaBroMaWS72mzWNnDRYNqU6h3J4JfOkq0d36XKOltWiDYsrT9Jd/Rg4g1MVJ2CPY/8zTLzA3NkSDqyzYG7Vp6RcSfH5Gzd8HIOyVkajZ7rdWsDFlaYuLiykpiHRN5p0cFJArq2Rv0WdEUuKPPPlRk9832ZZGYXhAJZhHFM+3EBtlL9Y73Y1MoYB4DPStUZJ1aULjwaTGdOS/LyFfJOBorMxFM4DYGFF2S0QMZT+7W5i1/04cOAJoxuX2KPhzPJgfAeb1LWOiLovXcY04+H1I5it8dHeinP//ABKMBWqE8iJ/g/Osa/vC1hPqD2r7JJXO5Tk=' 2023-09-06T03:02:35.974 INFO:scrape:1 jobs: ['7388200'] 2023-09-06T03:02:35.974 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1-mds-3-client', 'conf/{client', 'distros/ubuntu_latest', 'fs/multiclient/{begin/{0-install', 'ignorelist_wrongly_marked_down}', 'mds', 'mon', 'mount/fuse', 'objectstore-ec/bluestore-comp', 'osd}', 'overrides/{ignorelist_health', 'tasks/ior-shared-file}'] 2023-09-06T03:02:35.975 INFO:scrape: 2023-09-06T03:02:35.975 INFO:scrape:Failure: Command failed on smithi173 with status 1: 'sudo yum -y install ceph-radosgw ceph-test ceph ceph-fuse cephfs-java libcephfs_jni1 libcephfs1 librados2 librbd1 python-ceph rbd-fuse python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib' 2023-09-06T03:02:35.975 INFO:scrape:3 jobs: ['7388265', '7388258', '7388250'] 2023-09-06T03:02:35.976 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}'] 2023-09-06T03:02:35.976 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', '2-three-per-cluster}', 'cephfs-mirror/{1-volume-create-rm', 'clients/{mirror}', 'cluster/{1-node}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_20.04}', 'distro/{ubuntu_latest}', 'fs/functional/{begin/{0-install', 'fs/mirror-ha/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{ignorelist_health', 'overrides/{whitelist_health}', 'rhel_8}', 'supported-random-distro$/{centos_latest}', 'tasks/client-recovery}', 'tasks/exports}', 'workloads/cephfs-mirror-ha-workunit}'] 2023-09-06T03:02:35.976 INFO:scrape: 2023-09-06T03:02:35.977 INFO:scrape:Failure: Command failed (workunit test fs/quota/quota.sh) on smithi052 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=9d3f068cec6c5fb39569ca38ddb0f19fce7352a2 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/quota/quota.sh' 2023-09-06T03:02:35.977 INFO:scrape:1 jobs: ['7388207'] 2023-09-06T03:02:35.978 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{ignorelist_health', 'rhel_8}', 'tasks/workunit/quota}'] 2023-09-06T03:02:35.978 INFO:scrape: 2023-09-06T03:02:35.978 INFO:scrape:Dead: b'2023-09-06T02:09:50.797 DEBUG:teuthology.exit:Finished running handlers' 2023-09-06T03:02:35.979 INFO:scrape:4 jobs: ['7388304', '7388323', '7388203', '7388201'] 2023-09-06T03:02:35.979 INFO:scrape:suites intersection: ['2-logrotate}', 'conf/{client', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}'] 2023-09-06T03:02:35.980 INFO:scrape:suites union: ['1-ceph', '1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/no', '3-snaps/yes', '4-flush/yes', '5-workunit/suites/dbench}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'clusters/1a3s-mds-2c-client', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_8}', 'distro/{rhel_8}', 'distro/{ubuntu_20.04}', 'frag', 'fs/functional/{begin/{0-install', 'fs/multifs/{begin/{0-install', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mon-debug}', 'mount', 'mount/fuse', 'mount/kclient/{base/{mount-syntax/{v2}', 'mount/kclient/{mount-syntax/{v1}', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/legacy', 'n/5', 'no_client_pidfile}', 'objectstore-ec/bluestore-comp', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'objectstore/bluestore-ec-root', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/stock/{k-stock', 'overrides/{ignorelist_health', 'ranks/multi/{export-check', 'replication/always}', 'rhel_8}', 'session_timeout}', 'standby-replay', 'tasks/multifs-auth}', 'tasks/xfstests-dev}', 'tasks/{0-subvolume/{no-subvolume}', 'wsync/yes}'] 2023-09-06T03:02:35.980 INFO:scrape: 2023-09-06T03:02:35.980 INFO:scrape:Failure: Host key for server 'smithi057.front.sepia.ceph.com' does not match: got 'AAAAC3NzaC1lZDI1NTE5AAAAICYlaLilwrGYOFacZgvfJMNiJG8T/bX383rWnoKGYVN7', expected 'AAAAB3NzaC1yc2EAAAADAQABAAABgQD0nd07+IelHEWXd3xOw/He2SPqgKDycxSwGvXGRxZg2Beyjj9xv0feO+qCBdI2RbI7bqjbLUCt2Vb4bDUlZYUoVUB42HlnnHyaqUP19/HpZJiyAGAGs6fpCcJingSqe7bxXfU4gVdJKUuwZfQGTwbY80+CV46jf5EkIl5a9a4wpkuazDTmJgsxugoCBUtxyMG8KaaMlq6w22oBipwcvT0Xz7+PAi04M789o54Fob0bwZVUom68MAqFb4TULqYbv8rjGgMNznXdfmhAzWZzUkCi7bJ2QBFJVQEzSJ9UwrdX632XpjbA2ErYIaYYtbTV1rsxiNWQY0DvDj7B79URhRaR+/HsApLKleLwaEm3gfBGXXuQS7Ull2mMfbRignariCqn9T1g9zu77Qv65QZSTfoRydqUJreImUthoxMvMgHON7yVSoqG5nbKt87EqI0BKkAZrn3oK4vBO9U0ADZoS19wrEfQwhkUpZDO14TI8k4VSW4zceYAD/W1hSzUhaMERVE=' 2023-09-06T03:02:35.981 INFO:scrape:1 jobs: ['7388321'] 2023-09-06T03:02:35.981 INFO:scrape:suites: ['1-ceph', '1-tests/fscrypt-ffsb}}', '2-logrotate}', 'bluestore-bitmap', 'clusters/1-mds-1-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/fscrypt/{begin/{0-install', 'ignorelist_health_more', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'pg-warn}', 'tasks/{0-client'] 2023-09-06T03:02:35.982 INFO:scrape: 2023-09-06T03:02:35.982 INFO:scrape:Dead: None 2023-09-06T03:02:35.982 INFO:scrape:1 jobs: ['7388353'] 2023-09-06T03:02:35.983 INFO:scrape: 2023-09-06T03:02:35.983 INFO:scrape:Failure: Host key for server 'smithi104.front.sepia.ceph.com' does not match: got 'AAAAC3NzaC1lZDI1NTE5AAAAIOB/s24jsVWbjV1fcyP5vkvF1H3Kkfv7S76W3oPYqhOZ', expected 'AAAAB3NzaC1yc2EAAAADAQABAAABgQC8jmGZWHMSkjYutKUS6w9V7wIvw7vsI9jYoz8rdgYIwC/TximP5GRDQCoWGCvjfOtjrfFSWW2rdMFFBw7pwDuGOeblVmKVGm/UJnEqZyYyvKMWCGyv1MBaSKMzU/yt4xIca01RC+ocPL6lxH4U2FEI+TrirlSG2ezitQVUkR90J1Xg6hwDB9Dm7JATrj5caheizMhuFr8y8+q3IrDj16VWhLISruWI6JSMKaGGvc6vrGiBmKTsVRD8oEN0HRQPlcq+r92Y40IVlt9E527BhDDeMtZ52i1rPErY7wDg3sS+iY21KjhzFMy0mALSWzdVbe6mOEBJtqvdKfWdPHqVXFtKV3akOr60uW7b/mkabeTOfvIN3xs97+Fqq4pUNkMOT13JKIfaoUQfj+7Jh68bPkaF43NwJetPo7zr4xXZCq5EgBbBuQIwAkjhWTK6sFrySE+NM6OB4tiqA1+GyjLbvmPbf8ZEtTFyHuJkP7Mo3PWykJ6F3D5MOsPiNSRfprxRRjU=' 2023-09-06T03:02:35.984 INFO:scrape:1 jobs: ['7388266'] 2023-09-06T03:02:35.984 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1-mds-2-client', 'conf/{client', 'distros/ubuntu_latest', 'fs/multiclient/{begin/{0-install', 'ignorelist_wrongly_marked_down}', 'mds', 'mon', 'mount/fuse', 'objectstore-ec/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'tasks/cephfs_misc_tests}'] 2023-09-06T03:02:35.984 INFO:scrape: 2023-09-06T03:02:35.985 INFO:scrape:Failure: Command failed on smithi059 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 196fe4f0-4be8-11ee-9ab3-7b867c8bd7da -- ceph orch host ls --format=json' 2023-09-06T03:02:35.985 INFO:scrape:2 jobs: ['7388279', '7388204'] 2023-09-06T03:02:35.985 INFO:scrape:suites intersection: ['1-volume/{0-create', '2-allow_standby_replay/no', '2-client', '3-inline/yes', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_8.stream_container_tools', 'conf/{client', 'fail_fs/no', 'fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'syntax}', 'tasks/{0-from/pacific'] 2023-09-06T03:02:35.986 INFO:scrape:suites union: ['1-ranks/1', '1-ranks/2', '1-volume/{0-create', '2-allow_standby_replay/no', '2-client', '3-inline/yes', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_8.stream_container_tools', 'conf/{client', 'fail_fs/no', 'fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'syntax}', 'tasks/{0-from/pacific'] 2023-09-06T03:02:35.986 INFO:scrape: 2023-09-06T03:02:35.987 INFO:scrape:Failure: Host key for server 'smithi087.front.sepia.ceph.com' does not match: got 'AAAAC3NzaC1lZDI1NTE5AAAAICCBThPdJT1HmLyckij2Eg+t8O/eoEBUntaTwXWKXTUb', expected 'AAAAB3NzaC1yc2EAAAADAQABAAABgQDASRSv0f41Fs3FnKFvGxS+ijKQ65yzHBKRJ2q4JmbG0HHLUgfGyxhYcFI8p0AFkjkb7LOaVnxDYJR/McIHCzXX/NUfgSuqzStmy3wLd1rVEBSj/XvgJi6RZVyqyKT/rczG1BkjgOdKY54A2dH/hRkIXRiVUGfqv7WoT1HcSx9xgem94DJ9PfqPLPLCugrQ3iR1HsTzro3HJzmM7S5adeP+NCYUUv1Mx9cVZPj7uSBtSlpPInnh7NpWtaBtd1KGvgofLSwucrBcFLTyZhyqKE3cqUFCHRjY3VHzRdGMXjgCn2N+Awvd4llibeSjeyNe0x0IvCxl1V7nUALecR+W2hkE/mcsvucQgJxoH8GPFeAIP6tY0y7+YVtLIJCvK13K2TfBp09rUI+kQP0zXjTbjrgI41ekR66WIAFdJrH9mB6dJ8SCXPP0S4nbPw0NC+RJw7d8qFlovzkZ/gQ0AyTkTLZcQrYN2n3q3VCE0hodBMKggU4cOsij8IdwEAjyjBwGiws=' 2023-09-06T03:02:35.987 INFO:scrape:1 jobs: ['7388289'] 2023-09-06T03:02:35.987 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/fixed-2-ucephfs', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/traceless/{begin/{0-install', 'ignorelist_health', 'ignorelist_wrongly_marked_down}', 'mds', 'mon', 'mount/fuse', 'objectstore-ec/bluestore-bitmap', 'osd}', 'overrides/{frag', 'tasks/cfuse_workunit_suites_blogbench', 'traceless/50pc}'] 2023-09-06T03:02:35.988 INFO:scrape: 2023-09-06T03:02:35.988 INFO:scrape:Failure: Host key for server 'smithi144.front.sepia.ceph.com' does not match: got 'AAAAC3NzaC1lZDI1NTE5AAAAIDd0X8b38UkcPXtcNaP3JUC6MzmqxZMJmXBEx07JjSXQ', expected 'AAAAB3NzaC1yc2EAAAADAQABAAABgQDNGTwTISv62m0Uc1OdWBG+FN+mPq6qwnbd2GBUdBzCoqYoXvm6mPA5cYRkFdw+9M+VvBY+CLpvItdB3URTzZd64CWIP5sUDUexyMoLVz7htV6Xc1RN4xkxxotH/wyEp/OiWRCRPXPzAZNiZRwGfnKJRbVyHHfhmLSfWyHMHhf+sDEMLTEFJaBX2CYBow1qhhjWIOhJPmlt4SbRiAfEm6o1qTiodnGGNvlx/fhRQuE/y+1ExdQ//MBI3vj04EyPOCkv3m+beYEJjZIdOio67oySpbpbPF6flcyv1+52/V8saTwvx07jlYsJ/ReYgVJhB2lh0KgO+WnYt4po7ZV+xJ8o+P+lbcu9F1luThk27JIDrqeCZbe91kYGrRi3/fkz/Lprv4Erwu/3BBg5wzcXTlytVa4SI6dNP3Xw1mz+8U2GQFtV5I8y/rDADVchGLSl7ZehDrc1NBqWatis3Q1yxpDWLvs37i2nqIaScfOAaLe54neAYi8gqIBRkW68mGev1I0=' 2023-09-06T03:02:35.989 INFO:scrape:1 jobs: ['7388239'] 2023-09-06T03:02:35.989 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/volumes/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'tasks/volumes/{overrides', 'test/snapshot}}'] 2023-09-06T03:02:35.989 INFO:scrape: 2023-09-06T03:02:35.990 INFO:scrape:Failure: Command failed on smithi158 with status 1: 'sudo yum -y install cephfs-java' 2023-09-06T03:02:35.990 INFO:scrape:2 jobs: ['7388235', '7388185'] 2023-09-06T03:02:35.991 INFO:scrape:suites intersection: ['conf/{client', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}'] 2023-09-06T03:02:35.991 INFO:scrape:suites union: ['1-cephadm', '1-check-counter', '1-ranks/1', '1-volume/{0-create', '2-allow_standby_replay/yes', '2-client', '2-logrotate}', '2-scrub/no', '3-inline/yes', '3-snaps/yes', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-flush/no', '4-verify}', '5-upgrade-with-workload', '5-workunit/fs/norstats}}', '6-verify}}', 'begin/{0-install', 'centos_8.stream_container_tools', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'fail_fs/no', 'frag', 'fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'ms_mode/crc', 'objectstore-ec/bluestore-comp', 'omap_limit/10', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'pg-warn', 'ranks/1', 'roles', 'session_timeout}', 'standby-replay', 'syntax}', 'tasks/{0-from/pacific', 'tasks/{0-subvolume/{with-namespace-isolated}', 'wsync/no}'] 2023-09-06T03:02:35.993 INFO:scrape: 2023-09-06T03:02:35.993 INFO:scrape:Valgrind: mon (InvalidRead, Leak_PossiblyLost, Leak_StillReachable) 2023-09-06T03:02:35.993 INFO:scrape:1 jobs: ['7388310'] 2023-09-06T03:02:35.994 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{ubuntu/{latest', 'fs/verify/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mon-debug', 'mount/fuse', 'objectstore-ec/bluestore-comp', 'osd}', 'overrides/{ignorelist_health', 'overrides}}', 'ranks/3', 'session_timeout}', 'tasks/fsstress', 'validater/valgrind}'] 2023-09-06T03:02:35.994 INFO:scrape: 2023-09-06T03:02:35.995 INFO:scrape:Failure: Test failure: test_disable_enable_human_readable_quota_values (tasks.cephfs.test_quota.TestQuota) 2023-09-06T03:02:35.995 INFO:scrape:1 jobs: ['7388316'] 2023-09-06T03:02:35.995 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_20.04}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'tasks/quota}'] 2023-09-06T03:02:35.996 INFO:scrape: 2023-09-06T03:02:35.996 INFO:scrape:Failure: Host key for server 'smithi096.front.sepia.ceph.com' does not match: got 'AAAAC3NzaC1lZDI1NTE5AAAAIICRbFLhMpLRnbUIqMIIWJvBRq0sjabYAHHeuAzXMuvY', expected 'AAAAB3NzaC1yc2EAAAADAQABAAABgQCixh09rEp5J3M1y+PImPscaCc7fg6nyYz4PmeG1TtdvVLzxhOqg0mSMc5nSi/pQJw+KQhe+SRSc/+a0Xeo8x6gzTkcpEXgvjSNvtCUXPpWHjf5lOwD32l3nmmBtsUwn7ANaSEaI/EGDgL4whTeST/N9T4yTyslKgLxlaT5B1/RuYp69cpn7uJZmCkhHewILWVZM6j0QUtv6LCfZTTynEdkbdQwjTnnblJluwKKNjswHycd2W72ecOIKTxrjQH6VonT9z6gYw7n4IsaHcPYo6is6T7HC3QXLAaULqXvGYSNQ2T2HxJO8vNu5Y4njEx4SwLuze/EE1fqVOwkPIwVs9OUoIgymVdxiu4Uo+tZU4HmMKaXixtGYEwtHOzPbZXwqxAc0dInfPAyCp38d413nqQyd0jLlNN3k9y12/wQK06IjZXiN2WdvDRfS/6zz3WJ6qwV3ficR5K4j/xdD0qbNcON+jIHkdlt8y+H0v9R/0JC9Yu1F4NG92U5snexnBIhb3c=' 2023-09-06T03:02:35.997 INFO:scrape:1 jobs: ['7388349'] 2023-09-06T03:02:35.997 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/volumes/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'tasks/volumes/{overrides', 'test/misc}}'] 2023-09-06T03:02:35.997 INFO:scrape: 2023-09-06T03:02:35.998 INFO:scrape:Failure: Test failure: test_snapshot_remove (tasks.cephfs.test_strays.TestStrays) 2023-09-06T03:02:35.998 INFO:scrape:1 jobs: ['7388186'] 2023-09-06T03:02:35.999 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'tasks/strays}'] 2023-09-06T03:02:35.999 INFO:scrape: 2023-09-06T03:02:35.999 INFO:scrape:Failure: Host key for server 'smithi052.front.sepia.ceph.com' does not match: got 'AAAAC3NzaC1lZDI1NTE5AAAAIFwMNJYOGZEXS/mam0yk8zWBpnV2/ePuJIGA5QQ7OOu1', expected 'AAAAB3NzaC1yc2EAAAADAQABAAABgQCu1F9cOuc1JdwZemXI6x+vjQsoxo5qmi+dyXHIOz4Zb7hFdVuApJfkg8FNE0HDJDT+IRf9fdDDJSOqapwrgKmh9HstHWeOEMsTBWxs6F47TptCjUBLXmRe+qUj6M/Ae19hd8NB1+V1dKlskviY48vQC1Vu5s2qfqctM1E3uCGvwI5D37G5G+tDo55tKLJVmG/5umc/dfK7VLagtsslnurUeT5iISIhKP6BiDmfDySB6PIkaDmkAwSpujTHiJqBG9IC0TqDhOvtlXNuzcLW68gWG9ef68j/NOQPp9I69i4jet0wOPBlKrBDpAOPQKG7YXtUjv0vhaiJAIGfpSRF6W1f9JDrR0yZwtUJ+Zqgw+iEoTqUcwfMwOBx8naw9kwIm8uXXXk5VUgPKjdrVmRyF9t2AoxuBxzY9HV3VFXLzydDPQUifdY4hF/gjL056bGQvGjPqVtOpRFLRpH8iA6ROYNwkeW0/ikMtigxdnR6Q/s9Fz2j1Fykf0hja/pt/K6DeE8=' 2023-09-06T03:02:36.000 INFO:scrape:1 jobs: ['7388296'] 2023-09-06T03:02:36.000 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/fixed-2-ucephfs', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/permission/{begin/{0-install', 'ignorelist_wrongly_marked_down}', 'mds', 'mon', 'mount/fuse', 'objectstore-ec/bluestore-comp-ec-root', 'osd}', 'overrides/{ignorelist_health', 'tasks/cfuse_workunit_suites_pjd}'] 2023-09-06T03:02:36.000 INFO:scrape: 2023-09-06T03:02:36.001 INFO:scrape:Failure: Host key for server 'smithi007.front.sepia.ceph.com' does not match: got 'AAAAC3NzaC1lZDI1NTE5AAAAIAZ8bn1DHrDtuPYFrBpcjdcAY8srHHwKz8gBmEbonWpw', expected 'AAAAB3NzaC1yc2EAAAADAQABAAABgQC35y/lnvAX8KextUvgRYkHZQOP+ugDe1xsmc6eFaoez2U2UyPVFUbFSBefae+B0pBWr3DXDy9Lk3BioZUtNPsxaQQJ5bayl0sRwfKTk3VzzF2qNTG/TGVOapxZBb0wIJT7epy/KnkBTVDFhuCzt+LA4LjhAFfJ+8IH2KlJNELN2PJTUxx5gYnyJe5ymYqsgzQ67f4yi3synwaqMbuC+07/laD1UHxCw7SUAKHDUgD0Xbdgr5mzIeqRWo1BVx0xG8InR0KDWYv6CfoeXG5eCAjLzNGRjj9davG6kT+WyTPiUs6/vc/ntGy6m19F6mp5U6e+H/NolwAB6SKwPc1PFaszPunDNqsNWX0nhpyw0STXbWKEo9+EpkLttzHoccvKjWb1XM/DDEsVa5o1VLd1T6BTUognXdX1jhbZOK/xA5HMjjRqTJjDA90hUk0gSMUqnygPUWVZBlBu3eMHOk+f+xsyl85ihwLLaE8jRq5gnUE9JSJt0sXBKOYx4+stQE3LpvE=' 2023-09-06T03:02:36.001 INFO:scrape:1 jobs: ['7388268'] 2023-09-06T03:02:36.002 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/fixed-2-ucephfs', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/permission/{begin/{0-install', 'ignorelist_wrongly_marked_down}', 'mds', 'mon', 'mount/fuse', 'objectstore-ec/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'tasks/cfuse_workunit_misc}'] 2023-09-06T03:02:36.002 INFO:scrape: 2023-09-06T03:02:36.002 INFO:scrape:Failure: Host key for server 'smithi019.front.sepia.ceph.com' does not match: got 'AAAAC3NzaC1lZDI1NTE5AAAAIDeH4bqzR8hf+lroo4SUwkAh6jJWaxoyznsd1FeBIt2U', expected 'AAAAB3NzaC1yc2EAAAADAQABAAABgQDT0g4ZDixzZb/2Y65pwhyhD5LjtLRJV7iM/EAYauhIEyhhpmC5nCHg/efuHXGtbSgK4VBC2ZSjxtVaanIhK1AugX+1Xlzoy9ygNNkYfQQvjg7OjUvAkPXFnZhU0XIffXh2BKOtbEO8H4Mzl3g20Qy960qOXdAtqBlcohaSQvdAnLH3YTOFYhNi8I4zV5kvfsIpydAdtedxElsMfrutQ5VNfwvL6VRS3gPw4IER1A/72ZR1WlvxIuQFgE/Mg2C6yW+1FsinmmAFkkHCn6fm+hmjeEHzTkWR5lV5U3TJe/v42NCGKICepFAhmBSjTv0q+xZE1q0NM102yHp0k4eH1RqC/6uZMysMN6UFtcA2wfefWGedHD0OGlHOI1Yi6dC7A2ceTXyd1jsVlV1KuJC5K8HQ5ztDKQ36X9nUkavqj+O60NCx/hm2OzqAzgmTBSGwv0N54Y+hH6W7imSQuJy9m733//ZqbJFCgBSJ3LyX/yzAd9IgKxY8W8UJ7mdCKbVfnic=' 2023-09-06T03:02:36.003 INFO:scrape:1 jobs: ['7388169'] 2023-09-06T03:02:36.003 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1-mds-2-client', 'conf/{client', 'distros/ubuntu_latest', 'fs/multiclient/{begin/{0-install', 'ignorelist_wrongly_marked_down}', 'mds', 'mon', 'mount/fuse', 'objectstore-ec/bluestore-comp-ec-root', 'osd}', 'overrides/{ignorelist_health', 'tasks/cephfs_misc_tests}'] 2023-09-06T03:02:36.004 INFO:scrape: 2023-09-06T03:02:36.004 INFO:scrape:Failure: error during scrub thrashing: rank damage found: {'backtrace'} 2023-09-06T03:02:36.004 INFO:scrape:1 jobs: ['7388176'] 2023-09-06T03:02:36.005 INFO:scrape:suites: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/yes', '3-snaps/no', '4-flush/yes', '5-workunit/suites/ffsb}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'n/3', 'objectstore-ec/bluestore-comp-ec-root', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'ranks/multi/{export-check', 'replication/default}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-namespace-isolated-and-quota}'] 2023-09-06T03:02:36.005 INFO:scrape: 2023-09-06T03:02:36.006 INFO:scrape:Failure: Host key for server 'smithi089.front.sepia.ceph.com' does not match: got 'AAAAC3NzaC1lZDI1NTE5AAAAIDe0KZusXAy3hcyh7yAfd/ii8w41dA9OboRgiTOj+Zne', expected 'AAAAB3NzaC1yc2EAAAADAQABAAABgQDUSFZbey49O+ZQbtWmyvBx5qyxDUcP4O49PxovirJOlb1I3Qbbn3mGsdKu4pkEI7V9TJzGEQBaa6JwTxafr/iKYk2gO8+u9WPcHfIdDJc6zdSz50Ik8KJ+bn+eLmHYQFSR/yP0i6fII4W4M4rTuiixLlTnDf8m5Ja9S2KmT3dJEqOkRPeyqN66X93eOQAENLpcKcM9XqZ49Ehtwe6W/Vkt4ILGM/CPhgr2ogqP6Q4Od5xf59p5SC032MrCSySvirbycWcOp6PCcXnzoyy+vMgr4RwY/nA2iP/YXZGiCJghZNTWGfBOTL5BmmreqkuGsSFQ7TCvZqKd7+dqWAqIbfZP09ewNyXh0qHX65IoZJPombjQY8RtUM8BEymY4fkg2t1ryUKHLtZ1gjEaZrFbqIow6wyObCJD4NF/EbXwWYVJhLCCoLeeh6m4Xrh7ETid8WsZtL+oS7mjX809QABl3XWlDKFXWQnvs1dbn5CzyI0jTb0sgZUYoisU0YoRZFPnvkM=' 2023-09-06T03:02:36.006 INFO:scrape:1 jobs: ['7388231'] 2023-09-06T03:02:36.006 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1-mds-2-client', 'conf/{client', 'distros/ubuntu_latest', 'fs/multiclient/{begin/{0-install', 'ignorelist_wrongly_marked_down}', 'mds', 'mon', 'mount/fuse', 'objectstore-ec/bluestore-ec-root', 'osd}', 'overrides/{ignorelist_health', 'tasks/mdtest}'] 2023-09-06T03:02:36.007 INFO:scrape: 2023-09-06T03:02:36.007 INFO:scrape:Failure: Command failed on smithi157 with status 100: 'sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=18.0.0-5849-g9d3f068c-1focal ceph-mds=18.0.0-5849-g9d3f068c-1focal ceph-common=18.0.0-5849-g9d3f068c-1focal ceph-fuse=18.0.0-5849-g9d3f068c-1focal ceph-test=18.0.0-5849-g9d3f068c-1focal radosgw=18.0.0-5849-g9d3f068c-1focal python-ceph=18.0.0-5849-g9d3f068c-1focal libcephfs1=18.0.0-5849-g9d3f068c-1focal libcephfs-java=18.0.0-5849-g9d3f068c-1focal libcephfs-jni=18.0.0-5849-g9d3f068c-1focal librados2=18.0.0-5849-g9d3f068c-1focal librbd1=18.0.0-5849-g9d3f068c-1focal rbd-fuse=18.0.0-5849-g9d3f068c-1focal python3-cephfs=18.0.0-5849-g9d3f068c-1focal cephfs-shell=18.0.0-5849-g9d3f068c-1focal cephfs-top=18.0.0-5849-g9d3f068c-1focal cephfs-mirror=18.0.0-5849-g9d3f068c-1focal' 2023-09-06T03:02:36.007 INFO:scrape:1 jobs: ['7388244'] 2023-09-06T03:02:36.008 INFO:scrape:suites: ['1-ceph', '2-logrotate}', '2-workunit/fs/trivial_sync}}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{ubuntu_20.04}', 'fs/thrash/workloads/{begin/{0-install', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'msgr-failures/none', 'objectstore-ec/bluestore-ec-root', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{frag', 'prefetch_dirfrags/yes', 'prefetch_entire_dirfrags/yes', 'races', 'ranks/3', 'session_timeout', 'tasks/{1-thrash/mon', 'thrashosds-health}'] 2023-09-06T03:02:36.008 INFO:scrape: 2023-09-06T03:02:36.009 INFO:scrape:Failure: Host key for server 'smithi159.front.sepia.ceph.com' does not match: got 'AAAAC3NzaC1lZDI1NTE5AAAAIOb3SstfDBSJ0g7HBo2dIKX0FT1spKYJ19SQpkCq/fvA', expected 'AAAAB3NzaC1yc2EAAAADAQABAAABgQDADn0rd2aMYBj1YOLWiFiwtYfR9HojGxmXKQZJxu0ZtB4Rtj9LOKjrjtAK0ywk3o1jCblgCZTPxVZVpppRqMoFMS83qGakDzar1cqNksxD5qV8PpqjrPipCwR1k1EJRj+lF1Kc9zqA1sc9FCj0SrRfoHhkASnk8qg5YDCCZDwbrbr9hYRdQzXEOaO5F6jjA978tKJpyKVXD7oOdv5ECUfqse1Iuc1C8Poxd4ZkcU/4Emu8L/NlRXoyuUgCdU5McYMO2kw9m4nmV/y5rVi8ebJu/reSXFX9WWbssy2T4X+P0h6g86av7JSJghk3xUI1BZ9JsxiwQsBfIh6J5AI28N8TVkzwbLZAEC6IuAOc3qzC0sbzZmn6YApXHLTefoGmN+TOMYQgwWPsKzMtjjBmhmbpQxr27cXKctmZWr62dVa9QOvg9Bt4ac7OrSIkLpiDE2hBf7pYNLfJIWKQ8q2q0Ae8vjOB2Ko95ug44/+kfPjqX3EFHa1GU0g3z7Co5by1Is0=' 2023-09-06T03:02:36.009 INFO:scrape:1 jobs: ['7388181'] 2023-09-06T03:02:36.009 INFO:scrape:suites: ['1-ceph', '2-logrotate}', '2-workunit/cfuse_workunit_trivial_sync}}', 'clusters/1a3s-mds-2c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/thrash/multifs/{begin/{0-install', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'msgr-failures/osd-mds-delay', 'multifs', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{frag', 'session_timeout', 'tasks/{1-thrash/mon', 'thrashosds-health}'] 2023-09-06T03:02:36.010 INFO:scrape: 2023-09-06T03:02:36.010 INFO:scrape:Failure: Host key for server 'smithi026.front.sepia.ceph.com' does not match: got 'AAAAC3NzaC1lZDI1NTE5AAAAIPZp6wOILlcOVYZjlBmIruM6AFvsaBMKMhL+T3kylK03', expected 'AAAAB3NzaC1yc2EAAAADAQABAAABgQCTqi4QsBDDKbSS87JrAL5PCT9gK8O7STWIlhRnX+v5NqDPeErkcCeFQkNiLn0VxjdfUXHvIaOp2gpcKhd3Vjcb+9sHPtiNoaMWWApISdpMTZR/dUsy3/RYFkn/S2rZOz9PqNxkI7VUTu2jZuuTnv01a49FFTvaxYQrLU1VufWHxD4D8lpqhVi7sV3nWJCjpiM6cz1XM1aqUuXEiyaRR5sItrLWA4NbB5sRZcHaL775vrHTIoCj52oyliTdXcRp4TXjtPcOh+kK9UhTJKRpqmSVgh0Iw+SvM3hlNmzzaBPBnKxSOxbZda5SvZuDsi+p/dKa1Igeq9N2Z2EVfzT+eDUsDtXpczv6rdEWpUM2nl+J3M5kTDg/sLfQJJGS3YLbudCjof/FoiuTmSVW0X2tf+d+jYurb7p1STyNJDSfbQoaU1Qr7mka9PjWmh0hHVtGuoTyPwAbdKoE4wUjUDYMyioTjB2Yr529Lro2UTN7d32sCe7BQeCWBE8SQQmKsc6cDd8=' 2023-09-06T03:02:36.011 INFO:scrape:1 jobs: ['7388335'] 2023-09-06T03:02:36.011 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1-mds-2-client', 'conf/{client', 'distros/ubuntu_latest', 'fs/multiclient/{begin/{0-install', 'ignorelist_wrongly_marked_down}', 'mds', 'mon', 'mount/fuse', 'objectstore-ec/bluestore-comp', 'osd}', 'overrides/{ignorelist_health', 'tasks/mdtest}'] 2023-09-06T03:02:36.011 INFO:scrape: 2023-09-06T03:02:36.012 INFO:scrape:Failure: Host key for server 'smithi107.front.sepia.ceph.com' does not match: got 'AAAAC3NzaC1lZDI1NTE5AAAAIHu0jKzltV4lyU8/IqLVQHdQQi7+e72cMVhmWrLR+NFi', expected 'AAAAB3NzaC1yc2EAAAADAQABAAABgQCUq0SfhkeoOQSbWAIJSVnxnSYMpnzbDt0wWiUx+glK/E+4q64qnLSM9sRV347iQmsY4AHvL9MOrTBIGBLEm6xHVHRNNZi/1hsn5gRv/sZy1KAQ6A9Idv7y3mvvYe3HCzz+MwQ5syWrQ98HeYG3Ukqo5xmtFWvXNmHf8nAySwJmwQYs/WJapNuWNUZCQTLbUtGMrP7UL6TBLmNlt+UOaQClv3XhWWMvgEa2rQxo7dRlOWFOSb/VdVZI+CcjvlgMVfHB6fplkU6yjlb28EfW27iRJr4iZqgz0iHHuN6p5zNMdq7Kf8VBrTUehgXivzwVh+j3abXXFPzhEGj7e212NTPKkGXoiBDQMBM80ggjfWfHiqYe//Og/TbgSHyPFw6nBt5F47NyAvcOyT7hoagIoce9HNpiiAU4PzwVUBAaJfxUi5qHFa/Zq189gtH3FxV9XSdBjNfM/iAv/QHQeKBGS3aGIEQCZmxuG14sobmkEv2+jAz3k3XJoZswOmnL41NB7gE=' 2023-09-06T03:02:36.012 INFO:scrape:1 jobs: ['7388325'] 2023-09-06T03:02:36.012 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-2c-client', 'conf/{client', 'distro/{ubuntu/{latest', 'fs/mixed-clients/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'kclient-overrides/{distro/testing/k-testing', 'mds', 'mon', 'ms-die-on-skipped}', 'objectstore-ec/bluestore-comp-ec-root', 'osd-asserts}', 'osd}', 'overrides/{ignorelist_health', 'overrides}}', 'tasks/kernel_cfuse_workunits_untarbuild_blogbench}'] 2023-09-06T03:02:36.013 INFO:scrape: 2023-09-06T03:02:36.013 INFO:scrape:Failure: Command failed (workunit test suites/dbench.sh) on smithi023 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=9d3f068cec6c5fb39569ca38ddb0f19fce7352a2 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/dbench.sh' 2023-09-06T03:02:36.014 INFO:scrape:1 jobs: ['7388162'] 2023-09-06T03:02:36.014 INFO:scrape:suites: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/no', '3-snaps/yes', '4-flush/no', '5-workunit/suites/dbench}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'ms_mode/secure', 'n/5', 'objectstore-ec/bluestore-bitmap', 'omap_limit/10', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/testing/k-testing', 'ranks/multi/{export-check', 'replication/always}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-no-extra-options}', 'wsync/no}'] 2023-09-06T03:02:36.014 INFO:scrape: 2023-09-06T03:02:36.015 INFO:scrape:Failure: Command failed (workunit test fs/test_python.sh) on smithi131 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=9d3f068cec6c5fb39569ca38ddb0f19fce7352a2 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/test_python.sh' 2023-09-06T03:02:36.015 INFO:scrape:1 jobs: ['7388219'] 2023-09-06T03:02:36.016 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1-mds-1-client-coloc', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/libcephfs/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'tasks/libcephfs_python}'] 2023-09-06T03:02:36.016 INFO:scrape: 2023-09-06T03:02:36.016 INFO:scrape:Failure: Host key for server 'smithi118.front.sepia.ceph.com' does not match: got 'AAAAC3NzaC1lZDI1NTE5AAAAILtURNgi6yyNTUi4mh1PMkzVCq6Pw2DLM4dR7pqzMB1E', expected 'AAAAB3NzaC1yc2EAAAADAQABAAABgQDITUwYyDNVd9scXmBRkKQtjFQ522KWzrFoXygmpc8CIIp4LTzOePO53J0hlkUtzZqt6DYfJPVeDU24pRKidEClzHuWdijGq6KkptRxRtVnbI8GGeO7dEIIfP0wsrt9DADMxZvfKlq0YFqp6J8dmNf4eHOsrP501Gl1BJqbrVMEpgl599bq2Ot28uERsxM5ADLa2H2aAzgNNfwrFEYiVVSaaGJ2Z1ZBju1ot9C4QuoFK4QbgG1OkbGIJefRbjUDA/Hm9lseSeqx0BVY90StvqLtPlXsTsPUuw2dacikrB370y1IuFg/lCVBAR3ZCLic+vaA1ubOZotyQ0xWgXl01bOLWrO/i5fECj9cMxRAZjFvjorF8Ik1ffxoWtdRovPnBwnK2pFKaJ9bmpWqy82vJem8WHoccWYUkrqRWFqVwh1L2v9YC0rUnu4L/RhNNsEuSzUqf2C34wen1BUune/Szp7xxuUHxStw0SaEMxcukcBKJ/Z/iPrIC4v89ePSzMts0K0=' 2023-09-06T03:02:36.017 INFO:scrape:1 jobs: ['7388241'] 2023-09-06T03:02:36.017 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/fixed-2-ucephfs', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/32bits/{begin/{0-install', 'ignorelist_health', 'ignorelist_wrongly_marked_down}', 'mds', 'mon', 'mount/fuse', 'objectstore-ec/bluestore-ec-root', 'osd}', 'overrides/{faked-ino', 'tasks/cfuse_workunit_suites_fsstress}'] 2023-09-06T03:02:36.018 INFO:scrape: 2023-09-06T03:02:36.018 INFO:scrape:Failure: Host key for server 'smithi017.front.sepia.ceph.com' does not match: got 'AAAAC3NzaC1lZDI1NTE5AAAAIJwDHjpXiLzq7xnpjgxT7R4ubS9U+rNWfzZ/U/V6NO/G', expected 'AAAAB3NzaC1yc2EAAAADAQABAAABgQCO9qHbIxuv1YIKO+ICusv7rdu4Z1DfTIRSosc1NIh6aiXrPuq9xObx+6s4BHME3P3DE9pLW5EUJ8SIXzAhJSqJOTa84N8A9XHJF+29x8tgsDfUKCz5pCuoKXxkxSt16qYt1BOrLu2LSvpdVcAxu20/jhK8sMJSA7hLIRn0ZZtZyBAhRC8pU+YhkTroCVd3gMysPeqOSq7+JFSGMZ4t17LDrt49QvwpwZ7e9tmYSrXQuv8IdY//52KlCOmowzSXZf1/fWsmtVq7pTIJkmW/sZDICu33Fyvv/n23AtzhF2lYYvkz/4EKnhgJCmfGr9P3G762Zi83czcCpUG7OvkVd7pOxTzWknM17rAI5KMUfkkTUcad3b1/QGO6ibPNf7wMkscxt3PkUs6Zm53gqCkAYZltLBtDhsTPlp737wXLIPSmFH4I/xpZv2dzhQjIj2IhHxtWlpD3Dsb68da8KSCEX5BNHkX07VmvOXxEXA46chpLu8384zMhv2E5w8jzQQejFKU=' 2023-09-06T03:02:36.018 INFO:scrape:1 jobs: ['7388314'] 2023-09-06T03:02:36.019 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'tasks/pool-perm}'] 2023-09-06T03:02:36.019 INFO:scrape: 2023-09-06T03:02:36.019 INFO:scrape:Failure: Host key for server 'smithi104.front.sepia.ceph.com' does not match: got 'AAAAC3NzaC1lZDI1NTE5AAAAIFg8qLZboxZbUQ7nI6Dda6dxG/+x1Wln7dnEOklGWNR0', expected 'AAAAB3NzaC1yc2EAAAADAQABAAABgQCr0BX7cpOmQMEEHvf3vS0Hc1eGERmaoShI5dOsgJ3/kaJ0QfKRyYxj9v9NLqoX9cHNLdD4UGsQmuXQtx03Ad2+MHXoC4xmDCdp93qQmRNZukuT9JsMTqPYTU5V42ZtZ6Rx3t38V0tmQdL5JonZv0xT9MtjQgBJ0ouN/VjMZcoCUZ6aPN0jMIXAVmUU7OgPATkO8NQwt0x7X9pJgq/9CMk/jHVKymzhJq18YU8vxQza1U/XtBxNNpVVOzoDp2CL8veT0cVCtSoDS2RMnAnoWlulW2DUZ9SpfdWzmUYot6IyaWdXbLmSy9aQi+hUh57CpXt8CK0xVB62324HHrOooT8sdh9CuCASDxRPIBkc5TWuXulvMaPdLz8QfyMCKJeAGTkvq1PffmPHmWR2KxZ0au9Sa6R4e4ieDuZCaP0nT1bwC0IAJ2fS6AbjDb7h/23wlSClXem2C/Qfy5CfMp2EwDQZyGs/5zvW078C8+6VQxc1ALTIP54gleiz1DHzLAYqIck=' 2023-09-06T03:02:36.020 INFO:scrape:1 jobs: ['7388238'] 2023-09-06T03:02:36.020 INFO:scrape:suites: ['1-ceph', '1-tests/fscrypt-pjd}}', '2-logrotate}', 'bluestore-bitmap', 'clusters/1-mds-1-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/fscrypt/{begin/{0-install', 'ignorelist_health_more', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'pg-warn}', 'tasks/{0-client'] 2023-09-06T03:02:36.021 INFO:scrape: 2023-09-06T03:02:36.021 INFO:scrape:Failure: Host key for server 'smithi042.front.sepia.ceph.com' does not match: got 'AAAAC3NzaC1lZDI1NTE5AAAAIO5pg356dEuedI0vJKiCLAAQYSIzibQek8cpeqWboA09', expected 'AAAAB3NzaC1yc2EAAAADAQABAAABgQC3TpfIWWqYwhzBOCZBwGH0vp3i/VxqS4ueiIsm/xVpASL8aRjT5eR6TyLNHSttzwgyYCxyTxDRZEhzu2KDP7YaaRzUJApCab4/2ALaVwYVdTSn/o+GD3LmcP0IG7MQfbL1KbvksNuRy8leuhuKnmMaWQqJcSvaRAvLBhr3bTl0pWo13qbH8yJ2oDefdL7Yp9b0Vka3RaODKU3lgXi4DQ4GqpGI7N05dzedeFkjR0ykRu1euhM7z3sizu4dzpemFIpjDZgE8G6/hlSQ4E54/LPGFhHIDsb5KpHCBtdOIOd7t0KEGYb7ae/UgjE0dfdTqZ9loBbL/Wq0IwjJd1u3PcSzuZCS+7LmzYbd+ZHdaIuRlpoiAYeejI2IkVH6zuK2buyaY2pHReU3CNfziruAexXe+1ikgIW5tbMvB+ju67AP4xz/6s/v9Pbl9STXrAhhwmywUkdua1HocKEd56pTnFaS2R2xnrB7OqmEySiMRqHXRAI9w7IXQpz/ry1FGDeIQW0=' 2023-09-06T03:02:36.021 INFO:scrape:1 jobs: ['7388312'] 2023-09-06T03:02:36.022 INFO:scrape:suites: ['1-ceph', '2-logrotate}', '2-workunit/suites/iozone}}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/thrash/workloads/{begin/{0-install', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'msgr-failures/osd-mds-delay', 'objectstore-ec/bluestore-comp', 'osd}', 'overrides/{frag', 'prefetch_dirfrags/no', 'prefetch_entire_dirfrags/yes', 'races', 'ranks/5', 'session_timeout', 'tasks/{1-thrash/mon', 'thrashosds-health}'] 2023-09-06T03:02:36.022 INFO:scrape: 2023-09-06T03:02:36.023 INFO:scrape:Timeout 6h running clone.client.0/qa/workunits/suites/fsstress.sh 2023-09-06T03:02:36.023 INFO:scrape:1 jobs: ['7388227'] 2023-09-06T03:02:36.023 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{rhel_8}', 'fs/verify/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mon-debug', 'mount/fuse', 'objectstore-ec/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'ranks/5', 'session_timeout}', 'tasks/fsstress', 'validater/valgrind}'] 2023-09-06T03:02:36.024 INFO:scrape: 2023-09-06T03:02:36.033 INFO:teuthology.results:Sending results to mchangir@redhat.com: Test Run: mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi ================================================================= info: http://pulpito.ceph.com/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/ failed: 65 dead: 6 running: 1 waiting: 0 queued: 0 passed: 125 Fail ================================================================= [7388161] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/clone}} ----------------------------------------------------------------- time: 01:01:08 info: http://pulpito.ceph.com/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388161/ log: http://qa-proxy.ceph.com/teuthology/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388161/ "2023-09-05T11:26:04.170981+0000 mon.a (mon.0) 217 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7388162] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/suites/dbench}} ----------------------------------------------------------------- time: 00:54:19 info: http://pulpito.ceph.com/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388162/ log: http://qa-proxy.ceph.com/teuthology/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388162/ sentry: https://sentry.ceph.com/organizations/ceph/?query=b1db44a2b9744134b05c7c489ca0cc24 Command failed (workunit test suites/dbench.sh) on smithi023 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=9d3f068cec6c5fb39569ca38ddb0f19fce7352a2 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/dbench.sh' [7388168] fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{centos_latest} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/libcephfs/{frag test}} ----------------------------------------------------------------- time: 00:30:15 info: http://pulpito.ceph.com/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388168/ log: http://qa-proxy.ceph.com/teuthology/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388168/ "2023-09-05T11:26:18.800097+0000 mon.a (mon.0) 130 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7388169] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cephfs_misc_tests} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388169/ log: http://qa-proxy.ceph.com/teuthology/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388169/ sentry: https://sentry.ceph.com/organizations/ceph/?query=160004cfbec240cea7b5cef18e594247 Host key for server 'smithi019.front.sepia.ceph.com' does not match: got 'AAAAC3NzaC1lZDI1NTE5AAAAIDeH4bqzR8hf+lroo4SUwkAh6jJWaxoyznsd1FeBIt2U', expected 'AAAAB3NzaC1yc2EAAAADAQABAAABgQDT0g4ZDixzZb/2Y65pwhyhD5LjtLRJV7iM/ EAYauhIEyhhpmC5nCHg/efuHXGtbSgK4VBC2ZSjxtVaanIhK1AugX+1Xlzoy9ygNNkYfQQvjg7O jUvAkPXFnZhU0XIffXh2BKOtbEO8H4Mzl3g20Qy960qOXdAtqBlcohaSQvdAnLH3YTOFYhNi8I4 zV5kvfsIpydAdtedxElsMfrutQ5VNfwvL6VRS3gPw4IER1A/72ZR1WlvxIuQFgE/Mg2C6yW+1Fs inmmAFkkHCn6fm+hmjeEHzTkWR5lV5U3TJe/v42NCGKICepFAhmBSjTv0q+xZE1q0NM102yHp0k 4eH1RqC/6uZMysMN6UFtcA2wfefWGedHD0OGlHOI1Yi6dC7A2ceTXyd1jsVlV1KuJC5K8HQ5ztD KQ36X9nUkavqj+O60NCx/hm2OzqAzgmTBSGwv0N54Y+hH6W7imSQuJy9m733//ZqbJFCgBSJ3Ly X/yzAd9IgKxY8W8UJ7mdCKbVfnic=' [7388173] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snap-schedule} ----------------------------------------------------------------- time: 01:46:49 info: http://pulpito.ceph.com/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388173/ log: http://qa-proxy.ceph.com/teuthology/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388173/ "2023-09-05T11:20:39.861764+0000 mon.a (mon.0) 424 : cluster [WRN] Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7388176] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/ffsb}} ----------------------------------------------------------------- time: 01:04:53 info: http://pulpito.ceph.com/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388176/ log: http://qa-proxy.ceph.com/teuthology/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388176/ error during scrub thrashing: rank damage found: {'backtrace'} [7388177] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snap_schedule_snapdir} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388177/ log: http://qa-proxy.ceph.com/teuthology/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388177/ sentry: https://sentry.ceph.com/organizations/ceph/?query=39cb2b99fce14d0b92a51ee71c0444c0 Host key for server 'smithi033.front.sepia.ceph.com' does not match: got 'AAAAC3NzaC1lZDI1NTE5AAAAII2b8+EHn0tR7UVQyFU4c9cfLZglh/CJaxT5iamCaOjc', expected 'AAAAB3NzaC1yc2EAAAADAQABAAABgQDgt/+m+Zx3ddB7u5WO5g/qF/s8evT93NNbd C+4MOjU2BoV/iWn6Gv9I7WhfVM+i557fgOj5eUMc7C2Bwp691ZtZARduEyt2U5FkF2jwLmI7bpM 9DBDuedosDjAqtS6fk0PqVmlMtyJhZ5E6OBN472dkceH3+TGMHk3OVMA/lnwPpQonhb3oILDQQG 8aANxFI7+JSZ0KJxhuNn9ZUdUK1sjp1FeH+sV4rGt2zzQpiub0atq/wBJ8eSy3BdoBYEuN3Pz/b Bz4wyZec4ZkI1P2yMjp7dxPAFmwlOH2Q9e4qjtw90OOZdF8j1WwQx7umDJYoO3uAwDDJaWzW5VJ a5aaYmgbP5K4ZUxAOkpKMcPNRtCEr5Ay+586lD8LX1M+9ynPgM3X6NBK17HgCIIp3IeRd3tD9im opTHPSIxPkSsQ35rp0HMUEiTM4zQfeV7fknUiRJ+puqeQjeig8s1V13peoSdiflC5ldnM5ybVpa JURqSBTxXXmA+Q2Y+crb7eKbyfzc=' [7388179] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_dbench traceless/50pc} ----------------------------------------------------------------- time: 00:41:48 info: http://pulpito.ceph.com/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388179/ log: http://qa-proxy.ceph.com/teuthology/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388179/ "2023-09-05T11:25:57.218416+0000 mon.a (mon.0) 137 : cluster [WRN] Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7388181] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_trivial_sync}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388181/ log: http://qa-proxy.ceph.com/teuthology/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388181/ sentry: https://sentry.ceph.com/organizations/ceph/?query=0e0c24c6aff84e358dc01b8c3c4beedf Host key for server 'smithi159.front.sepia.ceph.com' does not match: got 'AAAAC3NzaC1lZDI1NTE5AAAAIOb3SstfDBSJ0g7HBo2dIKX0FT1spKYJ19SQpkCq/fvA', expected 'AAAAB3NzaC1yc2EAAAADAQABAAABgQDADn0rd2aMYBj1YOLWiFiwtYfR9HojGxmXK QZJxu0ZtB4Rtj9LOKjrjtAK0ywk3o1jCblgCZTPxVZVpppRqMoFMS83qGakDzar1cqNksxD5qV8 PpqjrPipCwR1k1EJRj+lF1Kc9zqA1sc9FCj0SrRfoHhkASnk8qg5YDCCZDwbrbr9hYRdQzXEOaO 5F6jjA978tKJpyKVXD7oOdv5ECUfqse1Iuc1C8Poxd4ZkcU/4Emu8L/NlRXoyuUgCdU5McYMO2k w9m4nmV/y5rVi8ebJu/reSXFX9WWbssy2T4X+P0h6g86av7JSJghk3xUI1BZ9JsxiwQsBfIh6J5 AI28N8TVkzwbLZAEC6IuAOc3qzC0sbzZmn6YApXHLTefoGmN+TOMYQgwWPsKzMtjjBmhmbpQxr2 7cXKctmZWr62dVa9QOvg9Bt4ac7OrSIkLpiDE2hBf7pYNLfJIWKQ8q2q0Ae8vjOB2Ko95ug44/+ kfPjqX3EFHa1GU0g3z7Co5by1Is0=' [7388185] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/fs/norstats}} ----------------------------------------------------------------- time: 00:19:07 info: http://pulpito.ceph.com/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388185/ log: http://qa-proxy.ceph.com/teuthology/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388185/ sentry: https://sentry.ceph.com/organizations/ceph/?query=3ec9a0f212f745edb3609bb798014b82 Command failed on smithi184 with status 5: 'sudo systemctl stop ceph-1a52ed58-4bdf-11ee-9ab3-7b867c8bd7da@mon.b' [7388186] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/strays} ----------------------------------------------------------------- time: 00:32:44 info: http://pulpito.ceph.com/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388186/ log: http://qa-proxy.ceph.com/teuthology/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388186/ sentry: https://sentry.ceph.com/organizations/ceph/?query=71ead77083a04269b2948e662b6c8842 Test failure: test_snapshot_remove (tasks.cephfs.test_strays.TestStrays) [7388187] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/test_journal_migration} ----------------------------------------------------------------- time: 00:23:41 info: http://pulpito.ceph.com/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388187/ log: http://qa-proxy.ceph.com/teuthology/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388187/ "2023-09-05T11:31:56.509547+0000 mon.a (mon.0) 314 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7388189] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/ffsb}} ----------------------------------------------------------------- time: 00:43:55 info: http://pulpito.ceph.com/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388189/ log: http://qa-proxy.ceph.com/teuthology/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388189/ "2023-09-05T11:55:06.960035+0000 mds.c (mds.0) 1 : cluster [WRN] evicting unresponsive client smithi123: (4926), after waiting 47.2263 seconds during MDS startup" in cluster log [7388200] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/ior-shared-file} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388200/ log: http://qa-proxy.ceph.com/teuthology/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388200/ sentry: https://sentry.ceph.com/organizations/ceph/?query=d4ce6a5f95454b74b7d839b016f6f38f Host key for server 'smithi032.front.sepia.ceph.com' does not match: got 'AAAAC3NzaC1lZDI1NTE5AAAAIDOtm5am3jRJYE/T1fxj9/g+WXT6PoaQiADP5H0gdFe0', expected 'AAAAB3NzaC1yc2EAAAADAQABAAABgQC2tMjUxLR1CzgQy34DbMB6w2ntYb+VKflQ0 Lst2baI41oz7RySTQ8BZXmm4qbRQSjZODLpbvC5nCtb5zTftAuLNKuGV//exhPmq9UtunHTKN+7 eLfm2fUQPE8kCD7w8ZqxvTZjl6W8Z/Y7IDHMDoF78dEEz98ePdwZu6wWTADfBzSE+mYQ5oJuWLZ a1gToYaBroMaWS72mzWNnDRYNqU6h3J4JfOkq0d36XKOltWiDYsrT9Jd/Rg4g1MVJ2CPY/8zTLz A3NkSDqyzYG7Vp6RcSfH5Gzd8HIOyVkajZ7rdWsDFlaYuLiykpiHRN5p0cFJArq2Rv0WdEUuKPP PlRk9832ZZGYXhAJZhHFM+3EBtlL9Y73Y1MoYB4DPStUZJ1aULjwaTGdOS/LyFfJOBorMxFM4DY GFF2S0QMZT+7W5i1/04cOAJoxuX2KPhzPJgfAeb1LWOiLovXcY04+H1I5it8dHeinP//ABKMBWq E8iJ/g/Osa/vC1hPqD2r7JJXO5Tk=' [7388204] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:18:11 info: http://pulpito.ceph.com/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388204/ log: http://qa-proxy.ceph.com/teuthology/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388204/ sentry: https://sentry.ceph.com/organizations/ceph/?query=dc93ee17e17942a782ab046409348dd1 Command failed on smithi156 with status 127: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 553c70f0-4be0-11ee-9ab3-7b867c8bd7da -- ceph mon dump -f json' [7388207] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/workunit/quota} ----------------------------------------------------------------- time: 00:16:34 info: http://pulpito.ceph.com/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388207/ log: http://qa-proxy.ceph.com/teuthology/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388207/ sentry: https://sentry.ceph.com/organizations/ceph/?query=4fa3d84113b846a1b4a02e6d537f913e Command failed (workunit test fs/quota/quota.sh) on smithi052 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=9d3f068cec6c5fb39569ca38ddb0f19fce7352a2 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/quota/quota.sh' [7388209] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mon 2-workunit/fs/snaps}} ----------------------------------------------------------------- time: 00:55:39 info: http://pulpito.ceph.com/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388209/ log: http://qa-proxy.ceph.com/teuthology/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388209/ sentry: https://sentry.ceph.com/organizations/ceph/?query=ef235b5ff3274781b8c3bbc2fb2af647 Command failed (workunit test fs/snaps/snaptest-git-ceph.sh) on smithi110 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=9d3f068cec6c5fb39569ca38ddb0f19fce7352a2 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-git- ceph.sh' [7388212] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_ffsb traceless/50pc} ----------------------------------------------------------------- time: 01:15:04 info: http://pulpito.ceph.com/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388212/ log: http://qa-proxy.ceph.com/teuthology/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388212/ "2023-09-05T11:19:52.429907+0000 mon.a (mon.0) 134 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7388218] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-comp overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} ----------------------------------------------------------------- time: 00:13:48 info: http://pulpito.ceph.com/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388218/ log: http://qa-proxy.ceph.com/teuthology/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388218/ "2023-09-05T11:26:29.506515+0000 mon.a (mon.0) 220 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7388219] fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{ubuntu_latest} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/libcephfs_python} ----------------------------------------------------------------- time: 00:17:19 info: http://pulpito.ceph.com/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388219/ log: http://qa-proxy.ceph.com/teuthology/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388219/ sentry: https://sentry.ceph.com/organizations/ceph/?query=3e5bd0a794b2473f953b3bd3bd137e53 Command failed (workunit test fs/test_python.sh) on smithi131 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=9d3f068cec6c5fb39569ca38ddb0f19fce7352a2 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/test_python.sh' [7388227] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/5 tasks/fsstress validater/valgrind} ----------------------------------------------------------------- time: 06:45:06 info: http://pulpito.ceph.com/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388227/ log: http://qa-proxy.ceph.com/teuthology/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388227/ sentry: https://sentry.ceph.com/organizations/ceph/?query=d1667adf36e4405cb7309bae553fcc7e Command failed (workunit test suites/fsstress.sh) on smithi045 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=9d3f068cec6c5fb39569ca38ddb0f19fce7352a2 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsstress.sh' [7388228] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/auto-repair} ----------------------------------------------------------------- time: 00:11:19 info: http://pulpito.ceph.com/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388228/ log: http://qa-proxy.ceph.com/teuthology/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388228/ "2023-09-05T11:27:09.768538+0000 mon.a (mon.0) 132 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7388231] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/mdtest} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388231/ log: http://qa-proxy.ceph.com/teuthology/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388231/ sentry: https://sentry.ceph.com/organizations/ceph/?query=84914243dce642cba8140115d138588e Host key for server 'smithi089.front.sepia.ceph.com' does not match: got 'AAAAC3NzaC1lZDI1NTE5AAAAIDe0KZusXAy3hcyh7yAfd/ii8w41dA9OboRgiTOj+Zne', expected 'AAAAB3NzaC1yc2EAAAADAQABAAABgQDUSFZbey49O+ZQbtWmyvBx5qyxDUcP4O49P xovirJOlb1I3Qbbn3mGsdKu4pkEI7V9TJzGEQBaa6JwTxafr/iKYk2gO8+u9WPcHfIdDJc6zdSz 50Ik8KJ+bn+eLmHYQFSR/yP0i6fII4W4M4rTuiixLlTnDf8m5Ja9S2KmT3dJEqOkRPeyqN66X93 eOQAENLpcKcM9XqZ49Ehtwe6W/Vkt4ILGM/CPhgr2ogqP6Q4Od5xf59p5SC032MrCSySvirbycW cOp6PCcXnzoyy+vMgr4RwY/nA2iP/YXZGiCJghZNTWGfBOTL5BmmreqkuGsSFQ7TCvZqKd7+dqW AqIbfZP09ewNyXh0qHX65IoZJPombjQY8RtUM8BEymY4fkg2t1ryUKHLtZ1gjEaZrFbqIow6wyO bCJD4NF/EbXwWYVJhLCCoLeeh6m4Xrh7ETid8WsZtL+oS7mjX809QABl3XWlDKFXWQnvs1dbn5C zyI0jTb0sgZUYoisU0YoRZFPnvkM=' [7388232] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/failover} ----------------------------------------------------------------- time: 00:12:11 info: http://pulpito.ceph.com/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388232/ log: http://qa-proxy.ceph.com/teuthology/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388232/ sentry: https://sentry.ceph.com/organizations/ceph/?query=077aedd9d74b4ca28330b6e00bc47cd2 No module named 'tasks.ceph' [7388233] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps} ----------------------------------------------------------------- time: 00:11:32 info: http://pulpito.ceph.com/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388233/ log: http://qa-proxy.ceph.com/teuthology/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388233/ sentry: https://sentry.ceph.com/organizations/ceph/?query=0d2f5686ed6c4811a5f396872b91e852 No module named 'tasks.ceph' [7388235] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:13:07 info: http://pulpito.ceph.com/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388235/ log: http://qa-proxy.ceph.com/teuthology/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388235/ sentry: https://sentry.ceph.com/organizations/ceph/?query=af3ec257ec43444c8917f4ebc10c0343 Command failed on smithi158 with status 1: 'sudo yum -y install cephfs- java' [7388238] fs/fscrypt/{begin/{0-install 1-ceph 2-logrotate} bluestore-bitmap clusters/1-mds-1-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} overrides/{ignorelist_health ignorelist_health_more ignorelist_wrongly_marked_down pg-warn} tasks/{0-client 1-tests/fscrypt-pjd}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388238/ log: http://qa-proxy.ceph.com/teuthology/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388238/ sentry: https://sentry.ceph.com/organizations/ceph/?query=3c1e9ba94f844331b1de2b1260f6f1eb Host key for server 'smithi104.front.sepia.ceph.com' does not match: got 'AAAAC3NzaC1lZDI1NTE5AAAAIFg8qLZboxZbUQ7nI6Dda6dxG/+x1Wln7dnEOklGWNR0', expected 'AAAAB3NzaC1yc2EAAAADAQABAAABgQCr0BX7cpOmQMEEHvf3vS0Hc1eGERmaoShI5 dOsgJ3/kaJ0QfKRyYxj9v9NLqoX9cHNLdD4UGsQmuXQtx03Ad2+MHXoC4xmDCdp93qQmRNZukuT 9JsMTqPYTU5V42ZtZ6Rx3t38V0tmQdL5JonZv0xT9MtjQgBJ0ouN/VjMZcoCUZ6aPN0jMIXAVmU U7OgPATkO8NQwt0x7X9pJgq/9CMk/jHVKymzhJq18YU8vxQza1U/XtBxNNpVVOzoDp2CL8veT0c VCtSoDS2RMnAnoWlulW2DUZ9SpfdWzmUYot6IyaWdXbLmSy9aQi+hUh57CpXt8CK0xVB62324HH rOooT8sdh9CuCASDxRPIBkc5TWuXulvMaPdLz8QfyMCKJeAGTkvq1PffmPHmWR2KxZ0au9Sa6R4 e4ieDuZCaP0nT1bwC0IAJ2fS6AbjDb7h/23wlSClXem2C/Qfy5CfMp2EwDQZyGs/5zvW078C8+6 VQxc1ALTIP54gleiz1DHzLAYqIck=' [7388239] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/snapshot}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388239/ log: http://qa-proxy.ceph.com/teuthology/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388239/ sentry: https://sentry.ceph.com/organizations/ceph/?query=9cb91c38176e49c6a3651db5c92a4caf Host key for server 'smithi144.front.sepia.ceph.com' does not match: got 'AAAAC3NzaC1lZDI1NTE5AAAAIDd0X8b38UkcPXtcNaP3JUC6MzmqxZMJmXBEx07JjSXQ', expected 'AAAAB3NzaC1yc2EAAAADAQABAAABgQDNGTwTISv62m0Uc1OdWBG+FN+mPq6qwnbd2 GBUdBzCoqYoXvm6mPA5cYRkFdw+9M+VvBY+CLpvItdB3URTzZd64CWIP5sUDUexyMoLVz7htV6X c1RN4xkxxotH/wyEp/OiWRCRPXPzAZNiZRwGfnKJRbVyHHfhmLSfWyHMHhf+sDEMLTEFJaBX2CY Bow1qhhjWIOhJPmlt4SbRiAfEm6o1qTiodnGGNvlx/fhRQuE/y+1ExdQ//MBI3vj04EyPOCkv3m +beYEJjZIdOio67oySpbpbPF6flcyv1+52/V8saTwvx07jlYsJ/ReYgVJhB2lh0KgO+WnYt4po7 ZV+xJ8o+P+lbcu9F1luThk27JIDrqeCZbe91kYGrRi3/fkz/Lprv4Erwu/3BBg5wzcXTlytVa4S I6dNP3Xw1mz+8U2GQFtV5I8y/rDADVchGLSl7ZehDrc1NBqWatis3Q1yxpDWLvs37i2nqIaScfO AaLe54neAYi8gqIBRkW68mGev1I0=' [7388241] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-ec-root overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388241/ log: http://qa-proxy.ceph.com/teuthology/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388241/ sentry: https://sentry.ceph.com/organizations/ceph/?query=7ac766c4ed0648e0b73dc0fd763fbe42 Host key for server 'smithi118.front.sepia.ceph.com' does not match: got 'AAAAC3NzaC1lZDI1NTE5AAAAILtURNgi6yyNTUi4mh1PMkzVCq6Pw2DLM4dR7pqzMB1E', expected 'AAAAB3NzaC1yc2EAAAADAQABAAABgQDITUwYyDNVd9scXmBRkKQtjFQ522KWzrFoX ygmpc8CIIp4LTzOePO53J0hlkUtzZqt6DYfJPVeDU24pRKidEClzHuWdijGq6KkptRxRtVnbI8G GeO7dEIIfP0wsrt9DADMxZvfKlq0YFqp6J8dmNf4eHOsrP501Gl1BJqbrVMEpgl599bq2Ot28uE RsxM5ADLa2H2aAzgNNfwrFEYiVVSaaGJ2Z1ZBju1ot9C4QuoFK4QbgG1OkbGIJefRbjUDA/Hm9l seSeqx0BVY90StvqLtPlXsTsPUuw2dacikrB370y1IuFg/lCVBAR3ZCLic+vaA1ubOZotyQ0xWg Xl01bOLWrO/i5fECj9cMxRAZjFvjorF8Ik1ffxoWtdRovPnBwnK2pFKaJ9bmpWqy82vJem8WHoc cWYUkrqRWFqVwh1L2v9YC0rUnu4L/RhNNsEuSzUqf2C34wen1BUune/Szp7xxuUHxStw0SaEMxc ukcBKJ/Z/iPrIC4v89ePSzMts0K0=' [7388242] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_misc} ----------------------------------------------------------------- time: 00:14:10 info: http://pulpito.ceph.com/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388242/ log: http://qa-proxy.ceph.com/teuthology/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388242/ "2023-09-05T11:59:27.403975+0000 mon.a (mon.0) 210 : cluster [WRN] Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7388244] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mon 2-workunit/fs/trivial_sync}} ----------------------------------------------------------------- time: 00:08:10 info: http://pulpito.ceph.com/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388244/ log: http://qa-proxy.ceph.com/teuthology/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388244/ sentry: https://sentry.ceph.com/organizations/ceph/?query=a8e5e019a0764e85bc5fca15cea12cc6 Command failed on smithi157 with status 100: 'sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::=" --force-confdef" -o Dpkg::Options::="--force-confold" install ceph=18.0.0-5849-g9d3f068c-1focal ceph-mds=18.0.0-5849-g9d3f068c-1focal ceph-common=18.0.0-5849-g9d3f068c-1focal ceph- fuse=18.0.0-5849-g9d3f068c-1focal ceph-test=18.0.0-5849-g9d3f068c-1focal radosgw=18.0.0-5849-g9d3f068c-1focal python- ceph=18.0.0-5849-g9d3f068c-1focal libcephfs1=18.0.0-5849-g9d3f068c-1focal libcephfs-java=18.0.0-5849-g9d3f068c-1focal libcephfs- jni=18.0.0-5849-g9d3f068c-1focal librados2=18.0.0-5849-g9d3f068c-1focal librbd1=18.0.0-5849-g9d3f068c-1focal rbd-fuse=18.0.0-5849-g9d3f068c-1focal python3-cephfs=18.0.0-5849-g9d3f068c-1focal cephfs- shell=18.0.0-5849-g9d3f068c-1focal cephfs-top=18.0.0-5849-g9d3f068c-1focal cephfs-mirror=18.0.0-5849-g9d3f068c-1focal' [7388248] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} kclient-overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_dbench_iozone} ----------------------------------------------------------------- time: 00:11:28 info: http://pulpito.ceph.com/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388248/ log: http://qa-proxy.ceph.com/teuthology/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388248/ sentry: https://sentry.ceph.com/organizations/ceph/?query=e5e0db8bdfbd4c05b81640aa6475bf3d No module named 'tasks.ceph' [7388250] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/client-recovery} ----------------------------------------------------------------- time: 00:09:43 info: http://pulpito.ceph.com/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388250/ log: http://qa-proxy.ceph.com/teuthology/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388250/ sentry: https://sentry.ceph.com/organizations/ceph/?query=983e54f6bffb411ba17841f52e5e0f61 Command failed on smithi181 with status 1: 'sudo yum -y install ceph- radosgw ceph-test ceph ceph-fuse cephfs-java libcephfs_jni1 libcephfs1 librados2 librbd1 python-ceph rbd-fuse python3-cephfs cephfs-top cephfs- mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid- devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf- devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio- devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql- server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql- contrib' [7388251] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_trivial_sync}} ----------------------------------------------------------------- time: 00:11:31 info: http://pulpito.ceph.com/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388251/ log: http://qa-proxy.ceph.com/teuthology/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388251/ sentry: https://sentry.ceph.com/organizations/ceph/?query=6fb06c703bf54028800e2855a976a894 No module named 'tasks.kclient' [7388255] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mds 2-workunit/suites/pjd}} ----------------------------------------------------------------- time: 00:11:40 info: http://pulpito.ceph.com/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388255/ log: http://qa-proxy.ceph.com/teuthology/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388255/ sentry: https://sentry.ceph.com/organizations/ceph/?query=587539490b7b4e0fa1df609242203a25 No module named 'tasks.ceph' [7388258] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/exports} ----------------------------------------------------------------- time: 00:09:46 info: http://pulpito.ceph.com/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388258/ log: http://qa-proxy.ceph.com/teuthology/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388258/ sentry: https://sentry.ceph.com/organizations/ceph/?query=537a7c9b94614fb6a93c79a6eefd3715 Command failed on smithi037 with status 1: 'sudo yum -y install ceph- radosgw ceph-test ceph ceph-fuse cephfs-java libcephfs_jni1 libcephfs1 librados2 librbd1 python-ceph rbd-fuse python3-cephfs cephfs-top cephfs- mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid- devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf- devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio- devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql- server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql- contrib' [7388260] fs/bugs/client_trim_caps/{begin/{0-install 1-ceph 2-logrotate} centos_latest clusters/small-cluster conf/{client mds mon osd} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/trim-i24137} ----------------------------------------------------------------- time: 00:05:35 info: http://pulpito.ceph.com/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388260/ log: http://qa-proxy.ceph.com/teuthology/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388260/ sentry: https://sentry.ceph.com/organizations/ceph/?query=736251a20b01453ea06bd830bfec6233 No module named 'tasks.ceph' [7388265] fs/mirror-ha/{begin/{0-install 1-ceph 2-logrotate} cephfs-mirror/{1-volume-create-rm 2-three-per-cluster} clients/{mirror} cluster/{1-node} objectstore/bluestore-bitmap overrides/{whitelist_health} supported-random-distro$/{centos_latest} workloads/cephfs-mirror-ha-workunit} ----------------------------------------------------------------- time: 00:04:12 info: http://pulpito.ceph.com/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388265/ log: http://qa-proxy.ceph.com/teuthology/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388265/ sentry: https://sentry.ceph.com/organizations/ceph/?query=8c8e1b7928fd4d4b8df1a5506961645d Command failed on smithi173 with status 1: 'sudo yum -y install ceph- radosgw ceph-test ceph ceph-fuse cephfs-java libcephfs_jni1 libcephfs1 librados2 librbd1 python-ceph rbd-fuse python3-cephfs cephfs-top cephfs- mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid- devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib' [7388266] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cephfs_misc_tests} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388266/ log: http://qa-proxy.ceph.com/teuthology/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388266/ sentry: https://sentry.ceph.com/organizations/ceph/?query=b5a188bc0e2f4c24afc3e15bfcb7f823 Host key for server 'smithi104.front.sepia.ceph.com' does not match: got 'AAAAC3NzaC1lZDI1NTE5AAAAIOB/s24jsVWbjV1fcyP5vkvF1H3Kkfv7S76W3oPYqhOZ', expected 'AAAAB3NzaC1yc2EAAAADAQABAAABgQC8jmGZWHMSkjYutKUS6w9V7wIvw7vsI9jYo z8rdgYIwC/TximP5GRDQCoWGCvjfOtjrfFSWW2rdMFFBw7pwDuGOeblVmKVGm/UJnEqZyYyvKMW CGyv1MBaSKMzU/yt4xIca01RC+ocPL6lxH4U2FEI+TrirlSG2ezitQVUkR90J1Xg6hwDB9Dm7JA Trj5caheizMhuFr8y8+q3IrDj16VWhLISruWI6JSMKaGGvc6vrGiBmKTsVRD8oEN0HRQPlcq+r9 2Y40IVlt9E527BhDDeMtZ52i1rPErY7wDg3sS+iY21KjhzFMy0mALSWzdVbe6mOEBJtqvdKfWdP HqVXFtKV3akOr60uW7b/mkabeTOfvIN3xs97+Fqq4pUNkMOT13JKIfaoUQfj+7Jh68bPkaF43Nw JetPo7zr4xXZCq5EgBbBuQIwAkjhWTK6sFrySE+NM6OB4tiqA1+GyjLbvmPbf8ZEtTFyHuJkP7M o3PWykJ6F3D5MOsPiNSRfprxRRjU=' [7388268] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_misc} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388268/ log: http://qa-proxy.ceph.com/teuthology/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388268/ sentry: https://sentry.ceph.com/organizations/ceph/?query=2dd82c37e69c459194856324c8089b6b Host key for server 'smithi007.front.sepia.ceph.com' does not match: got 'AAAAC3NzaC1lZDI1NTE5AAAAIAZ8bn1DHrDtuPYFrBpcjdcAY8srHHwKz8gBmEbonWpw', expected 'AAAAB3NzaC1yc2EAAAADAQABAAABgQC35y/lnvAX8KextUvgRYkHZQOP+ugDe1xsm c6eFaoez2U2UyPVFUbFSBefae+B0pBWr3DXDy9Lk3BioZUtNPsxaQQJ5bayl0sRwfKTk3VzzF2q NTG/TGVOapxZBb0wIJT7epy/KnkBTVDFhuCzt+LA4LjhAFfJ+8IH2KlJNELN2PJTUxx5gYnyJe5 ymYqsgzQ67f4yi3synwaqMbuC+07/laD1UHxCw7SUAKHDUgD0Xbdgr5mzIeqRWo1BVx0xG8InR0 KDWYv6CfoeXG5eCAjLzNGRjj9davG6kT+WyTPiUs6/vc/ntGy6m19F6mp5U6e+H/NolwAB6SKwP c1PFaszPunDNqsNWX0nhpyw0STXbWKEo9+EpkLttzHoccvKjWb1XM/DDEsVa5o1VLd1T6BTUogn XdX1jhbZOK/xA5HMjjRqTJjDA90hUk0gSMUqnygPUWVZBlBu3eMHOk+f+xsyl85ihwLLaE8jRq5 gnUE9JSJt0sXBKOYx4+stQE3LpvE=' [7388273] fs/valgrind/{begin/{0-install 1-ceph 2-logrotate} centos_latest debug mirror/{cephfs-mirror/one-per-cluster clients/mirror cluster/1-node mount/fuse overrides/whitelist_health tasks/mirror}} ----------------------------------------------------------------- time: 02:22:42 info: http://pulpito.ceph.com/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388273/ log: http://qa-proxy.ceph.com/teuthology/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388273/ "2023-09-05T12:24:28.459341+0000 mon.a (mon.0) 107 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7388275] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/basic}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388275/ log: http://qa-proxy.ceph.com/teuthology/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388275/ sentry: https://sentry.ceph.com/organizations/ceph/?query=beb26993a876457db01ebb4b4a10bb10 Host key for server 'smithi148.front.sepia.ceph.com' does not match: got 'AAAAC3NzaC1lZDI1NTE5AAAAIJc7byGD1pFofuIKgJk7qTBZPWs0c19aJbLO4zQeUYIn', expected 'AAAAB3NzaC1yc2EAAAADAQABAAABgQDH9teGDjQfIJg6eYYqkcRP/VRrJdqv7Ju2y WBhO1BVnrx5f0zaSy0dqqVZBrW0Gqf7GtKZoZFjdD4i3k5BlJV9DokGEn7j06ZAvDaMTQn6S9ml 7jOiLhyDBeugzpeAgzFkoo9t/auF06c9ScRMzD9cbocH7F00fi0xcNPwCeJyB0l7Szmh4OKcC6U pNd/w4zk4t56FIO3lbHE4cusW4QBceHNWOrf+WBroXBHkUXixaUkToEfhxwVi5JD5eh5sOzQhqN aTDP6iwHrFxD0/N6zA8b6L26VONQDwymX1aeSlIuF8vAlB8NpD4830ri1SRxNsILYy0hmiA+fAM jpCXnP3L72qk2RS5og+BL94k1MZbm7KUfuLthehSFMIITVmt6qx2ZrxaaUgBK8V8s7yFbExMVqG 851LlM4HJ1QxFkr6ArRJkBdRSeunfeDdMw/eUwklh065poCIGWoSR7CJ1sO8hOTuGC0p+WdJ72S NxVuVSN25h8N0maBfTImZxDzYsCk=' [7388277] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mds 2-workunit/fs/snaps}} ----------------------------------------------------------------- time: 00:36:17 info: http://pulpito.ceph.com/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388277/ log: http://qa-proxy.ceph.com/teuthology/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388277/ sentry: https://sentry.ceph.com/organizations/ceph/?query=e1e66c53af0f4d75a6c457adcf4a2664 Command failed (workunit test fs/snaps/snaptest-git-ceph.sh) on smithi131 with status 128: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=9d3f068cec6c5fb39569ca38ddb0f19fce7352a2 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-git- ceph.sh' [7388278] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/forward-scrub} ----------------------------------------------------------------- time: 00:13:00 info: http://pulpito.ceph.com/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388278/ log: http://qa-proxy.ceph.com/teuthology/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388278/ "2023-09-05T12:21:07.839623+0000 mon.a (mon.0) 222 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7388279] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:17:13 info: http://pulpito.ceph.com/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388279/ log: http://qa-proxy.ceph.com/teuthology/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388279/ sentry: https://sentry.ceph.com/organizations/ceph/?query=2a48b0a5ac6045be90bd8defd47fe0bc Command failed on smithi059 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 196fe4f0-4be8-11ee-9ab3-7b867c8bd7da -- ceph orch host ls --format=json' [7388280] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/fragment} ----------------------------------------------------------------- time: 00:19:13 info: http://pulpito.ceph.com/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388280/ log: http://qa-proxy.ceph.com/teuthology/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388280/ "2023-09-05T12:21:30.856960+0000 mon.a (mon.0) 137 : cluster [WRN] Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7388287] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/mds-flush} ----------------------------------------------------------------- time: 00:09:43 info: http://pulpito.ceph.com/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388287/ log: http://qa-proxy.ceph.com/teuthology/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388287/ "2023-09-05T12:23:04.290034+0000 mon.a (mon.0) 234 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7388289] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_blogbench traceless/50pc} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388289/ log: http://qa-proxy.ceph.com/teuthology/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388289/ sentry: https://sentry.ceph.com/organizations/ceph/?query=2fa8b91b850b4df68972f4240b9210a9 Host key for server 'smithi087.front.sepia.ceph.com' does not match: got 'AAAAC3NzaC1lZDI1NTE5AAAAICCBThPdJT1HmLyckij2Eg+t8O/eoEBUntaTwXWKXTUb', expected 'AAAAB3NzaC1yc2EAAAADAQABAAABgQDASRSv0f41Fs3FnKFvGxS+ijKQ65yzHBKRJ 2q4JmbG0HHLUgfGyxhYcFI8p0AFkjkb7LOaVnxDYJR/McIHCzXX/NUfgSuqzStmy3wLd1rVEBSj /XvgJi6RZVyqyKT/rczG1BkjgOdKY54A2dH/hRkIXRiVUGfqv7WoT1HcSx9xgem94DJ9PfqPLPL CugrQ3iR1HsTzro3HJzmM7S5adeP+NCYUUv1Mx9cVZPj7uSBtSlpPInnh7NpWtaBtd1KGvgofLS wucrBcFLTyZhyqKE3cqUFCHRjY3VHzRdGMXjgCn2N+Awvd4llibeSjeyNe0x0IvCxl1V7nUALec R+W2hkE/mcsvucQgJxoH8GPFeAIP6tY0y7+YVtLIJCvK13K2TfBp09rUI+kQP0zXjTbjrgI41ek R66WIAFdJrH9mB6dJ8SCXPP0S4nbPw0NC+RJw7d8qFlovzkZ/gQ0AyTkTLZcQrYN2n3q3VCE0ho dBMKggU4cOsij8IdwEAjyjBwGiws=' [7388295] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} ----------------------------------------------------------------- time: 00:13:27 info: http://pulpito.ceph.com/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388295/ log: http://qa-proxy.ceph.com/teuthology/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388295/ "2023-09-05T12:27:10.402937+0000 mon.a (mon.0) 214 : cluster [WRN] Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7388296] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388296/ log: http://qa-proxy.ceph.com/teuthology/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388296/ sentry: https://sentry.ceph.com/organizations/ceph/?query=d0e1bacaf43c4775bbd3b416cba1a42d Host key for server 'smithi052.front.sepia.ceph.com' does not match: got 'AAAAC3NzaC1lZDI1NTE5AAAAIFwMNJYOGZEXS/mam0yk8zWBpnV2/ePuJIGA5QQ7OOu1', expected 'AAAAB3NzaC1yc2EAAAADAQABAAABgQCu1F9cOuc1JdwZemXI6x+vjQsoxo5qmi+dy XHIOz4Zb7hFdVuApJfkg8FNE0HDJDT+IRf9fdDDJSOqapwrgKmh9HstHWeOEMsTBWxs6F47TptC jUBLXmRe+qUj6M/Ae19hd8NB1+V1dKlskviY48vQC1Vu5s2qfqctM1E3uCGvwI5D37G5G+tDo55 tKLJVmG/5umc/dfK7VLagtsslnurUeT5iISIhKP6BiDmfDySB6PIkaDmkAwSpujTHiJqBG9IC0T qDhOvtlXNuzcLW68gWG9ef68j/NOQPp9I69i4jet0wOPBlKrBDpAOPQKG7YXtUjv0vhaiJAIGfp SRF6W1f9JDrR0yZwtUJ+Zqgw+iEoTqUcwfMwOBx8naw9kwIm8uXXXk5VUgPKjdrVmRyF9t2Aoxu BxzY9HV3VFXLzydDPQUifdY4hF/gjL056bGQvGjPqVtOpRFLRpH8iA6ROYNwkeW0/ikMtigxdnR 6Q/s9Fz2j1Fykf0hja/pt/K6DeE8=' [7388301] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/metrics} ----------------------------------------------------------------- time: 00:29:45 info: http://pulpito.ceph.com/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388301/ log: http://qa-proxy.ceph.com/teuthology/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388301/ "2023-09-05T12:32:08.652566+0000 mon.a (mon.0) 572 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7388303] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/ior-shared-file} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388303/ log: http://qa-proxy.ceph.com/teuthology/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388303/ sentry: https://sentry.ceph.com/organizations/ceph/?query=e1fb4ef1ca4646f5b0e91b50c9997f32 Host key for server 'smithi070.front.sepia.ceph.com' does not match: got 'AAAAC3NzaC1lZDI1NTE5AAAAIBtMSN1hD6JjOPqkidrP6QuhdjhpZuL0mUqQPWYcMSgE', expected 'AAAAB3NzaC1yc2EAAAADAQABAAABgQCQZj1In+CZD14/mPNuuKYl2Mom6qcGk+l7z co5s0D80pIsjm5RVslNTIBCmINeR8cSo5LETiSjr8UL2JkxhyU6KsW5oYw2Ltf3JUm9/Pu6Ie/h bDYtyLXMipwgckzPbE9dbszQh/qUTu+ZmG/HSuRW8pj5DNuti+D2QanoU6nxund9m3i7XnKxC3w x2ch1kVjukuvRbYnGU82yVvgL1XqJxgDr6Nw501XIxQ3lLAU5buAW9BqWshw5y2Z7pgHEKCF+rK SayY74Qweg7smolveH/a8MVOwH6BAwBfBwvo5Rbxl1+892Q85Vf8KVIm+j7XmAKQU9O1ABg1HcW nkK/DBHoQXjYDRhprAtiEMdMi+7vG4dGi4xPgtkT8/vPImX6svNmRCcmiJ0pwAAxbhti8JiaGcA 03W9WFExf+qyHNFoy8WzrGIiRXgLhNgn2DGCeXOXYTdMWjGC7E63JJnIa2N6BZPk2hhNInFGdVi bOdjmmw8UncFreo67MHWuSUvQMJM=' [7388310] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu/{latest overrides}} mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/3 tasks/fsstress validater/valgrind} ----------------------------------------------------------------- time: 00:31:35 info: http://pulpito.ceph.com/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388310/ log: http://qa-proxy.ceph.com/teuthology/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388310/ sentry: https://sentry.ceph.com/organizations/ceph/?query=38f970c45b304c0dac9bf1917459b472 saw valgrind issues [7388312] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/iozone}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388312/ log: http://qa-proxy.ceph.com/teuthology/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388312/ sentry: https://sentry.ceph.com/organizations/ceph/?query=c134c4e5dc3b4f80bf04aae3131a0f25 Host key for server 'smithi042.front.sepia.ceph.com' does not match: got 'AAAAC3NzaC1lZDI1NTE5AAAAIO5pg356dEuedI0vJKiCLAAQYSIzibQek8cpeqWboA09', expected 'AAAAB3NzaC1yc2EAAAADAQABAAABgQC3TpfIWWqYwhzBOCZBwGH0vp3i/VxqS4uei Ism/xVpASL8aRjT5eR6TyLNHSttzwgyYCxyTxDRZEhzu2KDP7YaaRzUJApCab4/2ALaVwYVdTSn /o+GD3LmcP0IG7MQfbL1KbvksNuRy8leuhuKnmMaWQqJcSvaRAvLBhr3bTl0pWo13qbH8yJ2oDe fdL7Yp9b0Vka3RaODKU3lgXi4DQ4GqpGI7N05dzedeFkjR0ykRu1euhM7z3sizu4dzpemFIpjDZ gE8G6/hlSQ4E54/LPGFhHIDsb5KpHCBtdOIOd7t0KEGYb7ae/UgjE0dfdTqZ9loBbL/Wq0IwjJd 1u3PcSzuZCS+7LmzYbd+ZHdaIuRlpoiAYeejI2IkVH6zuK2buyaY2pHReU3CNfziruAexXe+1ik gIW5tbMvB+ju67AP4xz/6s/v9Pbl9STXrAhhwmywUkdua1HocKEd56pTnFaS2R2xnrB7OqmEySi MRqHXRAI9w7IXQpz/ry1FGDeIQW0=' [7388314] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/pool-perm} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388314/ log: http://qa-proxy.ceph.com/teuthology/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388314/ sentry: https://sentry.ceph.com/organizations/ceph/?query=189b89d91b704b6c82de5630b3638c43 Host key for server 'smithi017.front.sepia.ceph.com' does not match: got 'AAAAC3NzaC1lZDI1NTE5AAAAIJwDHjpXiLzq7xnpjgxT7R4ubS9U+rNWfzZ/U/V6NO/G', expected 'AAAAB3NzaC1yc2EAAAADAQABAAABgQCO9qHbIxuv1YIKO+ICusv7rdu4Z1DfTIRSo sc1NIh6aiXrPuq9xObx+6s4BHME3P3DE9pLW5EUJ8SIXzAhJSqJOTa84N8A9XHJF+29x8tgsDfU KCz5pCuoKXxkxSt16qYt1BOrLu2LSvpdVcAxu20/jhK8sMJSA7hLIRn0ZZtZyBAhRC8pU+YhkTr oCVd3gMysPeqOSq7+JFSGMZ4t17LDrt49QvwpwZ7e9tmYSrXQuv8IdY//52KlCOmowzSXZf1/fW smtVq7pTIJkmW/sZDICu33Fyvv/n23AtzhF2lYYvkz/4EKnhgJCmfGr9P3G762Zi83czcCpUG7O vkVd7pOxTzWknM17rAI5KMUfkkTUcad3b1/QGO6ibPNf7wMkscxt3PkUs6Zm53gqCkAYZltLBtD hsTPlp737wXLIPSmFH4I/xpZv2dzhQjIj2IhHxtWlpD3Dsb68da8KSCEX5BNHkX07VmvOXxEXA4 6chpLu8384zMhv2E5w8jzQQejFKU=' [7388316] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/quota} ----------------------------------------------------------------- time: 00:16:36 info: http://pulpito.ceph.com/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388316/ log: http://qa-proxy.ceph.com/teuthology/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388316/ sentry: https://sentry.ceph.com/organizations/ceph/?query=77c111ab051f49c585ed9b30c497190d Test failure: test_disable_enable_human_readable_quota_values (tasks.cephfs.test_quota.TestQuota) [7388317] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-comp overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress} ----------------------------------------------------------------- time: 00:16:13 info: http://pulpito.ceph.com/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388317/ log: http://qa-proxy.ceph.com/teuthology/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388317/ "2023-09-05T14:28:18.472332+0000 mon.a (mon.0) 133 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7388321] fs/fscrypt/{begin/{0-install 1-ceph 2-logrotate} bluestore-bitmap clusters/1-mds-1-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} overrides/{ignorelist_health ignorelist_health_more ignorelist_wrongly_marked_down pg-warn} tasks/{0-client 1-tests/fscrypt-ffsb}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388321/ log: http://qa-proxy.ceph.com/teuthology/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388321/ sentry: https://sentry.ceph.com/organizations/ceph/?query=4c47807e5aa64bbf9f7fb1d0e7869123 Host key for server 'smithi057.front.sepia.ceph.com' does not match: got 'AAAAC3NzaC1lZDI1NTE5AAAAICYlaLilwrGYOFacZgvfJMNiJG8T/bX383rWnoKGYVN7', expected 'AAAAB3NzaC1yc2EAAAADAQABAAABgQD0nd07+IelHEWXd3xOw/He2SPqgKDycxSwG vXGRxZg2Beyjj9xv0feO+qCBdI2RbI7bqjbLUCt2Vb4bDUlZYUoVUB42HlnnHyaqUP19/HpZJiy AGAGs6fpCcJingSqe7bxXfU4gVdJKUuwZfQGTwbY80+CV46jf5EkIl5a9a4wpkuazDTmJgsxugo CBUtxyMG8KaaMlq6w22oBipwcvT0Xz7+PAi04M789o54Fob0bwZVUom68MAqFb4TULqYbv8rjGg MNznXdfmhAzWZzUkCi7bJ2QBFJVQEzSJ9UwrdX632XpjbA2ErYIaYYtbTV1rsxiNWQY0DvDj7B7 9URhRaR+/HsApLKleLwaEm3gfBGXXuQS7Ull2mMfbRignariCqn9T1g9zu77Qv65QZSTfoRydqU JreImUthoxMvMgHON7yVSoqG5nbKt87EqI0BKkAZrn3oK4vBO9U0ADZoS19wrEfQwhkUpZDO14T I8k4VSW4zceYAD/W1hSzUhaMERVE=' [7388325] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu/{latest overrides}} kclient-overrides/{distro/testing/k-testing ms-die-on-skipped} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_untarbuild_blogbench} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388325/ log: http://qa-proxy.ceph.com/teuthology/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388325/ sentry: https://sentry.ceph.com/organizations/ceph/?query=4949c66791ef4db5a15aa4612f230cc8 Host key for server 'smithi107.front.sepia.ceph.com' does not match: got 'AAAAC3NzaC1lZDI1NTE5AAAAIHu0jKzltV4lyU8/IqLVQHdQQi7+e72cMVhmWrLR+NFi', expected 'AAAAB3NzaC1yc2EAAAADAQABAAABgQCUq0SfhkeoOQSbWAIJSVnxnSYMpnzbDt0wW iUx+glK/E+4q64qnLSM9sRV347iQmsY4AHvL9MOrTBIGBLEm6xHVHRNNZi/1hsn5gRv/sZy1KAQ 6A9Idv7y3mvvYe3HCzz+MwQ5syWrQ98HeYG3Ukqo5xmtFWvXNmHf8nAySwJmwQYs/WJapNuWNUZ CQTLbUtGMrP7UL6TBLmNlt+UOaQClv3XhWWMvgEa2rQxo7dRlOWFOSb/VdVZI+CcjvlgMVfHB6f plkU6yjlb28EfW27iRJr4iZqgz0iHHuN6p5zNMdq7Kf8VBrTUehgXivzwVh+j3abXXFPzhEGj7e 212NTPKkGXoiBDQMBM80ggjfWfHiqYe//Og/TbgSHyPFw6nBt5F47NyAvcOyT7hoagIoce9HNpi iAU4PzwVUBAaJfxUi5qHFa/Zq189gtH3FxV9XSdBjNfM/iAv/QHQeKBGS3aGIEQCZmxuG14sobm kEv2+jAz3k3XJoZswOmnL41NB7gE=' [7388335] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/mdtest} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388335/ log: http://qa-proxy.ceph.com/teuthology/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388335/ sentry: https://sentry.ceph.com/organizations/ceph/?query=d28cefd93a584f31bbdb9a7bb6368222 Host key for server 'smithi026.front.sepia.ceph.com' does not match: got 'AAAAC3NzaC1lZDI1NTE5AAAAIPZp6wOILlcOVYZjlBmIruM6AFvsaBMKMhL+T3kylK03', expected 'AAAAB3NzaC1yc2EAAAADAQABAAABgQCTqi4QsBDDKbSS87JrAL5PCT9gK8O7STWIl hRnX+v5NqDPeErkcCeFQkNiLn0VxjdfUXHvIaOp2gpcKhd3Vjcb+9sHPtiNoaMWWApISdpMTZR/ dUsy3/RYFkn/S2rZOz9PqNxkI7VUTu2jZuuTnv01a49FFTvaxYQrLU1VufWHxD4D8lpqhVi7sV3 nWJCjpiM6cz1XM1aqUuXEiyaRR5sItrLWA4NbB5sRZcHaL775vrHTIoCj52oyliTdXcRp4TXjtP cOh+kK9UhTJKRpqmSVgh0Iw+SvM3hlNmzzaBPBnKxSOxbZda5SvZuDsi+p/dKa1Igeq9N2Z2EVf zT+eDUsDtXpczv6rdEWpUM2nl+J3M5kTDg/sLfQJJGS3YLbudCjof/FoiuTmSVW0X2tf+d+jYur b7p1STyNJDSfbQoaU1Qr7mka9PjWmh0hHVtGuoTyPwAbdKoE4wUjUDYMyioTjB2Yr529Lro2UTN 7d32sCe7BQeCWBE8SQQmKsc6cDd8=' [7388337] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps} ----------------------------------------------------------------- time: 00:38:57 info: http://pulpito.ceph.com/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388337/ log: http://qa-proxy.ceph.com/teuthology/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388337/ "2023-09-05T15:00:04.411876+0000 mon.a (mon.0) 129 : cluster [WRN] Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7388341] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snap_schedule_snapdir} ----------------------------------------------------------------- time: 00:15:53 info: http://pulpito.ceph.com/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388341/ log: http://qa-proxy.ceph.com/teuthology/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388341/ "2023-09-05T15:11:35.565416+0000 mon.a (mon.0) 136 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7388344] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-ec-root overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} ----------------------------------------------------------------- time: 00:13:27 info: http://pulpito.ceph.com/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388344/ log: http://qa-proxy.ceph.com/teuthology/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388344/ "2023-09-05T15:03:47.473632+0000 mon.a (mon.0) 133 : cluster [WRN] Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7388345] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} ----------------------------------------------------------------- time: 00:14:04 info: http://pulpito.ceph.com/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388345/ log: http://qa-proxy.ceph.com/teuthology/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388345/ "2023-09-05T15:05:40.225734+0000 mon.a (mon.0) 216 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7388349] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/misc}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388349/ log: http://qa-proxy.ceph.com/teuthology/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388349/ sentry: https://sentry.ceph.com/organizations/ceph/?query=4fa286f0580746e7ba25842f689dcf4e Host key for server 'smithi096.front.sepia.ceph.com' does not match: got 'AAAAC3NzaC1lZDI1NTE5AAAAIICRbFLhMpLRnbUIqMIIWJvBRq0sjabYAHHeuAzXMuvY', expected 'AAAAB3NzaC1yc2EAAAADAQABAAABgQCixh09rEp5J3M1y+PImPscaCc7fg6nyYz4P meG1TtdvVLzxhOqg0mSMc5nSi/pQJw+KQhe+SRSc/+a0Xeo8x6gzTkcpEXgvjSNvtCUXPpWHjf5 lOwD32l3nmmBtsUwn7ANaSEaI/EGDgL4whTeST/N9T4yTyslKgLxlaT5B1/RuYp69cpn7uJZmCk hHewILWVZM6j0QUtv6LCfZTTynEdkbdQwjTnnblJluwKKNjswHycd2W72ecOIKTxrjQH6VonT9z 6gYw7n4IsaHcPYo6is6T7HC3QXLAaULqXvGYSNQ2T2HxJO8vNu5Y4njEx4SwLuze/EE1fqVOwkP IwVs9OUoIgymVdxiu4Uo+tZU4HmMKaXixtGYEwtHOzPbZXwqxAc0dInfPAyCp38d413nqQyd0jL lNN3k9y12/wQK06IjZXiN2WdvDRfS/6zz3WJ6qwV3ficR5K4j/xdD0qbNcON+jIHkdlt8y+H0v9 R/0JC9Yu1F4NG92U5snexnBIhb3c=' Dead ================================================================= [7388201] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/multifs-auth} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388201/ log: http://qa-proxy.ceph.com/teuthology/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388201/ hit max job timeout [7388203] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/xfstests-dev} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388203/ log: http://qa-proxy.ceph.com/teuthology/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388203/ hit max job timeout [7388304] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/multifs-auth} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388304/ log: http://qa-proxy.ceph.com/teuthology/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388304/ hit max job timeout [7388309] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_8 clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/quincy}} ----------------------------------------------------------------- time: 00:09:29 info: http://pulpito.ceph.com/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388309/ log: http://qa-proxy.ceph.com/teuthology/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388309/ sentry: https://sentry.ceph.com/organizations/ceph/?query=63a7a881cd924f60b4f6ae8804d4e841 [',', 'Bugzilla. In case of problems, contact the owner of this repository.', 'Enabling a Copr repository. Please note that this repository is not part', 'Error: Failed to connect to https://copr.fedorainfracloud.or g/coprs/ceph/python3-asyncssh/repo/epel-8/dnf.repo?arch=x86_64: Network is unreachable', 'Please do not file bug reports about these packages in Fedora', 'The Fedora Project does not exercise any power over the contents of', 'and packages are not held to any quality or security level.', 'of the main distribution, and quality may vary.', 'this repository beyond the rules outlined in the Copr FAQ at'] [7388313] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/suites/blogbench}} ----------------------------------------------------------------- time: 00:10:53 info: http://pulpito.ceph.com/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388313/ log: http://qa-proxy.ceph.com/teuthology/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388313/ sentry: https://sentry.ceph.com/organizations/ceph/?query=bbcd22822a304176beb6cc8cb01f36ba [',', 'Bugzilla. In case of problems, contact the owner of this repository.', "Curl error (7): Couldn't connect to server for https://copr. fedorainfracloud.org/coprs/ceph/python3-asyncssh/repo/epel-8/dnf.repo?arch= x86_64 [Failed to connect to copr.fedorainfracloud.org port 443: Network is unreachable]", 'Enabling a Copr repository. Please note that this repository is not part', 'Please do not file bug reports about these packages in Fedora', 'The Fedora Project does not exercise any power over the contents of', 'and packages are not held to any quality or security level.', 'of the main distribution, and quality may vary.', 'this repository beyond the rules outlined in the Copr FAQ at'] [7388353] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/test_journal_migration} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388353/ log: http://qa-proxy.ceph.com/teuthology/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388353/ Error reimaging machines: reached maximum tries (101) after waiting for 600 seconds Running ================================================================= [7388323] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/suites/dbench}} info: http://pulpito.ceph.com/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388323/ Pass ================================================================= [7388163] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} time: 00:37:02 info: http://pulpito.ceph.com/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388163/ [7388164] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/sessionmap} time: 00:17:52 info: http://pulpito.ceph.com/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388164/ [7388165] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_suites_pjd}} time: 00:23:44 info: http://pulpito.ceph.com/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388165/ [7388166] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} time: 00:20:21 info: http://pulpito.ceph.com/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388166/ [7388167] fs/cephadm/renamevolume/{0-start 1-rename distro/single-container-host overrides/ignorelist_health} time: 00:25:04 info: http://pulpito.ceph.com/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388167/ [7388170] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/failover} time: 01:23:16 info: http://pulpito.ceph.com/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388170/ [7388171] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} time: 00:24:17 info: http://pulpito.ceph.com/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388171/ [7388172] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps} time: 01:16:16 info: http://pulpito.ceph.com/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388172/ [7388174] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mon 2-workunit/suites/fsstress}} time: 00:21:48 info: http://pulpito.ceph.com/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388174/ [7388175] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_8 clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}} time: 00:32:14 info: http://pulpito.ceph.com/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388175/ [7388178] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} kclient-overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_dbench_iozone} time: 00:47:26 info: http://pulpito.ceph.com/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388178/ [7388180] fs/upgrade/nofs/{bluestore-bitmap centos_8 conf/{client mds mon osd} no-mds-cluster overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-pacific 1-upgrade}} time: 00:17:18 info: http://pulpito.ceph.com/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388180/ [7388182] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snapshots} time: 00:29:25 info: http://pulpito.ceph.com/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388182/ [7388183] fs/upgrade/upgraded_client/{bluestore-bitmap centos_8 clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-from/nautilus 1-mount/mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} 2-workload/new_ops/{0-clients/kclient 1-client-sanity}}} time: 00:15:26 info: http://pulpito.ceph.com/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388183/ [7388184] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/osd 2-workunit/fs/trivial_sync}} time: 00:20:04 info: http://pulpito.ceph.com/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388184/ [7388188] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/finisher_per_module}} time: 00:16:59 info: http://pulpito.ceph.com/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388188/ [7388190] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu/{latest overrides}} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/3 tasks/dbench validater/lockdep} time: 00:59:40 info: http://pulpito.ceph.com/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388190/ [7388191] fs/upgrade/upgraded_client/{bluestore-bitmap centos_8 clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-from/pacific 1-mount/mount/fuse 2-workload/stress_tests/{0-client-upgrade 1-tests/iozone}}} time: 00:22:47 info: http://pulpito.ceph.com/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388191/ [7388192] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/truncate_delay} time: 00:16:23 info: http://pulpito.ceph.com/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388192/ [7388193] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress} time: 00:22:03 info: http://pulpito.ceph.com/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388193/ [7388194] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_misc} time: 00:21:01 info: http://pulpito.ceph.com/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388194/ [7388195] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/fsstress}} time: 00:31:03 info: http://pulpito.ceph.com/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388195/ [7388196] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mds 2-workunit/suites/iozone}} time: 00:24:15 info: http://pulpito.ceph.com/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388196/ [7388197] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_8 clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} time: 00:27:03 info: http://pulpito.ceph.com/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388197/ [7388198] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/workunit/dir-max-entries} time: 00:19:06 info: http://pulpito.ceph.com/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388198/ [7388199] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mon 2-workunit/suites/pjd}} time: 00:21:27 info: http://pulpito.ceph.com/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388199/ [7388202] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps} time: 00:38:49 info: http://pulpito.ceph.com/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388202/ [7388205] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/suites/fsx}} time: 00:44:52 info: http://pulpito.ceph.com/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388205/ [7388206] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/iozone}} time: 00:20:41 info: http://pulpito.ceph.com/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388206/ [7388208] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_8 clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/quincy}} time: 00:26:17 info: http://pulpito.ceph.com/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388208/ [7388210] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/acls} time: 00:17:36 info: http://pulpito.ceph.com/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388210/ [7388211] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} kclient-overrides/{distro/testing/k-testing ms-die-on-skipped} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_untarbuild_blogbench} time: 00:35:46 info: http://pulpito.ceph.com/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388211/ [7388213] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/misc}} time: 00:23:13 info: http://pulpito.ceph.com/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388213/ [7388214] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/fsync-tester}} time: 00:38:15 info: http://pulpito.ceph.com/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388214/ [7388215] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/admin} time: 01:30:34 info: http://pulpito.ceph.com/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388215/ [7388216] fs/upgrade/upgraded_client/{bluestore-bitmap centos_8 clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-from/nautilus 1-mount/mount/fuse 2-workload/stress_tests/{0-client-upgrade 1-tests/blogbench}}} time: 00:29:11 info: http://pulpito.ceph.com/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388216/ [7388217] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_snaptests}} time: 01:11:40 info: http://pulpito.ceph.com/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388217/ [7388220] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} time: 00:22:31 info: http://pulpito.ceph.com/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388220/ [7388221] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/alternate-pool} time: 00:23:07 info: http://pulpito.ceph.com/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388221/ [7388222] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/ffsb}} time: 00:55:56 info: http://pulpito.ceph.com/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388222/ [7388223] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_8 clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}} time: 00:33:06 info: http://pulpito.ceph.com/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388223/ [7388224] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/asok_dump_tree} time: 00:20:37 info: http://pulpito.ceph.com/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388224/ [7388225] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/fs/test_o_trunc}} time: 00:44:15 info: http://pulpito.ceph.com/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388225/ [7388226] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_suites_fsstress}} time: 00:30:07 info: http://pulpito.ceph.com/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388226/ [7388229] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mds 2-workunit/suites/fsstress}} time: 00:25:02 info: http://pulpito.ceph.com/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388229/ [7388230] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/backtrace} time: 00:20:18 info: http://pulpito.ceph.com/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388230/ [7388234] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/iogen}} time: 00:51:04 info: http://pulpito.ceph.com/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388234/ [7388236] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_suites_pjd}} time: 00:18:57 info: http://pulpito.ceph.com/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388236/ [7388237] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/cap-flush} time: 00:15:03 info: http://pulpito.ceph.com/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388237/ [7388240] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_8 clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} time: 00:26:23 info: http://pulpito.ceph.com/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388240/ [7388243] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/client-limits} time: 00:25:36 info: http://pulpito.ceph.com/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388243/ [7388245] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/suites/iozone}} time: 00:38:59 info: http://pulpito.ceph.com/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388245/ [7388246] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/client-readahead} time: 00:19:17 info: http://pulpito.ceph.com/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388246/ [7388247] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/iozone}} time: 00:22:47 info: http://pulpito.ceph.com/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388247/ [7388249] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress traceless/50pc} time: 00:21:28 info: http://pulpito.ceph.com/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388249/ [7388252] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/damage} time: 00:28:20 info: http://pulpito.ceph.com/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388252/ [7388253] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/pjd}} time: 00:35:29 info: http://pulpito.ceph.com/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388253/ [7388254] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_8 clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/quincy}} time: 00:26:19 info: http://pulpito.ceph.com/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388254/ [7388256] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/data-scan} time: 01:44:44 info: http://pulpito.ceph.com/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388256/ [7388257] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/ffsb}} time: 00:45:28 info: http://pulpito.ceph.com/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388257/ [7388259] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress} time: 00:22:14 info: http://pulpito.ceph.com/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388259/ [7388261] fs/cephadm/multivolume/{0-start 1-mount 2-workload/dbench distro/single-container-host} time: 00:50:42 info: http://pulpito.ceph.com/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388261/ [7388262] fs/full/{begin/{0-install 1-ceph 2-logrotate} clusters/1-node-1-mds-1-osd conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse objectstore/bluestore-bitmap overrides overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/mgr-osd-full} time: 00:23:57 info: http://pulpito.ceph.com/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388262/ [7388263] fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{rhel_8} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/client} time: 00:19:32 info: http://pulpito.ceph.com/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388263/ [7388264] fs/mirror/{begin/{0-install 1-ceph 2-logrotate} cephfs-mirror/one-per-cluster clients/{mirror} cluster/{1-node} mount/fuse objectstore/bluestore-bitmap overrides/{whitelist_health} supported-random-distros$/{rhel_8} tasks/mirror} time: 02:14:44 info: http://pulpito.ceph.com/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388264/ [7388267] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/failover} time: 01:20:30 info: http://pulpito.ceph.com/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388267/ [7388269] fs/shell/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/cephfs-shell} time: 00:52:04 info: http://pulpito.ceph.com/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388269/ [7388270] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps} time: 01:15:24 info: http://pulpito.ceph.com/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388270/ [7388271] fs/top/{begin/{0-install 1-ceph 2-logrotate} cluster/{1-node} mount/fuse objectstore/bluestore-bitmap overrides/ignorelist_health supported-random-distros$/{ubuntu_20.04} tasks/fstop} time: 00:17:45 info: http://pulpito.ceph.com/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388271/ [7388272] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_8 clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}} time: 00:32:18 info: http://pulpito.ceph.com/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388272/ [7388274] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu/{latest overrides}} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/1 tasks/dbench validater/lockdep} time: 00:50:46 info: http://pulpito.ceph.com/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388274/ [7388276] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/direct_io}} time: 00:32:17 info: http://pulpito.ceph.com/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388276/ [7388281] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/iozone}} time: 00:19:20 info: http://pulpito.ceph.com/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388281/ [7388282] fs/upgrade/nofs/{bluestore-bitmap centos_8 conf/{client mds mon osd} no-mds-cluster overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-pacific 1-upgrade}} time: 00:17:04 info: http://pulpito.ceph.com/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388282/ [7388283] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/fs/misc}} time: 01:25:36 info: http://pulpito.ceph.com/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388283/ [7388284] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/journal-repair} time: 00:22:11 info: http://pulpito.ceph.com/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388284/ [7388285] fs/upgrade/upgraded_client/{bluestore-bitmap centos_8 clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-from/nautilus 1-mount/mount/fuse 2-workload/new_ops/{0-clients/fuse-upgrade 1-client-sanity}}} time: 00:16:56 info: http://pulpito.ceph.com/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388285/ [7388286] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/ffsb}} time: 00:49:21 info: http://pulpito.ceph.com/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388286/ [7388288] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} kclient-overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_dbench_iozone} time: 00:42:04 info: http://pulpito.ceph.com/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388288/ [7388290] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_snaptests}} time: 00:53:53 info: http://pulpito.ceph.com/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388290/ [7388291] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/kernel_untar_build}} time: 00:49:50 info: http://pulpito.ceph.com/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388291/ [7388292] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/mds-full} time: 00:25:57 info: http://pulpito.ceph.com/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388292/ [7388293] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_8 clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} time: 00:26:34 info: http://pulpito.ceph.com/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388293/ [7388294] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/osd 2-workunit/suites/fsstress}} time: 00:23:34 info: http://pulpito.ceph.com/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388294/ [7388297] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/mds_creation_retry} time: 00:16:53 info: http://pulpito.ceph.com/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388297/ [7388298] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} time: 00:37:20 info: http://pulpito.ceph.com/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388298/ [7388299] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/clone}} time: 00:59:08 info: http://pulpito.ceph.com/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388299/ [7388300] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mds 2-workunit/fs/trivial_sync}} time: 00:10:41 info: http://pulpito.ceph.com/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388300/ [7388302] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/postgres}} time: 01:18:31 info: http://pulpito.ceph.com/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388302/ [7388305] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps} time: 00:48:18 info: http://pulpito.ceph.com/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388305/ [7388306] fs/upgrade/upgraded_client/{bluestore-bitmap centos_8 clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-from/pacific 1-mount/mount/fuse 2-workload/stress_tests/{0-client-upgrade 1-tests/fsstress}}} time: 00:37:10 info: http://pulpito.ceph.com/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388306/ [7388307] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/multimds_misc} time: 00:23:37 info: http://pulpito.ceph.com/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388307/ [7388308] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_suites_fsstress}} time: 00:21:47 info: http://pulpito.ceph.com/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388308/ [7388311] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/openfiletable} time: 00:15:37 info: http://pulpito.ceph.com/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388311/ [7388315] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_suites_pjd}} time: 00:20:35 info: http://pulpito.ceph.com/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388315/ [7388318] fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{rhel_8} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/ino_release_cb} time: 00:18:26 info: http://pulpito.ceph.com/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388318/ [7388319] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_misc} time: 00:22:13 info: http://pulpito.ceph.com/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388319/ [7388320] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/osd 2-workunit/suites/pjd}} time: 00:18:14 info: http://pulpito.ceph.com/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388320/ [7388322] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/finisher_per_module}} time: 00:15:51 info: http://pulpito.ceph.com/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388322/ [7388324] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/recovery-fs} time: 00:14:51 info: http://pulpito.ceph.com/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388324/ [7388326] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_dbench traceless/50pc} time: 00:59:18 info: http://pulpito.ceph.com/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388326/ [7388327] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_8 clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}} time: 00:45:25 info: http://pulpito.ceph.com/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388327/ [7388328] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_trivial_sync}} time: 00:16:58 info: http://pulpito.ceph.com/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388328/ [7388329] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/scrub} time: 00:34:24 info: http://pulpito.ceph.com/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388329/ [7388330] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} time: 00:46:10 info: http://pulpito.ceph.com/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388330/ [7388331] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/sessionmap} time: 00:25:25 info: http://pulpito.ceph.com/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388331/ [7388332] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/osd 2-workunit/fs/snaps}} time: 02:10:34 info: http://pulpito.ceph.com/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388332/ [7388333] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/suites/ffsb}} time: 00:41:49 info: http://pulpito.ceph.com/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388333/ [7388334] fs/upgrade/upgraded_client/{bluestore-bitmap centos_8 clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-from/nautilus 1-mount/mount/fuse 2-workload/stress_tests/{0-client-upgrade 1-tests/kernel_untar_build}}} time: 01:48:37 info: http://pulpito.ceph.com/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388334/ [7388336] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/failover} time: 01:21:11 info: http://pulpito.ceph.com/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388336/ [7388338] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snap-schedule} time: 01:49:55 info: http://pulpito.ceph.com/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388338/ [7388339] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/ffsb}} time: 00:31:21 info: http://pulpito.ceph.com/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388339/ [7388340] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_8 clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} time: 00:44:34 info: http://pulpito.ceph.com/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388340/ [7388342] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mds 2-workunit/suites/ffsb}} time: 01:02:03 info: http://pulpito.ceph.com/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388342/ [7388343] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/fs/norstats}} time: 00:42:24 info: http://pulpito.ceph.com/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388343/ [7388346] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snapshots} time: 00:42:40 info: http://pulpito.ceph.com/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388346/ [7388347] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu/{latest overrides}} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/5 tasks/dbench validater/lockdep} time: 01:00:46 info: http://pulpito.ceph.com/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388347/ [7388348] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mon 2-workunit/suites/fsstress}} time: 00:22:32 info: http://pulpito.ceph.com/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388348/ [7388350] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/strays} time: 00:34:06 info: http://pulpito.ceph.com/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388350/ [7388351] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/suites/fsstress}} time: 00:33:00 info: http://pulpito.ceph.com/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388351/ [7388352] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/iozone}} time: 00:21:57 info: http://pulpito.ceph.com/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388352/ [7388354] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_8 clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/quincy}} time: 00:35:17 info: http://pulpito.ceph.com/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388354/ [7388355] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/truncate_delay} time: 00:16:54 info: http://pulpito.ceph.com/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388355/ [7388356] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/osd 2-workunit/fs/trivial_sync}} time: 00:20:22 info: http://pulpito.ceph.com/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388356/ [7388357] fs/upgrade/upgraded_client/{bluestore-bitmap centos_8 clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-from/pacific 1-mount/mount/fuse 2-workload/stress_tests/{0-client-upgrade 1-tests/dbench}}} time: 01:02:04 info: http://pulpito.ceph.com/mchangir-2023-09-05_10:53:07-fs-wip-mchangir-testing-20230905.131537-testing-default-smithi/7388357/