2022-10-07T08:31:17.190 INFO:root:teuthology version: 1.1.1.dev427+g35ea38a9 2022-10-07T08:31:17.192 INFO:teuthology.results:subset: '8/128' 2022-10-07T08:31:17.192 INFO:teuthology.results:seed: '591' 2022-10-07T08:46:57.430 INFO:root:teuthology version: 1.1.1.dev427+g35ea38a9 2022-10-07T08:46:57.434 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2022-10-07T20:47:52.915 WARNING:teuthology.results:test(s) did not finish before timeout of 43200 seconds 2022-10-07T20:47:54.688 INFO:scrape:Found 11 jobs 2022-10-07T20:48:01.026 INFO:scrape:Found 6 distinct failure reasons 2022-10-07T20:48:01.027 INFO:scrape:Failure: Command failed (workunit test kernel_untar_build.sh) on smithi111 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=df5f48022ffccda1a35889e4b376569077633846 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/kernel_untar_build.sh' 2022-10-07T20:48:01.028 INFO:scrape:1 jobs: ['7057960'] 2022-10-07T20:48:01.028 INFO:scrape:suites: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/yes', '3-snaps/yes', '4-flush/yes', '5-workunit/kernel_untar_build}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/crc', 'n/3', 'objectstore-ec/bluestore-ec-root', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/stock/{k-stock', 'ranks/multi/{export-check', 'replication/always}', 'rhel_8}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-quota}', 'wsync/yes}'] 2022-10-07T20:48:01.029 INFO:scrape: 2022-10-07T20:48:01.029 INFO:scrape:Failure: Test failure: test_newops_getvxattr (tasks.cephfs.test_newops.TestNewOps) 2022-10-07T20:48:01.030 INFO:scrape:2 jobs: ['7057958', '7057952'] 2022-10-07T20:48:01.030 INFO:scrape:suites intersection: ['1-client-upgrade', '2-client-sanity}}', 'centos_latest', 'clusters/{1-mds-1-client-micro}', 'conf/{client', 'fs/upgrade/upgraded_client/from_nautilus/{bluestore-bitmap', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn}', 'tasks/{0-nautilus'] 2022-10-07T20:48:01.031 INFO:scrape:suites union: ['1-client-upgrade', '2-client-sanity}}', 'centos_latest', 'clusters/{1-mds-1-client-micro}', 'conf/{client', 'fs/upgrade/upgraded_client/from_nautilus/{bluestore-bitmap', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn}', 'tasks/{0-nautilus'] 2022-10-07T20:48:01.031 INFO:scrape: 2022-10-07T20:48:01.032 INFO:scrape:Failure: "1665133537.7721372 mon.a (mon.0) 543 : cluster [WRN] Health check failed: Reduced data availability: 2 pgs peering (PG_AVAILABILITY)" in cluster log 2022-10-07T20:48:01.032 INFO:scrape:1 jobs: ['7057951'] 2022-10-07T20:48:01.033 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_8}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'tasks/snap-schedule}'] 2022-10-07T20:48:01.033 INFO:scrape: 2022-10-07T20:48:01.034 INFO:scrape:Failure: error during scrub thrashing: rank damage found: {'backtrace'} 2022-10-07T20:48:01.034 INFO:scrape:1 jobs: ['7057953'] 2022-10-07T20:48:01.035 INFO:scrape:suites: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/yes', '3-snaps/yes', '4-flush/yes', '5-workunit/suites/fsstress}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/secure', 'n/5', 'objectstore-ec/bluestore-comp-ec-root', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/stock/{k-stock', 'ranks/multi/{export-check', 'replication/always}', 'rhel_8}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-quota}', 'wsync/yes}'] 2022-10-07T20:48:01.035 INFO:scrape: 2022-10-07T20:48:01.036 INFO:scrape:Dead: b'2022-10-07T20:44:30.417 DEBUG:teuthology.exit:Finished running handlers' 2022-10-07T20:48:01.036 INFO:scrape:2 jobs: ['7057954', '7057957'] 2022-10-07T20:48:01.037 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-2c-client', 'conf/{client', 'fs/mixed-clients/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'kclient-overrides/{distro/testing/k-testing', 'mds', 'mon', 'ms-die-on-skipped}', 'osd-asserts}', 'osd}', 'overrides/{ignorelist_health', 'tasks/kernel_cfuse_workunits_untarbuild_blogbench}'] 2022-10-07T20:48:01.037 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-2c-client', 'conf/{client', 'fs/mixed-clients/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'kclient-overrides/{distro/testing/k-testing', 'mds', 'mon', 'ms-die-on-skipped}', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp', 'osd-asserts}', 'osd}', 'overrides/{ignorelist_health', 'tasks/kernel_cfuse_workunits_untarbuild_blogbench}'] 2022-10-07T20:48:01.038 INFO:scrape: 2022-10-07T20:48:01.038 INFO:scrape:Failure: Test failure: test_client_cache_size (tasks.cephfs.test_client_limits.TestClientLimits) 2022-10-07T20:48:01.039 INFO:scrape:1 jobs: ['7057955'] 2022-10-07T20:48:01.041 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{rhel_8}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{ignorelist_health', 'tasks/client-limits}'] 2022-10-07T20:48:01.043 INFO:scrape: 2022-10-07T20:48:01.045 INFO:teuthology.results:Sending results to mchangir@redhat.com: Test Run: mchangir-2022-10-07_06:38:36-fs-wip-mchangir-qa-no-such-file-or-directory-57655-testing-default-smithi ================================================================= info: http://pulpito.ceph.com/mchangir-2022-10-07_06:38:36-fs-wip-mchangir-qa-no-such-file-or-directory-57655-testing-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/mchangir-2022-10-07_06:38:36-fs-wip-mchangir-qa-no-such-file-or-directory-57655-testing-default-smithi/ failed: 6 dead: 1 running: 1 waiting: 0 queued: 0 passed: 3 Fail ================================================================= [7057951] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snap-schedule} ----------------------------------------------------------------- time: 00:49:19 info: http://pulpito.ceph.com/mchangir-2022-10-07_06:38:36-fs-wip-mchangir-qa-no-such-file-or-directory-57655-testing-default-smithi/7057951/ log: http://qa-proxy.ceph.com/teuthology/mchangir-2022-10-07_06:38:36-fs-wip-mchangir-qa-no-such-file-or-directory-57655-testing-default-smithi/7057951/ "1665133537.7721372 mon.a (mon.0) 543 : cluster [WRN] Health check failed: Reduced data availability: 2 pgs peering (PG_AVAILABILITY)" in cluster log [7057952] fs/upgrade/upgraded_client/from_nautilus/{bluestore-bitmap centos_latest clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-nautilus 1-client-upgrade 2-client-sanity}} ----------------------------------------------------------------- time: 00:13:38 info: http://pulpito.ceph.com/mchangir-2022-10-07_06:38:36-fs-wip-mchangir-qa-no-such-file-or-directory-57655-testing-default-smithi/7057952/ log: http://qa-proxy.ceph.com/teuthology/mchangir-2022-10-07_06:38:36-fs-wip-mchangir-qa-no-such-file-or-directory-57655-testing-default-smithi/7057952/ sentry: https://sentry.ceph.com/organizations/ceph/?query=6974f24af683407e8c926dcc66d28041 Test failure: test_newops_getvxattr (tasks.cephfs.test_newops.TestNewOps) [7057953] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/fsstress}} ----------------------------------------------------------------- time: 00:37:43 info: http://pulpito.ceph.com/mchangir-2022-10-07_06:38:36-fs-wip-mchangir-qa-no-such-file-or-directory-57655-testing-default-smithi/7057953/ log: http://qa-proxy.ceph.com/teuthology/mchangir-2022-10-07_06:38:36-fs-wip-mchangir-qa-no-such-file-or-directory-57655-testing-default-smithi/7057953/ error during scrub thrashing: rank damage found: {'backtrace'} [7057955] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/client-limits} ----------------------------------------------------------------- time: 00:20:57 info: http://pulpito.ceph.com/mchangir-2022-10-07_06:38:36-fs-wip-mchangir-qa-no-such-file-or-directory-57655-testing-default-smithi/7057955/ log: http://qa-proxy.ceph.com/teuthology/mchangir-2022-10-07_06:38:36-fs-wip-mchangir-qa-no-such-file-or-directory-57655-testing-default-smithi/7057955/ sentry: https://sentry.ceph.com/organizations/ceph/?query=4e7101020e474eeaa372beecb3f08369 Test failure: test_client_cache_size (tasks.cephfs.test_client_limits.TestClientLimits) [7057958] fs/upgrade/upgraded_client/from_nautilus/{bluestore-bitmap centos_latest clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-nautilus 1-client-upgrade 2-client-sanity}} ----------------------------------------------------------------- time: 00:13:05 info: http://pulpito.ceph.com/mchangir-2022-10-07_06:38:36-fs-wip-mchangir-qa-no-such-file-or-directory-57655-testing-default-smithi/7057958/ log: http://qa-proxy.ceph.com/teuthology/mchangir-2022-10-07_06:38:36-fs-wip-mchangir-qa-no-such-file-or-directory-57655-testing-default-smithi/7057958/ sentry: https://sentry.ceph.com/organizations/ceph/?query=ada3db65ef9e4ff2bbbfaf8b9cf84249 Test failure: test_newops_getvxattr (tasks.cephfs.test_newops.TestNewOps) [7057960] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/kernel_untar_build}} ----------------------------------------------------------------- time: 00:34:42 info: http://pulpito.ceph.com/mchangir-2022-10-07_06:38:36-fs-wip-mchangir-qa-no-such-file-or-directory-57655-testing-default-smithi/7057960/ log: http://qa-proxy.ceph.com/teuthology/mchangir-2022-10-07_06:38:36-fs-wip-mchangir-qa-no-such-file-or-directory-57655-testing-default-smithi/7057960/ sentry: https://sentry.ceph.com/organizations/ceph/?query=c3fbc9d862cb4de1b0bcf63329f83e35 Command failed (workunit test kernel_untar_build.sh) on smithi111 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=df5f48022ffccda1a35889e4b376569077633846 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/kernel_untar_build.sh' Dead ================================================================= [7057954] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} kclient-overrides/{distro/testing/k-testing ms-die-on-skipped} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_untarbuild_blogbench} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/mchangir-2022-10-07_06:38:36-fs-wip-mchangir-qa-no-such-file-or-directory-57655-testing-default-smithi/7057954/ log: http://qa-proxy.ceph.com/teuthology/mchangir-2022-10-07_06:38:36-fs-wip-mchangir-qa-no-such-file-or-directory-57655-testing-default-smithi/7057954/ hit max job timeout Running ================================================================= [7057957] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} kclient-overrides/{distro/testing/k-testing ms-die-on-skipped} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_untarbuild_blogbench} info: http://pulpito.ceph.com/mchangir-2022-10-07_06:38:36-fs-wip-mchangir-qa-no-such-file-or-directory-57655-testing-default-smithi/7057957/ Pass ================================================================= [7057950] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/basic}} time: 02:21:20 info: http://pulpito.ceph.com/mchangir-2022-10-07_06:38:36-fs-wip-mchangir-qa-no-such-file-or-directory-57655-testing-default-smithi/7057950/ [7057956] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/forward-scrub} time: 00:17:53 info: http://pulpito.ceph.com/mchangir-2022-10-07_06:38:36-fs-wip-mchangir-qa-no-such-file-or-directory-57655-testing-default-smithi/7057956/ [7057959] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/fs/norstats}} time: 00:29:41 info: http://pulpito.ceph.com/mchangir-2022-10-07_06:38:36-fs-wip-mchangir-qa-no-such-file-or-directory-57655-testing-default-smithi/7057959/