2023-11-06T17:15:34.027 INFO:root:teuthology version: 0.0.1.dev203+g6899cd2
2023-11-06T17:15:34.028 INFO:teuthology.results:subset: '111/12000'
2023-11-06T17:15:34.028 INFO:teuthology.results:seed: '1834'
2023-11-06T19:38:11.271 INFO:root:teuthology version: 0.0.1.dev203+g6899cd2
2023-11-06T19:38:11.296 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2023-11-07T07:05:04.138 INFO:teuthology.results:Tests finished! gathering results...
2023-11-07T07:05:07.777 INFO:scrape:Found 46 jobs
2023-11-07T07:05:11.825 WARNING:scrape:Missing teuthology log /home/teuthworker/archive/yuriw-2023-11-06_15:46:57-fs-reef-release-distro-default-smithi/7450187/teuthology.log
2023-11-07T07:05:12.372 WARNING:scrape:Misunderstood line: 
2023-11-07T07:05:21.070 INFO:scrape:Found 9 distinct failure reasons
2023-11-07T07:05:21.071 INFO:scrape:Failure: "2023-11-06T18:29:23.657953+0000 mon.a (mon.0) 146 : cluster [WRN] Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log
2023-11-07T07:05:21.072 INFO:scrape:22 jobs: ['7450165', '7450153', '7450194', '7450185', '7450144', '7450151', '7450163', '7450188', '7450145', '7450177', '7450200', '7450195', '7450147', '7450143', '7450202', '7450189', '7450193', '7450157', '7450183', '7450199', '7450171', '7450167']
2023-11-07T07:05:21.072 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}']
2023-11-07T07:05:21.073 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', 'bluestore-bitmap', 'centos_latest', 'cluster/{1-node}', 'clusters/1-mds-1-client', 'clusters/1-mds-2-client', 'clusters/1a3s-mds-1c-client', 'clusters/1a3s-mds-2c-client', 'clusters/1a3s-mds-4c-client', 'clusters/fixed-2-ucephfs', 'clusters/small-cluster', 'conf/{client', 'distro/{centos_latest}', 'distro/{ubuntu_latest}', 'distros/ubuntu_latest', 'fs/32bits/{begin/{0-install', 'fs/bugs/client_trim_caps/{begin/{0-install', 'fs/fscrypt/{begin/{0-install', 'fs/functional/{begin/{0-install', 'fs/multiclient/{begin/{0-install', 'fs/multifs/{begin/{0-install', 'fs/permission/{begin/{0-install', 'fs/snaps/{begin/{0-install', 'fs/top/{begin/{0-install', 'fs/traceless/{begin/{0-install', 'fs/volumes/{begin/{0-install', 'ignorelist_health', 'ignorelist_health_more', 'ignorelist_wrongly_marked_down', 'ignorelist_wrongly_marked_down}', 'mds', 'mon', 'mon-debug}', 'mount', 'mount/fuse', 'mount/kclient/{mount-syntax/v1', 'mount/kclient/{mount-syntax/{v1}', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/ignorelist_health', 'overrides/{distro/testing/k-testing', 'overrides/{distro/testing/k-testing}}', 'overrides/{faked-ino', 'overrides/{frag', 'overrides/{ignorelist_health', 'pg-warn}', 'supported-random-distros$/{ubuntu_latest}', 'tasks/admin}', 'tasks/cfuse_workunit_misc}', 'tasks/cfuse_workunit_suites_ffsb', 'tasks/cfuse_workunit_suites_fsstress}', 'tasks/cfuse_workunit_suites_pjd}', 'tasks/failover}', 'tasks/forward-scrub}', 'tasks/fscrypt-dbench}', 'tasks/fscrypt-iozone}', 'tasks/fstop}', 'tasks/mdtest}', 'tasks/metrics}', 'tasks/scrub}', 'tasks/sessionmap}', 'tasks/snap-schedule}', 'tasks/strays}', 'tasks/trim-i24137}', 'tasks/volumes/{overrides', 'tasks/workunit/snaps}', 'test/basic}}', 'traceless/50pc}']
2023-11-07T07:05:21.074 INFO:scrape:
2023-11-07T07:05:21.074 INFO:scrape:Failure: Test failure: test_add_ancestor_and_child_directory (tasks.cephfs.test_mirroring.TestMirroring)
2023-11-07T07:05:21.075 INFO:scrape:1 jobs: ['7450186']
2023-11-07T07:05:21.076 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'centos_latest', 'clients/mirror', 'cluster/1-node', 'debug', 'fs/valgrind/{begin/{0-install', 'mirror/{cephfs-mirror/one-per-cluster', 'mount/fuse', 'overrides/whitelist_health', 'tasks/mirror}}']
2023-11-07T07:05:21.076 INFO:scrape:
2023-11-07T07:05:21.077 INFO:scrape:Failure: error during scrub thrashing: rank damage found: {'backtrace'}
2023-11-07T07:05:21.078 INFO:scrape:2 jobs: ['7450175', '7450155']
2023-11-07T07:05:21.079 INFO:scrape:suites intersection: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/yes', '4-flush/yes', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/crc', 'n/5', 'objectstore-ec/bluestore-comp', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/stock/{k-stock', 'ranks/multi/{export-check', 'rhel_8}', 'session_timeout}', 'standby-replay', 'wsync/yes}']
2023-11-07T07:05:21.079 INFO:scrape:suites union: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/yes', '3-snaps/no', '3-snaps/yes', '4-flush/yes', '5-workunit/postgres}}', '5-workunit/suites/pjd}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/crc', 'n/5', 'objectstore-ec/bluestore-comp', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/stock/{k-stock', 'ranks/multi/{export-check', 'replication/always}', 'replication/default}', 'rhel_8}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-namespace-isolated-and-quota}', 'tasks/{0-subvolume/{with-namespace-isolated}', 'wsync/yes}']
2023-11-07T07:05:21.080 INFO:scrape:
2023-11-07T07:05:21.081 INFO:scrape:Dead: b'2023-11-07T05:32:31.179 DEBUG:teuthology.exit:Finished running handlers'
2023-11-07T07:05:21.082 INFO:scrape:2 jobs: ['7450149', '7450179']
2023-11-07T07:05:21.082 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'conf/{client', 'mds', 'mon', 'osd}', 'overrides/{ignorelist_health']
2023-11-07T07:05:21.083 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', 'bluestore-bitmap', 'clusters/1-mds-1-client', 'clusters/1-mds-3-client', 'conf/{client', 'distro/{ubuntu_latest}', 'distros/ubuntu_latest', 'fs/fscrypt/{begin/{0-install', 'fs/multiclient/{begin/{0-install', 'ignorelist_health_more', 'ignorelist_wrongly_marked_down', 'ignorelist_wrongly_marked_down}', 'mds', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{mount-syntax/v1', 'objectstore-ec/bluestore-bitmap', 'osd}', 'overrides/{distro/testing/k-testing}}', 'overrides/{ignorelist_health', 'pg-warn}', 'tasks/fscrypt-common}', 'tasks/ior-shared-file}']
2023-11-07T07:05:21.084 INFO:scrape:
2023-11-07T07:05:21.085 INFO:scrape:Dead: None
2023-11-07T07:05:21.085 INFO:scrape:4 jobs: ['7450187', '7450196', '7450192', '7450181']
2023-11-07T07:05:21.086 INFO:scrape:
2023-11-07T07:05:21.087 INFO:scrape:Valgrind: mon (InvalidRead, Leak_PossiblyLost, Leak_StillReachable)
2023-11-07T07:05:21.088 INFO:scrape:1 jobs: ['7450142']
2023-11-07T07:05:21.088 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{ubuntu/{latest', 'fs/verify/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mon-debug', 'mount/fuse', 'objectstore-ec/bluestore-ec-root', 'osd}', 'overrides/{ignorelist_health', 'overrides}}', 'ranks/3', 'session_timeout}', 'tasks/dbench', 'validater/valgrind}']
2023-11-07T07:05:21.089 INFO:scrape:
2023-11-07T07:05:21.090 INFO:scrape:Failure: Test failure: test_pool_perm (tasks.cephfs.test_pool_perm.TestPoolPerm)
2023-11-07T07:05:21.091 INFO:scrape:1 jobs: ['7450198']
2023-11-07T07:05:21.091 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_latest}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{ignorelist_health', 'tasks/pool-perm}']
2023-11-07T07:05:21.092 INFO:scrape:
2023-11-07T07:05:21.093 INFO:scrape:Failure: Test failure: test_cluster_set_reset_user_config (tasks.cephfs.test_nfs.TestNFS)
2023-11-07T07:05:21.094 INFO:scrape:1 jobs: ['7450182']
2023-11-07T07:05:21.094 INFO:scrape:suites: ['fs/nfs/{cluster/{1-node}', 'overrides/ignorelist_health', 'supported-random-distros$/{centos_latest}', 'tasks/nfs}']
2023-11-07T07:05:21.095 INFO:scrape:
2023-11-07T07:05:21.096 INFO:scrape:Failure: Test failure: test_subvolumegroup_pin_distributed (tasks.cephfs.test_volumes.TestSubvolumeGroups)
2023-11-07T07:05:21.097 INFO:scrape:1 jobs: ['7450197']
2023-11-07T07:05:21.097 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{rhel_8}', 'fs/volumes/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{ignorelist_health', 'rhel_8}', 'tasks/volumes/{overrides', 'test/basic}}']
2023-11-07T07:05:21.098 INFO:scrape: